In this new agile world, many people question the role of architecture. And certainly the pre-planned architecture vision couldn’t fit in with modern dynamism. But there is another approach to architecture, one that embraces change in the agile manner. In this view architecture is an constant effort, one that works closely with programming so that architecture can react both to changing requirements but also to feedback from programming. It is called “Evolutionary Architecture”, to highlight that while changes are unpredictable, architecture can still move in a better direction.

Ethereum is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third-party interference.

These apps run on a custom built blockchain, an enormously powerful shared global infrastructure that can move value around and represent the ownership of property.

This enables developers to create markets, store registries of debts or promises, move funds in accordance with instructions given long in the past (like a will or a futures contract) and many other things that have not been invented yet, all without a middleman or counter party risk.

Progressive Web Apps are user experiences that have the reach of the web, and are:

• Reliable - Load instantly and never show the downasaur, even in uncertain network conditions.

• Fast - Respond quickly to user interactions with silky smooth animations and no janky scrolling.

• Engaging - Feel like a natural app on the device, with an immersive user experience.

This new level of quality allows Progressive Web Apps to earn a place on the user’s home screen.

Prisma is a data layer for your applications, designed to build GraphQL services as well as REST APIs, over your existing database and API actions.

GraphQL subscriptions allow us to make a query against a GraphQL database, and be notified whenever the results change.

For example, we could make a query for the number of stars for a GitHub repo, and be notified whenever someone stars/unstars the repo and the number changes.

Stenciljs is a Compiler for building Web Components that can be used in a JavaScript Projects(Angular, React, Vue) or in a vanilla project. The produced code also includes: A tiny virtual DOM layer Efficient one-way data binding An asynchronous rendering pipeline (similar to React Fiber) Lazy-loading

Stenciljs do not use the shadow dom

Stenciljs

Although browsers are running more and more applications, and less and less websites, server side rendering is still important for search engines, content previews when sharing apps, and is still makes our page load faster where static content is involved.

There are techniques for rendering content on the server side which involves running client code on the server, for SEO and initial page load, which we may recommend to start using.

Since HTML5 has come out, and the browsers have stopped running plugins, hosting non-javascript code in the browser may come alive with WebAssembly.

(From https://webassembly.org) WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.

Minimalistic approach for managing npm package lifecycle scripts (build, test, deploy, etc’) with cli tools.

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

• GraphQL queries always return predictable results.
• While typical REST APIs require loading from multiple URLs, GraphQL APIs get all the data your app needs in a single request.
• GraphQL uses types to ensure Apps only ask for what’s possible and provide clear and helpful errors.

GraphQL link

The Apollo Platform is a family of technologies you can incrementally add to your stack: Apollo Client to connect data to your UI, Apollo Engine for infrastructure and tooling, and Apollo Server to translate your REST API and backends into a GraphQL schema.

• Apollo Client Bind data to your UI with the ultra-flexible, community-driven GraphQL client for React, JavaScript, and native platforms.
• ** Apollo Server** Translate your existing REST APIs and backends into GraphQL with this powerful set of tools for buliding GraphQL APIs.
• Apollo Engine The GraphQL gateway that provides essential features including caching, performance tracing, and error tracking.

Apollo link

https://cssinjs.org

overview: although there are many advantages of using this library, JSS, as it names apply, does not try to convey a specific value besides the most obvious one - it emphasizes the isomorphic nature that comes with writing CSS in JS.

cons:

• hard to get used to, CSS is easier.
• requires setup to get started (CSS is native)
• SCSS/SASS is easier to learn and can output most features same as JSS.
• mostly used for heavy and complex Apps. wouldn’t use this for a simple app.
• adds complexity and requires more advanced know-how for writing simple CSS code.

pros:

• integrates well in JSX inline stylesheets.
• integrates well in React + React Native + Angular apps.
• modular stylesheets
• generates high-performance CSS selectors for faster style rendering
• easily integrated to GULP/WEB PACK
• auto-generate unique “post-CSS” like CSS selectors
• easier to create simple “one file” components that include both functions and styles in the same place. -support for CSS rules isolation
• Scoped selectors.

Flow is a static type checker for your JavaScript code developed and promoted by Facebook as an alternative to TypeScript. Flow uses static type annotations. This typing can also be done by flow implicitly by analyzing the code. For example - With type declaration:

// @flow
function square(n: number): number {
  return n * n;
}

square("2"); // Error!

and without:

// @flow
function square(n) {
  return n * n; // Error!
}

square("2");

Flow uses a compiler (Babel or flow-remove-types as the code compiler to remove the flow type annotations for the runtime JS.

We believe that Flow can provide the safety of type checking with minimal impact on the amount and style of code.

Vue.js is a javascript framework that has been around since 2014 but is becoming more and more popular in the js/ frontend community and considered (today) the third most used framework (after react and angular). Vue.js is designed in such a way that it can be incrementally adoptable and scale between a library and a framework depending on different use cases - from a view layer only in part of the application up to a full-blown ecosystem for complex Single Page Applications.

ES2017 are new standards that got into the JS language, partially supported by browsers and engines. New language features such as async / await, new browser API such as shared memory and so on.

Snapshot testing is a feature of the Jest testing framework (Since v. 14.0, 2016) for asserting declarative code (produces simple views). While running snapshot test, jest produces a serializable (rendered) value for the react tree. In this way, snapshot testing allows one to:

1. See the expected rendered outcome, including testing multiple versions of the expected outcome (error msgs for examp.)
2. Test changes in the code and make sure that any change in the outcome is intentional
3. run a ‘semi-integration’ tests (a couple of components)

Websocket is the most efficient way for duplex communication between server and client. There are many libraries . implementing websocket communication and enable follback to other forms (SSE, long-pollng) when it is not supported.

CSS variables are entities defined by CSS authors that contain specific values to be reused throughout a document. They are set using custom property notation (e.g., --main-color: black;) and are accessed using the var() function (e.g., color: var(--main-color);).

Complex websites have very large amounts of CSS, often with a lot of repeated values. For example, the same color might be used in hundreds of different places, requiring global search and replace if that color needs to change. CSS variables allow a value to be stored in one place, then referenced in multiple other places. An additional benefit is semantic identifiers. For example, --main-text-color is easier to understand than #00ff00, especially if this same color is also used in other contexts.

CSS variables are supported in all currently used browsers except IE and Opera Mini. https://caniuse.com/#feat=css-variables

CSS modules are styles scoped locally to a component.

When creating a component, you may import style from a css file, using import statement, and all CSS rules will be scoped to the component.

The css module compiler changes class names to have a unique name that will match only the selector. It works almost the same way as emulated view encapsulation in angular, however, it has many implementations so it is framework agnostic.

https://github.com/css-modules/css-modules

GraphQL gains popularity due to its ability to have query language over HTTP along with getting results from multiple resources without the hassle of writing multiple requests and combining the results.

In case you need to have responses from multiple results, the GraphQL API enables you to achieve the responses as one, faster than in REST API.

Sass and Less are dynamic preprocessor style sheet languages that compile to css. The use of one of both is very wide-spread in the frontend domain, although, with the advancement of css itself (css3, css variables, etc.), more developers are returning to write ‘pure’ css.

RxJS is an implementation of ReactiveX in Javascript, a library for reactive programming using Observables, to make it easier to compose asynchronous or callback-based code.

ReactiveX is more than an API, it’s an idea and a breakthrough in programming. It has inspired several other APIs, frameworks, and even programming languages.

It has the following principles:

Functional Avoid intricate stateful programs, using clean input/output functions over observable streams.

Less is more ReactiveX’s operators often reduce what was once an elaborate challenge into a few lines of code.

Async error handling Traditional try/catch is powerless for errors in asynchronous computations, but ReactiveX is equipped with proper mechanisms for handling errors.

Concurrency made easy Observables and Schedulers in ReactiveX allow the programmer to abstract away low-level threading, synchronization, and concurrency issues.

Functional Programming is a programming paradigm that become more popular in the recent years, it has many benefits for even-driven architectures and other state/contexts handling/processing problems.

Blockchain is probably the hottest ever buzzword in tech. Hotter than AI, hotter than VR. Surely hotter than Kubernetes. $6B were raised in ICOs by blockchain related startups in 2017, this is more than the entire VC money invested in early stage startups.

Is blockchain just a buzz? Chris Dixon who is a VC that does believe in Blockchain says: [blockchain] is probably the largest and smartest software engineering organization in the world. There are probably 20,000 programmers, if not more, working on this stuff. There’s a lot of really really smart people. I spend a lot of time with them and I’m just constantly impressed. Do you want to bet against those 20,000 really smart engineers who are like super passionate about this and working on it all the time? I wouldn’t want to bet against them.

The $6B is going to be used mainly in R&D in the coming years, and I see 3 main opportunities for Tikal here:

• Support development of infrastructure projects: Most of the bigger ICOs are infrastructure projects. i.e blockchains and technologies that will enable future decentralized apps development. HyperLedger seems to be taking a major role in this emerging world.
• Support development of ethereum dapps: Ethereum is a rather mature blockchain and very popular. It already has vast dev tools such as Truffle, Ganache, OpenZeppelin, MetaMask and many more. This is the 1st blockchain were demand for expertise in such tools for developing dapps will be required
• Support usage of edge clouds based on blockchains: two major projects are Golem and Dfinity who aim to decentralize the cloud. It is not far fetched to believe that they will be major competition for AWS, Google Cloud and Azure and expertise in them will be required (DevOps)

Webpack (currently in v. 4.6) is is a module bundler. It analyzes a cluster of interdependent modules (mainly js but also other assets) and produces bundles of static items.

Webpack is highly configurable, which sometimes considered to be one its downsides) and have a very rich plugin ecosystem.

In the current version, Webpack introduced

Webpack is de facto a standard for frontend web development in general and build in particular.

React Native - building native apps with React.

React Native is the first and most popular paradigm for building native apps with JS. these are not the weak PhoneGap like web apps but real native applications, with real native-like performance.

RN is not dependent on any vendor in order support new native features, it is in sync with the native OS components and is partly what makes it is so strong.

It is also relatively easy to pick up, especially if you worked with React Web before.

it has a vivid community with many users and contributors.

this technology really bridges frontend and mobile in the best way I know of to date.

lastly, RN is used in React VR/AR.

refs:

• official site : https://facebook.github.io/react-native/
• popular ui framework - https://react-native-training.github.io/react-native-elements/docs/button.html
• trends overview - https://trends.google.com/trends/explore?q=react%20native,native%20base,phonegap,%2Fm%2F0gtv959,%2Fm%2F05q31

Typescript has become popular once again since Angular team has chosen it as their main programming language. Many of TS standards were embraced in ES2015-ES2018 and it has become heavily used in almost every javascript framework.

The Angular framework, less than two years after it’s first release, have gained strong community and large number of users.

From it’s v2 to v6, angular has proven relatively easy migration path along with frequent major releases which means that it is here to stay.

React is an old news, but still, it keeps changing and improving. React 16 has many improvements and feature which worth to track.

Among the changes are some long-standing feature requests, including fragments, error boundaries, portals, support for custom DOM attributes, improved server-side rendering, and reduced file size.

Reactive Programming is a paradigm that changes the direction of the flow. Instead of creating threads and blocking for communications, we create pipelines where data is pass though and each layer can react to the data received. The indirection changes the code flow from blocked to async.

Over time the reactive extensions has become a standard for adding higher level functionality on the stream of data while being agnostic to the data itself. The extensions are based on the observable pattern, and allow for cancellation of data process in stream

Client applications, have become more and more complex and contains more and more code throughout the years.

In many cases, organisation starts to think about migrating or rewriting their code.

On the server side, many organisation have solved their problem by splitting their monolith into microservices, while on the client side you can actually do the same.

Microfrontend is still fresh. Every organisation builds its own proprietary standard for their solution, nevertheless, the standards and patterns are the same.

We need to build a library / framework that implements general standard we can offer to organisations, to have this architecture implemented much quicker and easier.

Storybook has become an essential tool in the component development toolbox. It allows for development and showcasing of components easily with minimal environment configuration. Many libraries use storybook to create component catalogs.

Storybook is easily pluggable and there are very useful plugins.

A reference storybook by AirBnB can be found at [http://airbnb.io/react-dates/]

What is it? Prettier is an opinionated code formatter. It enforces a consistent style by parsing your code and re-printing it with its own rules.

Why does it matter? Having a code “style guide” is very valuable for a project, it minimizes friction between developers that can arise from different coding styles and it helps to focus on the code instead of focusing on formatting.

How does it work? Maintaining a codebase that is readable for everyone is not an easy task without automating the formatting process. Prettier is an automatic code formatter that comes preconfigured with “best practice” and standard rules that can be overridden via a configuration file.

When to use it? In any project! developed by the people that built React and react-Native and used by a lot of major projects - Prettier is a must have in any project.

Styled-Components is a library that easily allows developers to take advantage of the best of all worlds of modern CSS styling with a minimal setup effort. It enables CSS modules default to avoid styles collision. It enables complex style hierarchies, functions, and variables without setting up complex Webpack configuration. We think this is a very efficient and powerful solution for modern web app’s styling.

Web Components are the missing piece in UI development. It enables to write portable pieces of UI and cost zero effort when the next shiny framework arrives. It is compatible with all current trending libraries (i.e. React/Angular/AngularJS) and has a growing community and interest.

The standard is now supported in Chrome/Firefox/Safari and IE11/Edge with polyfills. Within the next 6 months Microsoft is expected to support it natively in Edge.

Pros: High-Performant, Community is based on high-skilled developers, Portable and TRULY reusable. Money saver. Cons: Community contribution is still early and there’s a lot of questions and void (that can be answered).

Tikal has an opportunity to create and provide knowledge and tools that are missing in the void, and be an “edge” company that is there before it explodes and becomes mainstream. Community’s missing tools are

• Automated Tests
• Dedicated developer tools
• Libraries/Components publishing mechanism.

I can provide knowledge about this technology and I have a beta version of an automated tool similar to facebook’s Storybook but is extended and provides integration with Puppeteer (Automated tests for UI).

Selenium has become a default e2e testing mechanism, nevertheless it require a great deal of setups for web drivers which can become a hassle especially when running automation on top of build machines, some organisations refuse to install browsers on their build machines.

TestCafe as decided not to build their engine on top of Selenium https://testcafe-discuss.devexpress.com/t/why-not-use-selenium-how-to-use-testcafe/47

Archery is an opensource vulnerability assessment and management tool which helps developers and pentesters to perform scans and manage vulnerabilities.

Although tool scanners, linters and other Quality Assurance techniques which were automated throughout the years, we do see a paradigm shift in terms of what may commonly be called the “Security First Approach” (Salo Shp), which indicates a change in how Security has climbed the ladder from the least important task or most tedious to the highest or most important one. And with the “mass production” of Microservices, we will most likely see more an more solutions which enable automation and streamlining of security processes and regulation down the Production Pipeline.

Archery uses popular opensource tools to perform comprehensive scanning for web application and network. It also performs web application dynamic authenticated scanning and covers the whole application by using selenium.

These types of tooling are a great complementary to many other security disciplines such as IAM or secret & token management, key rotations & policies, mTls, identity management and many other security related concerns which nowadays have become more a necessity than a luxury.

kube-bench is a Go application that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

Tests are configured with YAML files, making this tool easy to update as test specifications evolve

gRPC (Google Remote Procedure Call) is an open source remote procedure call (RPC) system initially developed at Google. It uses HTTP/2 for transport, Protocol Buffers as the interface description language, and provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, cancellation and timeouts. It generates cross-platform client and server bindings for many languages (C++, Java, Python Node, etc).

In this new agile world, many people question the role of architecture. And certainly the pre-planned architecture vision couldn’t fit in with modern dynamism. But there is another approach to architecture, one that embraces change in the agile manner. In this view architecture is an constant effort, one that works closely with programming so that architecture can react both to changing requirements but also to feedback from programming. It is called “Evolutionary Architecture”, to highlight that while changes are unpredictable, architecture can still move in a better direction.

AWS was the first provider to offer functions as a service already in November 2014. AWS initially released AWS lambda as an event-driven provisioning/operations and took just under 3 years to become the standard name of serverless and FaaS offerings.

AWS, as it’s competitors, offer Lambdas (a.k.a functions), as a complimentary to their BaaS offerings, stitching together services such as:

• Incognito
• S3
• CloudFormation
• DynamoDB
• RDS and many more

These integrations alongside Lambda’s “infinite” scalability and it’s newly introduced (at the time) “Price per 100ms” made it very popular among both startups achieving their MVP and enterprises wishing to scale out or experiment with Serverless and Micro Services Architectures.

AWS lambda provides many organizations the ability to write functions in a variety of programming languages and integrates well with many frameworks and other IaaS/PaaS/BaaS services.

More about AWS Lamda here

Cloud Native Solution With the broad adoption of SOA and Micro Service Architectures and many IaaS, PaaS offerings one of the “lessons learned” is that you need to be able to harness your infrastructure and start treating it in a standardized way.

So What is Cloud Native? The Cloud Native Computing Foundation (CNCF) describes it as “distributed systems capable of scaling to tens of thousands of self healing multi-tenant nodes”. That’s a “how” (distributed systems) and a “why” (high scaleability and automated resilience).

All the above in a complex / polyglot world will only be possible with standards and tooling to enable all these moving parts to suit small to large scale microservice based applications - This is what the Cloud Native Movement and the CNCF will promote in the years to come.

The following principles drive Cloud Native solutions:

• Treat your own / cloud / hybrid infrastructure as-a-service: run on servers that can be flexibly provisioned on demand.

• Microservices architecture: individual components are small, loosely coupled.

• Automated deployments and continuous ly integrated and test -> replace manual tasks with scripts or code.

• Containerize: package processes with their dependencies making them easy to test, move and deploy.

• Orchestrate: use standard / commonly used / battle tested orchestration tools.

Cloud Service Providers As of today, the cloud service providers play a major role in all startup companies and even in big companies. This get the programmers to the point where knowing Linux and ops utilities is not enough. Developers are expected to be experts in the different cloud provider in order to succeed with their tasks.

Each cloud provider comes with new platforms, techniques, API’s and databases.

Docker has revolutionised the world. However, business can stale development, and we believe this year we’ll see less usage of Docker and more of other alternatives:

• Distroless Docker Images (Google)
• rkt (CoreOS)
• Makisu (Uber)

A Short History of the Chaostoolkit “We were trying to define a common, declarative API for chaos engineering”, this tool is an example of how “planning to fail” works in distributed / cloud native / microservice architecture based products / solutions.

Tools like this become part of your development process, stress testing your application isn’t a luxury, getting accuracy of resilience & scalability is a must with the growing demand of elastic/scalable infrastructure.

As the growth in popularity of Functions in General and Serverless architecture in specific, vaying from Cloud Native/Cloud-based to On-Prem solutions. Serverless architecture and especially the “Function part” has become the go-to tool for many DevOps processes which need automating and in many cases act as the binding context of the by design “loosely coupled” architecture.

As we grow in event driven processes we will see more and more functions listening on event streams such as SNS on AWS, or NATS, Kafka etc on tools like OpenFaaS. There tools/frameworks enable many DevOps teams to achieve their goals in a consistent, monitored and repetative way, not to mantion the replacement of many utilities such as CronJobs etc.

With the growing usage of containers and Container orchestration platforms such as Kubernetes, it’s a good time to rethink storage completely. When we started with Docker and Containers Persistence and Storage was out of scope. As applications now consist of volatile sets of loosely coupled microservices running in containers, ever-changing in scale, constantly being updated and re-deployed, and always evolving, the traditional mindset of serving storage and data services has changed.

Kubernetes paved the way to enable these types of applications by making it inexpensive and manageable to run the hundreds of container instances that make up an application. Likewise, software-defined storage (SDS) made running dozens of systems that make up a highly elastic storage system serving hundreds of volumes/LUNs to many clients viable and manageable. Now is a perfect time to combine these two systems using Persistent Volumes and Claims and of course the rise of Operators such as Rook which introduce Software Defined Storage in the form of an Operator.

AWS was the first provider to offer functions as a service already in Nov 2014. AWS initially released AWS lambda as an event-driven provisioning/operations and took just under 3 years to become the standard name of serverless and FaaS offerings.

AWS, as it’s competitors, offer Lambdas (a.k.a functions), as a complimentary to their BaaS offerings, stitching together services such as:

• Incognito
• S3
• CloudFormation
• DynamoDB
• RDS and many more

These integrations alongside Lambda’s “infinite” scalability and it’s newly introduced (at the time) “Price per 100ms” made it very popular among both startups achiving their MVP and enterprises wishing to scale out or experiment with Serverless and Micro Services Architectures.

AWS lambda provides many orginazations the ability to write functions in a variaty of Software languages and integrates well with many pfameworks and other IaaS/PaaS/BaaS services.

More about AWS Lamda here

Google functions enable Serverless applications based on GCP IaaS offerings.

Cloud Functions lets application developers spin up code on demand in response to events originating from any api / http request. Serverless architectures utilizing Google Functions integrated with Google Endpoints and BaaS services could build applications that scale from zero to infinity, on demand - without provisioning or managing a single server.

As other serverless and function providers, google’s functions are the best fitted for Backend services such as Firebase, Cloud Datastore, and ML solutions also offered by the GCP.

More info on google functions in the following link

opensource : kubeless https://www.tikalk.com/posts/2018/05/21/intro-to-kubeless/

OpenFaaS https://blog.alexellis.io/introducing-functions-as-a-service/

Istio is the most talked about Service Mesh which is inversion 1.0.5 as of day and a growing adoption rate from self-manages to Istio as a service.

To clarify, the term service mesh is used to describe the network of microservices that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery __, __load balancing, failure recovery, __metrics __, and __monitoring __.

A service mesh also often has more complex operational requirements, like __A/B testing __, __canary releases __, __rate limiting __, __access control __, and __end-to-end authentication __.

The Service Mesh, in general, is the enabler of monolith apps to co-exist alongside one another until a full transition to microservices is done.

Istio.io -> Istio’s diverse feature set lets you successfully, and efficiently, run a distributed microservice architecture, and provides a uniform way to secure, connect, and monitor microservices - implementing all the technics and methodologies mentioned above (which used to be part of the Backend).

AWS was the first provider to offer functions as a service already in Nov 2014. AWS initially released AWS lambda as an event-driven provisioning/operations and took just under 3 years to become the standard name of serverless and FaaS offerings.

AWS, as it’s competitors, offer Lambdas (a.k.a functions), as a complimentary to their BaaS offerings, stitching together services such as:

• Incognito
• S3
• CloudFormation
• DynamoDB
• RDS and many more

These integrations alongside Lambda’s “infinite” scalability and it’s newly introduced (at the time) “Price per 100ms” made it very popular among both startups achiving their MVP and enterprises wishing to scale out or experiment with Serverless and Micro Services Architectures.

AWS lambda provides many orginazations the ability to write functions in a variaty of Software languages and integrates well with many pfameworks and other IaaS/PaaS/BaaS services.

More about AWS Lamda here

Google functions enable Serverless applications based on GCP IaaS offerings.

Cloud Functions lets application developers spin up code on demand in response to events originating from any api / http request. Serverless architectures utilizing Google Functions integrated with Google Endpoints and BaaS services could build applications that scale from zero to infinity, on demand - without provisioning or managing a single server.

As other serverless and function providers, google’s functions are the best fitted for Backend services such as Firebase, Cloud Datastore, and ML solutions also offered by the GCP.

More info on google functions in the following link

The Operator Pattern A Kubernetes Operator is simply a domain specific controller that can manage, configure and automate the lifecycle of stateful applications. Managing stateful applications, such as databases, caches and monitoring systems running on Kubernetes is notoriously difficult. By leveraging the power of Kubernetes API we can now build self managing, self driving infrastructure by encoding operational knowledge and best practices directly into code. For instance, if a MySQL instance dies, we can use an Operator to react and take the appropriate action to bring the system back online.

The Kubeflow project is dedicated to making deployments of machine learning workflows on Kubernetes simple, portable and scalable. This project includes services such as Jupyter notebooks, TensorFlow model training, and model service, all on a Kubernetes infrastructure. This project provides DevOps infrastructure and services to the ML world. At Tikal, we can leverage such infrastructure in customer’s projects.

GO is A programming language introduced by Google in 2009. It is a compiled and strongly typed language similar to C, but bringing a much more intuitive syntax. Golang is basically a functional language, rather than a strict OOP, designed for high performance (as compiled to a native machine code), without the bother to deal with thread synchronization.

Our Perspective is to define distinct aspects in which GoLang may give better performance than the ‘standard’ stack of Java / Python / NodeJS, in backend development.

With the growing list of vendors providing “vault”-like services from Hashicorp’s Atlas offering to AWS’s secret manager to GKE’s cloud KMS and project like Vault which have an Operator which provides “vaults” to deployments makes this trend a direct continuation of how Security / No passwords in Git trens strengthens.

Hasicorp Vault is one of the tools which are definitely a keep from “Secrets aaS” trend which varies from cloud platforms like AWS/GCP to self-hosted / As a Secrets/Tokens Vault.

The NGINX controller is built around the Kubernetes Ingress resource that uses ConfigMaps to store the NGINX configuration.

NGINX is in the route path of many applications, which all vary in needs. From basic ones like routing (a.k.a proxying) and authentication, to advanced services like, load balancing SSL termination and integrations with other projects like cert-manager (kube-lego) and many more.

NGINX started off as the faster Apche alternative and it seems to the most popular ingress used out there among out clientel, and gaining popularity due to its seamless integration with kuberenetes.

The active seeking to determine a foe’s intentions by collecting and gathering information about an enemy’s composition and capabilities along with pertinent environmental conditions.

• static code analysis [ checkmarks as an example ]
• obfuscation
• code fuzzing

Polyglot programming is the practice of writing code in multiple languages to capture additional functionality and efficiency not available in a single language. The use of domain specific languages (DSLs) has become a standard practice for enterprise application development. For example, a mobile development team might employ Java, JavaScript and HTML5 to create a fully functional application. Other DSLs such as SQL (for data queries), XML (embedded configuration) and CSS (document formatting) are often built into enterprise applications as well. One developer may be proficient in multiple languages, or a team with varying language skills may work together to perform polyglot programming.

Gradle is a build tool. Following a build-by-convention approach, Gradle allows for declaratively modelling your problem domain using a powerful and expressive domain-specific language (DSL) implemented in Groovy instead of XML. Because Gradle is a JVM native, it allows you to write custom logic in the language you’re most comfortable with, be it Java or Groovy. it too provides a powerful dependency management.

Although still many more developers use Maven (~60%) Gradle is a powerful contender/ally which needs to be considered when making the decision on which build tool to use.

The Android Studio build system is based on Gradle, and the Android Gradle plugin adds several features that are specific to building Android apps.

The selection, specification and application of security controls for a system.

• static code analysis [ checkmarks as an example ]
• obfuscation
• code fuzzing

Identity and access management (IAM) is, in computer security, the security and business discipline that “enables the right individuals to access the right resources at the right times and for the right reasons”. It addresses the need to ensure appropriate access to resources across increasingly heterogeneous technology environments and to meet increasingly rigorous compliance requirements.

• FreeIPA
• Cognito
• OpenLdap
• Octa
• JumpCloud
• Auth0

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Docker Compose is used for running multi-container applications on single machine or on multiple machines with Swarm mostly in Dev and QA environments, and in some places in Production as well.

The cluster management and orchestration features embedded in the Docker Engine

Continue Using it

• We identify a large client base in the process of either living within the limits of the swarm or moving to Kubernetes
• The Docker platform is getting support for Kubernetes. This means that developers and operators can build apps with Docker and seamlessly test and deploy them using both Docker Swarm and Kubernetes.
• looks like in most of the case when you going to production with container cluster it’s will be either swarm or k8s

TICK (Telegraf InfluxDB Chronograf Kapacitor) Modern Time Series Platform, designed from the ground up to handle metrics and events. InfluxData’s products are based on an open source core. This open source core consists of the projects—Telegraf, InfluxDB, Chronograf, and Kapacitor; collectively called the TICK Stack.

Telegraf is a plugin-driven server agent for collecting and reporting metrics.

• Telegraf has plugins or integrations to source a variety of metrics directly from the system it’s running on, to pull metrics from third party APIs, or even to listen for metrics via a StatsD and Kafka consumer services. It also has output plugins to send metrics to a variety of other datastores, services, and message queues, including InfluxDB, Graphite, OpenTSDB, Datadog, Librato, Kafka, MQTT, NSQ, and many others.

InfluxDB - Is a Time Series Database built from the ground up to handle high write & query loads.

• InfluxDB is a custom high performance datastore written specifically for timestamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics. Conserve space on your machine by configuring InfluxDB to keep data for a defined length of time, and automatically expiring and deleting any unwanted data from the system. - InfluxDB also offers a SQL-like query language for interacting with data.

Chronograf is the administrative user interface and visualization engine of the platform.

• It makes the monitoring and alerting for your infrastructure easy to setup and maintain. It is simple to use and includes templates and libraries to allow you to rapidly build dashboards with real-time visualizations of your data and to easily create alerting and automation rules.

Kapacitor is a native data processing engine. It can process both stream and batch data from InfluxDB.

• Kapacitor lets you plug in your own custom logic or user-defined functions to process alerts with dynamic thresholds, match metrics for patterns, compute statistical anomalies, and perform specific actions based on these alerts like dynamic load rebalancing. Kapacitor integrates with HipChat, OpsGenie, Alerta, Sensu, PagerDuty, Slack, and more.

Jenkins is the widely recognized as the de-facto standard solution for implementing CI and even CD, but there are other leading alternatives:

• Travis - the GitHub hosted CI
• CircleCI - a free cloud-based system (it also have an on-premise option)
• Gitlab-CI - part of GitLab platform
• TeamCity - JetBrain CI/CD server

Kubernetes is an open-source system for automating deployment, scaling and management of containerized application in a cluster.

Kubernetes uses a set of APIs and yaml/json based configuration language to define and maintain all aspects of containers running on a cluster. Including networking, service discovery, proxying and load balancing, horizontal and vertical scaling, security and more. Kubernetes as a service is part of all major cloud providers offering and there are some projects that can deploy Kubernetes cluster automatically on almost every compute environment. Kubernetes introduce the concept of a POD - a set of one or more containers that deployed as a single unit. (Same node, same namespace and same network configuration) PODs can be thought of as lightweight servers constructs from container images. PODs can be deployed using controllers that define the behavior of the POD in the cluster. Commonly used controllers are the ‘Deployment’ controller that define a replica-set to make sure a given number of POD instances will be available in any moment of time. And ‘DaemonSet’ controller that deploy one POD per running worker node in the cluster. Services, running as PODs can be exposed, internally to the cluster or externally to the world, via ‘Service’ configuration object that acts as reverse proxy and simple load balancer that provide a single endpoint for the service. All configuration objects (PODS, Controllers, Services etc…) are loosely coupled via tags and selectors that makes the infrastructure flexible and configurable.

Serverless computing is a cloud computing execution model in which the cloud provider dynamically manages the allocation of machine resources. Serverless is a form of utility computing which helps provide complex infrastructure and security requirements as a part of managed services these services are managed by the provider hence the name serverless { less management of servers }. Serverless should not be confused with the serverless framework …

Serverless framework Serverless.com is a toolkit for deploying and operating serverless architectures, serverless utilizes the different cloud provider’s function and api-gw api’s to expose backend services.

Highlights:

• Integrates with main cloud providers - AWS, GCP, Azure
• Has Kubeless plugin
• Yaml definition of both API gateway and lambdas
• Utility to test functions locally using a local docker daemon

Grafana’s motto is “The analytics platform for all your metrics”, in the past ~5 years or so Granfana has done just that, becoming a centralized hub of data processing and visualization. Grafana has a big Datasource variety supported, enabling processing and real-time visualization of time series data originating from many data sources simultaneously. From traditional databases such as MySql and Postgresql and to time series databases such as OpenTSDB, Graphite, Prometheus, Elasticsearch, and others.

Grafana also provides an extensible plugin interface adding the ability to enrich both visualizations with plugins or custom “data sources”.

The Suger Coating of Grafana is the Grafana community website, which maintains a centralized hub for hosting plugins and dashboard which the community can download & manage via source control. In many cases, there is either a ready dashboard for your use case or at least a good starting point.

Grafana also supports an HA installation method, authentication [Basic/LDAP/OAuth/etc] & authorization schemes and organization management.

Read more

Jenkins Pipeline (or simply “Pipeline” with a capital “P”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

A continuous delivery (CD) pipeline is an automated expression of your process for getting software from version control right through to your users and customers. Every change to your software (committed in source control) goes through a complex process on its way to being released. This process involves building the software in a reliable and repeatable manner, as well as progressing the built software (called a “build”) through multiple stages of testing and deployment.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code” via the Pipeline domain-specific language (DSL) syntax.

The definition of a Jenkins Pipeline is written into a text file (called a Jenkinsfile) which in turn can be committed to a project’s source control repository. This is the foundation of “Pipeline-as-code”; treating the CD pipeline a part of the application to be versioned and reviewed like any other code.

Out of the 2 options of ‘scripted Pipeline’ and ‘Declarative Pipeline’, it is mostly recommended to use the ‘Declarative Pipeline’ option.

Jenkins pipeline (as part of Jenkins) is in use by wide range of our customers and is currently the preferred orchestration tool offered by Tikal.

Service discovery is the practice of automatic detection of devices and services offered/hosted by a computer network. Service discovery has become the heart of SOA and MSA SW architectures and in the world of containers has become a method of establishing a distributed and dynamic configuration for containers/microservices.

Service discovery providers such as etcd, consul, cloud metadata services such as aws / gcp / azure are also considered authoritative service discovery providers. Many OpenSource project HA configures rely on service discovery in order to establish clusters, among many other applications like Kubernetes, Swarm, Mesos, RabbitMQ & Prometheus comes with a built-in support for service discovery which makes their integration seamless.

Service discovery comes in many forms and shapes and we will most definitely be seeing more and more providers/integration in the coming future.

A Time Series Database (TSDB) is a database optimized for time-stamped or time series data. Time series are simply measurements or events that are tracked, monitored, downsampled, and aggregated over time. This could be server metrics, application performance monitoring, network data, sensor data, events, clicks, trades in a market, and many other types of analytics data. The key difference with time series data from regular data is that you’re always asking questions about it over time.

There are quite a few popular Open Source Time Series Databases which are widely used some of them are listed below:

1. DalmatinerDB
2. InfluxDB
3. Prometheus
4. Riak TS
5. OpenTSDB
6. KairosDB
7. Elasticsearch
8. Druid
9. Blueflood
10. Graphite (Whisper)

A great source to read here

During development you generate a fair amount of different artifacts. These might include:

• The source code
• The compiled application
• A deployable package
• Documentation and potentially others as well

While you could use a source control system to store all of them, it’s usually massively inefficient, as source control systems are usually designed to handle text based files, and not binary files. You might be able to use them as a simple storage mechanism, if most of your releases are text based, and you don’t have to store a lot of binary data.

Artifact repositories however are designed to store all kinds of files, including binary ones. This includes anything from zipped up source codes, to build results, to things like docker containers as well. Also, they usually not only store these artifacts but also help manage them using various additional functions

Common artifact-repositories are:

• Artifactory, Nexus - mostly for code-build-artifacts (maven-based)
• Nuget - for .NET packages
• ECR (Amazon Elastic Container Registry), Docker-hub - for Docker-based container/images storage.

ELK is short for Elasticsearch Logstash Kibana.

Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents

Logstash is one of the pioneers of log forwarding and processing and has become a standard for shipping logs to Elasticsearch (and other desinations).

Kibana lets you visualize your Elasticsearch data and Discover events, create and share dashboards and more.

In the past 3 years or so we might also see the “L” replaced by “F” representing “the Beats from Elasticsearch”, fluentd and others.

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Docker Compose is used for running multi-container applications on single machine or on multiple machines with Swarm mostly in Dev and QA environments, and in some places in Production as well.

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Prometheus uses a pull model to scrape endpoint which exposes metric data in form of HTTP/s via standardized exporters such as JMX, MySql, advisor and node_exporter or via Prometheus client libraries implementations within common frameworks and SW languages.

Prometheus consists of a centralized server (written in go) which implements:

1. A time series database,
2. An endpoint collection mechanism based on common service discovery providers varying from all common cloud provider api’s, service discovery services such as etcd / consul and of course docker and kubernetes and even more
3. A web interface which also provides the PromQL interface for the time series data queries.

Additional (Optional) components:

1. push gateway - a server which provides monitoring capabilities for short-term/stateless services to push metrics (over the default pull method)
2. node-exporter - the official os level metric exporter
3. JMX exporter - java application monitoring + common JVM related metrics
4. Client libraries for all common languages nodejs, go, java, python …
5. many many more

Helm, short for Helm Charts which help manage Kubernetes applications — Helm Charts helps you define, install, and upgrade even the most complex Kubernetes applications.

Charts are easy to create, version, share, and publish. There are Helm Registry offerings from both Quay.io and Artifactory. HELM helps to manage Services, Deployment, Secrets and all part related to Kubernetes ready applications. Umbrella chart , To install a complex application with helm — There is something called umbrella chart Umbrella chart is chart with dependency (requirements.yaml) on other charts. and value.yaml (override charts values) To install umbrella-chart you run “helm dep update” will download depended chart , then “helm install” or “helm upgrade” will install the charts

Ansible is a simple automation language that can perfectly describe an IT application infrastructure. It’s easy-to-learn, self-documenting, and doesn’t require a grad-level computer science degree to read. Ansible has agentless architecture, it is used for Configuration management, it orchestrates the app lifecycle and deployment flows.

Ansible is the most popular deployment tool we use.

Apache Groovy is an object-oriented programming language for the Java platform. It is a dynamic language with features similar to those of Python, Ruby, Perl, and Smalltalk. It can be used as a scripting language for the Java Platform, is dynamically compiled to Java virtual machine (JVM) bytecode, and interoperates with other Java code and libraries. Groovy uses a Java-like curly-bracket syntax. Most Java code is also syntactically valid Groovy, although semantics may be different.

We use Groovy mostly for programming Jenkins-code (Jenkinsfile and Jenkins shared libraries) which are Groovy-based.

We use Nodej.s in DevOps -

• Write small tools or in serverless framework, lambda tasks (gluing processes)
• But I wouldn’t say it’s the main language in “devops domain”

Python is a powerful high-level, interpreted, open source, object-oriented programming language, portable, extensible and embeddable with simple syntax and large standard libraries to solve common tasks. Python is a general-purpose language. It has wide range of applications from Web development (like: Django and Bottle), scientific and mathematical computing (Orange, SymPy, NumPy) to desktop graphical user Interfaces (Pygame, Panda3D).

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is the upstream project for Tower, a commercial derivative of AWX.

Terraform is a provisioning tool (as opposed to a Configuration Management tool), working with Immutable Infrastructures, using a declarative language, is masterless and agentless

Being a popular tool, from a well established vendor, HashiCorp we should certainly keep offering Terraform as an alternative to our customers

Terraform has recently announced support of Kubernetes, going to show that HashiCorp is on it’s feet and adapting to changes

Subversion. The good old successor of the mythological CVS, represent the centralised source control tools in the OSS world. It shines in an environment of large repos and centralized management. In the age of microservices from one hand, and a good selection of github-like solutions on the other. There is no point to keep using it.

OpenStack is a free and open-source software platform for cloud computing, mostly deployed as infrastructure-as-a-service (IaaS), whereby virtual servers and other resources are made available to customers. The software platform consists of interrelated components that control diverse, multi-vendor hardware pools of processing, storage, and networking resources throughout a data center. Users either manage it through a web-based dashboard, through command-line tools, or through RESTful web services.

On top of OpenStack core services, that provide the basic cloud functionality of managing compute, storage and networking resources. OpenStack project contain a set of services (sometimes named ‘The big tenet’) that provide a lot of complementary functionality such as shared storage, databases, monitoring, orchestrating and much more.

It seems like the utilization of Ruby as a software development language which was very popular in the DevOps movement mainly with tools like Chef and Puppet Logstash Fluentd and a lot of scripting and utilities around ruby and ruby on rails application lifecycle such as Capistrano, has taken a punch in favor of python go and javascript. With the rise of popularity of these langs & frameworks, we see less and less use of ruby.

We assume that as there will always be something written in ruby but it will most definitely not be the language of choice when we are required to develop a utility / micro-app.

Cloud Spanner is a fully managed, relational database service that offers transactional consistency at global scale, with schemas. This combination of availability and consistency over the wide area is generally considered impossible due to the CAP Theorem. This is a claim that still needs to be checked if it really lives up to it

gRPC (Google Remote Procedure Call) is an open source remote procedure call (RPC) system initially developed at Google. It uses HTTP/2 for transport, Protocol Buffers as the interface description language, and provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, cancellation and timeouts. It generates cross-platform client and server bindings for many languages (C++, Java, Python Node, etc).

In this new agile world, many people question the role of architecture. And certainly the pre-planned architecture vision couldn’t fit in with modern dynamism. But there is another approach to architecture, one that embraces change in the agile manner. In this view architecture is an constant effort, one that works closely with programming so that architecture can react both to changing requirements but also to feedback from programming. It is called “Evolutionary Architecture”, to highlight that while changes are unpredictable, architecture can still move in a better direction.

AWS was the first provider to offer functions as a service already in November 2014. AWS initially released AWS lambda as an event-driven provisioning/operations and took just under 3 years to become the standard name of serverless and FaaS offerings.

AWS, as it’s competitors, offer Lambdas (a.k.a functions), as a complimentary to their BaaS offerings, stitching together services such as:

• Incognito
• S3
• CloudFormation
• DynamoDB
• RDS and many more

These integrations alongside Lambda’s “infinite” scalability and it’s newly introduced (at the time) “Price per 100ms” made it very popular among both startups achieving their MVP and enterprises wishing to scale out or experiment with Serverless and Micro Services Architectures.

AWS lambda provides many organizations the ability to write functions in a variety of programming languages and integrates well with many frameworks and other IaaS/PaaS/BaaS services.

More about AWS Lamda here

Ethereum is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third-party interference.

These apps run on a custom built blockchain, an enormously powerful shared global infrastructure that can move value around and represent the ownership of property.

This enables developers to create markets, store registries of debts or promises, move funds in accordance with instructions given long in the past (like a will or a futures contract) and many other things that have not been invented yet, all without a middleman or counter party risk.

Cloud Native Solution With the broad adoption of SOA and Micro Service Architectures and many IaaS, PaaS offerings one of the “lessons learned” is that you need to be able to harness your infrastructure and start treating it in a standardized way.

So What is Cloud Native? The Cloud Native Computing Foundation (CNCF) describes it as “distributed systems capable of scaling to tens of thousands of self healing multi-tenant nodes”. That’s a “how” (distributed systems) and a “why” (high scaleability and automated resilience).

All the above in a complex / polyglot world will only be possible with standards and tooling to enable all these moving parts to suit small to large scale microservice based applications - This is what the Cloud Native Movement and the CNCF will promote in the years to come.

The following principles drive Cloud Native solutions:

• Treat your own / cloud / hybrid infrastructure as-a-service: run on servers that can be flexibly provisioned on demand.

• Microservices architecture: individual components are small, loosely coupled.

• Automated deployments and continuous ly integrated and test -> replace manual tasks with scripts or code.

• Containerize: package processes with their dependencies making them easy to test, move and deploy.

• Orchestrate: use standard / commonly used / battle tested orchestration tools.

Cloud Service Providers As of today, the cloud service providers play a major role in all startup companies and even in big companies. This get the programmers to the point where knowing Linux and ops utilities is not enough. Developers are expected to be experts in the different cloud provider in order to succeed with their tasks.

Each cloud provider comes with new platforms, techniques, API’s and databases.

Docker has revolutionised the world. However, business can stale development, and we believe this year we’ll see less usage of Docker and more of other alternatives:

• Distroless Docker Images (Google)
• rkt (CoreOS)
• Makisu (Uber)

KSQL is the streaming SQL engine for Apache Kafka®. It provides an easy-to-use yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python. KSQL is scalable, elastic, fault-tolerant, and real-time. It supports a wide range of streaming operations, including data filtering, transformations, aggregations, joins, windowing and sessionization.

So what is a Microframework ? A microframework is a minimalist web framework that is meant to be lightweight and fast, thus making the development of applications much easier and faster.

Here is a list of possible Microframeworks:

• Sparkjava
• javalite
• Dropwizard
• Vert.x
• light4j
• jooby

So what is a Microframework ? A microframework is a minimalist web framework that is meant to be lightweight and fast, thus making the development of applications much easier and faster.

Here is a list of possible Microframeworks:

• Sparkjava
• Vert.x
• Ktor
• Javalin

MTL is a subfield of machine learning in which multiple learning tasks are solved at the same time, while utilizing similarities and differences across tasks, MTL can result in improved learning efficiency and prediction accuracy for task specific models, some say that especially when some of the tasks datasets are undersampled.

Siamese networks are NN architecture which contains two or more identical sub networks sharing the same topology and parameters, Siamese networks are popular in the task of finding similarity estimation for complex objects and distributions, e.g. separating all the images of a person from all the images of all the other people.

Generative adversarial networks are a class of AI algorithms, used in unsupervised learning, implemented by two NNs competing with each other in a zero some game, one training do generate quality augmented data, while the other training to differentiate the real data from the fake data, creating an iterative process of improvement, while In practice, GANs are used mostly for data augmentation which is important for supervised learning tasks, there are many more possible uses for GANs in many other fields.

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data.

Deep learning is part of a broader family of machine learning methods based on learning data representations, as opposed to task-specific algorithms. Learning can be supervised, semi-supervised or unsupervised. Deep learning architectures such as deep neural networks, deep belief networks and recurrent neural networks have been applied to fields including computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, bioinformatics, drug design, medical image analysis, material inspection and board game programs, where they have produced results comparable to and in some cases superior to human experts. Many of the nowadays companies, startups and large corporates, incorporate Deep Learning capabilities in their product. At Tikal, we should enhance our knowledge in deep learning to provide our customers with methods and tools to add deep learning capabilities to their portfolio.

Consent The conditions for consent have been strengthened, and companies are no longer able to use long illegible terms and conditions full of legalese. The request for consent must be given in an intelligible and easily accessible form, with the purpose for data processing attached to that consent. Consent must be clear and distinguishable from other matters and provided in an intelligible and easily accessible form, using clear and plain language. It must be as easy to withdraw consent as it is to give it.​

The Problem Certain technologies like blockchain and kafka do not allow or is very difficult to remove old data from the system. Therefore we must deploy different types of data architects so that we can address the GDPR issues, such as encryption with option of deleting decryption key

OpenJDK is an open source implementation of the Java Standard Edition platform with contribution from Oracle and open Java community. It is distributed under the GPL License. As of Java 11 Oracle has changed their license to a proprietary one. Usage of the Oracle JDK is not free any more. OracleJDK and the OpenJDK are basically the same. More than that; both JDKs are built from the same source code and undergo the same rigorous tests and testing cycles. Previous commercial products which were solely in the Oracle JDK are now included in the Open JDK as well. So fundamentally, both JDKs are the same, so why pay Oracle for their JDK if we can have the same JDK for free ?

Istio is the most talked about Service Mesh which is inversion 1.0.5 as of day and a growing adoption rate from self-manages to Istio as a service.

To clarify, the term service mesh is used to describe the network of microservices that make up applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery __, __load balancing, failure recovery, __metrics __, and __monitoring __.

A service mesh also often has more complex operational requirements, like __A/B testing __, __canary releases __, __rate limiting __, __access control __, and __end-to-end authentication __.

The Service Mesh, in general, is the enabler of monolith apps to co-exist alongside one another until a full transition to microservices is done.

Istio.io -> Istio’s diverse feature set lets you successfully, and efficiently, run a distributed microservice architecture, and provides a uniform way to secure, connect, and monitor microservices - implementing all the technics and methodologies mentioned above (which used to be part of the Backend).

GraalVM is a research project developed by Oracle Labs and already in production at Twitter. It is a universal virtual machine for running applications written in JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, Clojure, and LLVM-based languages such as C and C++.

GraalVM removes the isolation between programming languages and enables interoperability in a shared runtime. It can run either standalone or in the context of OpenJDK, Node.js, Oracle Database, or MySQL.

GraalVM is based on the Graal JIT compiler which is currently, as of Java 11, being used as an experimental JIT compiler for the JVM instead of the famous c2 JIT compiler, but only for those enabling this feature.

GraalVM is Polyglot allowing you even to write polyglot applications. Native images compiled with GraalVM ahead-of-time improve the startup time and reduce the memory footprint of JVM-based applications.

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

• GraphQL queries always return predictable results.
• While typical REST APIs require loading from multiple URLs, GraphQL APIs get all the data your app needs in a single request.
• GraphQL uses types to ensure Apps only ask for what’s possible and provide clear and helpful errors.

GraphQL link

The Apollo Platform is a family of technologies you can incrementally add to your stack: Apollo Client to connect data to your UI, Apollo Engine for infrastructure and tooling, and Apollo Server to translate your REST API and backends into a GraphQL schema.

• Apollo Client Bind data to your UI with the ultra-flexible, community-driven GraphQL client for React, JavaScript, and native platforms.
• ** Apollo Server** Translate your existing REST APIs and backends into GraphQL with this powerful set of tools for buliding GraphQL APIs.
• Apollo Engine The GraphQL gateway that provides essential features including caching, performance tracing, and error tracking.

Apollo link

Apache Mxnet is a open source deep learning infrastructure growing in popularity, it’s biggest and most significant strength is scale, it performs beautifully over multiple GPUs and in some cases over multiple machines.

Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka clusters. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka’s server-side cluster technology.

The library allows developing stateful stream-processing applications that are scalable, elastic, and fully fault-tolerant. The main API is a stream-processing DSL that offers high-level operators like filter, map, grouping, windowing, aggregation, joins, and the notion of tables

It will interesting to see how kafka streams will compete in the spark and other streaming worlds

Kaggle is a platform for one of the largest community of data scientists and machine learning engineers. On Kaggle you can find and investigate high-quality data sets, explore and build models with that data on web-based data science environment, connect and learn with other data scientists and machine learning engineer, and participate in enterprise awarding ML competitions. We, at Tikal, can use Kaggle to connect with the community, learn techniques, find data sets and models, and share our knowledge.

MLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, and deployment. Maintained and development by Databricks (Apache Spark maintainer), it offers the following components:

• MLFlow Tracking - ability to record and query ML experiments
• MLFlow Projects - packaging format for reproducible runs on any platform.
• MLFlow Models - General format for sending models to diverse deployment tools

Mastering this tool in Tikal provides us with production-level capabilities we can offer to customers.

Keras is a high-level neural networks API, build on top of TensorFlow, Theano or CNTK engines. It aims at fast experimenting in various topologies of neural networks. As opposed to the complicated API of TensorFlow, Keras is easy to understand, allowing straight-forward code of neural network construction. These characteristics make Keras a first choice library when building networks such as DNN, CNN, and RNN. For Tikal, mastering this library allows us implementing deep neural network efficiently for customers.

Airflow is today the most appreciated workflow engine in the distributed world. It allow building modular workflow with high scale, triggering, forking and joining and many more features. In addition, a good GUI that helps auditing the workflow.

This framework is run on Python, which makes it very attractive to devops groups.

The ramp up seems easy in the beginning, but requires deep knowledge on later phases. the solution is also based on other Python frameworks and databases for the queue management and databases, and for the web server. Celery, RabbitMQ, MySQL or Postgres

Some drawbacks of this solution is the fact that all configuration is deployed and Python code. This may cause one bug to halt the whole of the server. Another problem with Python is the different versions that may be installed and updated on the different servers and may not be managed properly and cause bugs in Airflow itself.

GO is A programming language introduced by Google in 2009. It is a compiled and strongly typed language similar to C, but bringing a much more intuitive syntax. Golang is basically a functional language, rather than a strict OOP, designed for high performance (as compiled to a native machine code), without the bother to deal with thread synchronization.

Our Perspective is to define distinct aspects in which GoLang may give better performance than the ‘standard’ stack of Java / Python / NodeJS, in backend development.

Kotlin is a statically typed programming language that runs on JVM (Java Virtual Machine). Development lead Andrey Breslav has said that Kotlin is designed to be an industrial-strength object-oriented language, and a “better language” than Java, but still be fully interoperable with Java code, allowing companies to make a gradual migration from Java to Kotlin.

At Google I/O 2017, Google announced first-class support for Kotlin on Android. As of Android Studio 3.0 (October 2017) Kotlin is a fully supported programming language by Google on the Android Operating System, and is directly included in the Android Studio 3.0 IDE package as an alternative to the standard Java compiler.

Kotlin is now an official language on Android. It’s expressive, concise, and powerful. Best of all, it’s interoperable with our existing Android languages and runtime.

Besides the stylistic advantages over Java, Kotlin has new and exciting features that are offering performance and stability improvements over Java code, such as Kotlin coroutines, Rx-like collection iteration modifiers, null-safety, operator overloading, etc.

Kinesis - Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more This tool serves as an IDE for the majority of data scientists. It is the inspiration for big web-based IDE projects such as Google DataLab and Apache Zeppelin. As part of the polyglot effort in the Backend team, and our route towards Machine Learning, this tool should be on our radar for daily uses.

Scikit-learn is a free software machine learning library for the Python programming language. It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy. It contains tools for data manipulation necessary for machine learning project and even if one is using other Python-based libraries such as Keras or TensorFlow, she probably finds herself using one of the tools provided by Scikit learn. As Tikal is advancing towards Machine Learning, Scikit learn is one of the basic tools to master.

Apache Zeppelin is a notebook for managing and visualizing Data and Data-lake. With Apache Zeppelin one can perform Data Ingestion, Data Discovery, Data Analytics and also have Data Visualization & Collaboration.

Apache Zeppelin is a Java runtime and supports the following languages and frameworks: Scala as Apache Spark, Python, JDBC, Markdown and Shell. It also supports many plugins

The results of the queries can be shown in nice built-in visuals like graphs, pie charts and more.

Apache Zeppelin supports many additional interpreters that can be loaded or integrated using Helium, Among those interpreters are for example: Cassandra, elasticsearch, flink, ignite, bean, angular and many more

Polyglot programming is the practice of writing code in multiple languages to capture additional functionality and efficiency not available in a single language. The use of domain specific languages (DSLs) has become a standard practice for enterprise application development. For example, a mobile development team might employ Java, JavaScript and HTML5 to create a fully functional application. Other DSLs such as SQL (for data queries), XML (embedded configuration) and CSS (document formatting) are often built into enterprise applications as well. One developer may be proficient in multiple languages, or a team with varying language skills may work together to perform polyglot programming.

Apart from the old monolithic paradigm of software engineering, the Micro-Service-Architecture (MSA) or Service-Oriented-Architecture (SOA) orients software engineers to break apart their software design to different self-contained, independent services, maintained by small teams of no more than 4 engineers, on a rather small code bases.

This new approach make it possible to design logic parts of the entire application as more generic services, for a future use of other clients.

For more info and best practices about MSA design, refer here: https://12factor.net/

GraphQL gains popularity due to its ability to have query language over HTTP along with getting results from multiple resources without the hassle of writing multiple requests and combining the results.

In case you need to have responses from multiple results, the GraphQL API enables you to achieve the responses as one, faster than in REST API.

Google BigQuery is considered as a Data Warehouse as a Service. Unlike traditional Data Warehouse that were based on RDBMS, Google BigQuery is based on large read-only datasets. Originally, with no indexes.

Google took the advantage of Dremel, their big data solution, and implemented it as a service. Originally, the aim was to provide big data analytics as a service that provide very high performance by leveraging the Google cloud. Today, Google brand it more as a Data Warehouse as a service that can replace Data Warehouses that were managed on premise.

The big benefit of BigQuery over RDBMS, is that you do not need to know the underlying architecture or use a DBA in order to use it. Just upload your data and start using it. But the main attraction of BigQuery is that it is a Big Data solution on the cloud. That means that you can scale your data without the need to worry about performance.

High Level Architecture

The data is stored in columnar format in Capacitor - a google technology of columnar storage. The storage file system is Google new distributed file system: Colossus. Colossus allows splitting of the data into multiple partitions to enable fast parallel read while Capacitor improves throughput. The two technologies together bring very good performance of returning query results within seconds.

The processing engine is Dremel, Google’s execution engine for running SQL queries over big data. the processing runs in a distributed cluster managed by Google called Borg.

Functional Programming is a programming paradigm that become more popular in the recent years, it has many benefits for even-driven architectures and other state/contexts handling/processing problems.

Blockchain is probably the hottest ever buzzword in tech. Hotter than AI, hotter than VR. Surely hotter than Kubernetes. $6B were raised in ICOs by blockchain related startups in 2017, this is more than the entire VC money invested in early stage startups.

Is blockchain just a buzz? Chris Dixon who is a VC that does believe in Blockchain says: [blockchain] is probably the largest and smartest software engineering organization in the world. There are probably 20,000 programmers, if not more, working on this stuff. There’s a lot of really really smart people. I spend a lot of time with them and I’m just constantly impressed. Do you want to bet against those 20,000 really smart engineers who are like super passionate about this and working on it all the time? I wouldn’t want to bet against them.

The $6B is going to be used mainly in R&D in the coming years, and I see 3 main opportunities for Tikal here:

• Support development of infrastructure projects: Most of the bigger ICOs are infrastructure projects. i.e blockchains and technologies that will enable future decentralized apps development. HyperLedger seems to be taking a major role in this emerging world.
• Support development of ethereum dapps: Ethereum is a rather mature blockchain and very popular. It already has vast dev tools such as Truffle, Ganache, OpenZeppelin, MetaMask and many more. This is the 1st blockchain were demand for expertise in such tools for developing dapps will be required
• Support usage of edge clouds based on blockchains: two major projects are Golem and Dfinity who aim to decentralize the cloud. It is not far fetched to believe that they will be major competition for AWS, Google Cloud and Azure and expertise in them will be required (DevOps)

Machine learning on mobile devices is offering an array of new and unique user experiences for problems that are close to data people are working on. Experiences that were impossible before, like OCR, image recognition, machine translation, speech recognition, etc, were in the realm of Science Fiction, but they are being brought into our daily lives. Thanks to machine learning on mobile and IoT devices, universal translators from Star Trek, Babel Fish from Hitchhiker’s Guide to the Galaxy, “neural net processor, a learning computer” from Terminator 2, and more, are not a figments of writers’s imagination, but actual products that you can buy, or are in development stages.

TensorFlow is a opensource library for dataflow programming, often used in deep learning. It was developed for internal use by Google Brain team. It was released to the public, and in 2017 mobile/lite version was released that is targeting mobile and embedded/IoT platforms, such as Android, iOS, Rasberry Pi, etc.

Reactive Programming is a paradigm that changes the direction of the flow. Instead of creating threads and blocking for communications, we create pipelines where data is pass though and each layer can react to the data received. The indirection changes the code flow from blocked to async.

Over time the reactive extensions has become a standard for adding higher level functionality on the stream of data while being agnostic to the data itself. The extensions are based on the observable pattern, and allow for cancellation of data process in stream

Kubernetes is an open-source system for automating deployment, scaling and management of containerized application in a cluster.

Kubernetes uses a set of APIs and yaml/json based configuration language to define and maintain all aspects of containers running on a cluster. Including networking, service discovery, proxying and load balancing, horizontal and vertical scaling, security and more. Kubernetes as a service is part of all major cloud providers offering and there are some projects that can deploy Kubernetes cluster automatically on almost every compute environment. Kubernetes introduce the concept of a POD - a set of one or more containers that deployed as a single unit. (Same node, same namespace and same network configuration) PODs can be thought of as lightweight servers constructs from container images. PODs can be deployed using controllers that define the behavior of the POD in the cluster. Commonly used controllers are the ‘Deployment’ controller that define a replica-set to make sure a given number of POD instances will be available in any moment of time. And ‘DaemonSet’ controller that deploy one POD per running worker node in the cluster. Services, running as PODs can be exposed, internally to the cluster or externally to the world, via ‘Service’ configuration object that acts as reverse proxy and simple load balancer that provide a single endpoint for the service. All configuration objects (PODS, Controllers, Services etc…) are loosely coupled via tags and selectors that makes the infrastructure flexible and configurable.

The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala. Flink executes arbitrary dataflow programs in a data-parallel and pipelined manner. Flink’s pipelined runtime system enables the execution of bulk/batch and stream processing programs. Furthermore, Flink’s runtime supports the execution of iterative algorithms natively. Flink provides a high-throughput, low-latency streaming engine as well as support for event-time processing and state management. Flink applications are fault-tolerant in the event of machine failure and support exactly-once semantics

Apache Spark is an open-source cluster-computing framework. It is used mostly to process big data and mostly used in clouds. Apache Spark has a Core package and provides 4 more libraries: SparkSQL, Spark Streaming, Spark MLlib and GraphX

We use Apache Spark as a preferred solution for Big Data processing because it has matured and has penetrated the market with big community.

Apache Kafka is a distributed streaming platform.

Kafka has three key capabilities:

• Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
• Store streams of records in a fault-tolerant durable way.
• Process streams of records as they occur.

Kafka is generally used for two broad classes of applications:

• Building real-time streaming data pipelines that reliably get data between systems or applications
• Building real-time streaming applications that transform or react to the streams of data

A more comprehensive definition could be read here

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Beam supplies an sdk in different languages (java, python, scala) to express the flow and processing of the data). The sdk then applies the runner according to the chosen platform (spark, flink, gear…). Beam aims to lead the industry on the proper way to deal with batch and streaming in the same pipeline. In addition Beam addresses issues like process time vs event time, and handling late data.

Spark MLlib is a Machine Learning library that comes on top of Apache Spark. It is the best solution for applying Machine Learning algorithms on big amounts of data that is stored in distributed file systems, like HDFS or in the cloud.

we promote Spark MLlib as a library for Machine Learning as it is part of the ecosystem of Spark and is most suitable for Big Data.

Serverless computing is a cloud computing execution model in which the cloud provider dynamically manages the allocation of machine resources. Serverless is a form of utility computing which helps provide complex infrastructure and security requirements as a part of managed services these services are managed by the provider hence the name serverless { less management of servers }. Serverless should not be confused with the serverless framework …

Serverless framework Serverless.com is a toolkit for deploying and operating serverless architectures, serverless utilizes the different cloud provider’s function and api-gw api’s to expose backend services.

Highlights:

• Integrates with main cloud providers - AWS, GCP, Azure
• Has Kubeless plugin
• Yaml definition of both API gateway and lambdas
• Utility to test functions locally using a local docker daemon

Akka is a free and open-source toolkit and runtime simplifying the construction of concurrent and distributed applications on the JVM. Akka supports multiple programming models for concurrency, but it emphasizes actor-based concurrency, with inspiration drawn from Erlang

Vert.x is a toolkit to build distributed reactive systems on the top of the Java Virtual Machine using an asynchronous and non-blocking development model. As a toolkit, Vert.x can be used in many contexts: in a standalone application or embedded in a Spring application. Vert.x and its ecosystem are just jar files used as any other library: just place them in your classpath and you are done. However, as Vert.x is a toolkit, it does not provide an all-in-one solution, but provides the building blocks to build your own solution.

Spring 5 introduces Reactive programming into its framework, in the core and web libraries. Also introducing WebFlux as a reactive alternative to WebMVC. This allows developing a reactive, i.e. asynchronous and non-blocking, Spring application. Mastering Spring 5, will:

1. Allow adding reactive capabilities to existing Spring applications
2. Explore more reactive libraries and options, when facing the need to choose a relevant framework.
3. Allow backend developers to advance to working with Spring 5, and fully understand it’s capabilities, advantages and disadvantages.

Overview Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions. Flask has become the standard web application for many open source projects such as dataflow, and should be adopted in other future projects Flask and Django are two of the most popular web frameworks for Python. Main contrasts: Flask provides simplicity, flexibility and fine-grained control. It is unopinionated (it lets you decide how you want to implement things). Django provides an all-inclusive experience: you get an admin panel, database interfaces, an ORM, and directory structure for your apps and projects out of the box.

Overview Scala combines object-oriented and functional programming in one concise, high-level language. Scala’s static types help avoid bugs in complex applications, and its JVM and JavaScript runtimes let you build high-performance systems with easy access to huge ecosystems of libraries.

Issues Due to the over academization of the language and abundance of features, it has had a low adoption. Our current position is that scala has moved to the maintenance only, and new projects should shy away from using scala

We use Nodej.s in DevOps -

• Write small tools or in serverless framework, lambda tasks (gluing processes)
• But I wouldn’t say it’s the main language in “devops domain”

Python is a powerful high-level, interpreted, open source, object-oriented programming language, portable, extensible and embeddable with simple syntax and large standard libraries to solve common tasks. Python is a general-purpose language. It has wide range of applications from Web development (like: Django and Bottle), scientific and mathematical computing (Orange, SymPy, NumPy) to desktop graphical user Interfaces (Pygame, Panda3D).

Robust, Age Proven and Renewing The Java language although being with us already for a long time is renewing itself. Starting form Java 9 we now see a new version of Java each 6 months. This means more features much faster. I still believe that using Java gives a very strict and obvious structure to the program which is sometimes lacking in other programming languages.

Framework Support The rise if Reactive Frameworks (Vert.x, Spring Reactor and others) and Micro frameworks (Spark Java, javalite, ligth4k) make java very versatile in it’s use and use cases. With the above mentioned frameworks we can now easily program reactive systems and micro services with minimal footprint. Java is always going with the Trends (although sometimes a bit behind) as we can see with Docker support in the JVM just recently.

The Java community is a huge community one which cannot be neglected.

A software monolithic architecture means that all the business logic is written in one piece, in many cases - in a single executable. This is in opposite to a Micro Services Architecture (MSA) where well defined pieces of logic are separated to different, stand-alone services, communicating via Network Techniques (HTTP requests, Sockets, Queues, etc)

We should stop leading monolithic architectures because of several reasons:

• MSA applications consists of different services, each one is a significantly smaller, simpler code base. Much more easier to maintain.
• in MSA Each service may serve many clients, and be used in the future for different purposes.
• In MSA each service can be written in a different language, regardless of the technology of other services.
• In MSA, each service can be fully developed in a different team, in another place in the world. Only interfaces matter.
• In MSA each service has it’s own life cycle, and can be updated independently.

All of the above are not the case in a monolithic application, therefore we should no longer lead to this approach.

Embedding a servlet container, such as Jetty or Tomcat, inside a Java application has many advantages over running the application inside a container. Testing is relatively painless because of the simple startup, and the development environment is closer to production. This is the reason all frameworks and libraries which provides HTTP interfaces embed the Servlet container within the framework itself (i.e. Spring, Spark-Framework, Vert.X and many DropWizard and others). For this reason, we recommend you stop using the outdated deployment approach of war creation/deployment.

It seems like the utilization of Ruby as a software development language which was very popular in the DevOps movement mainly with tools like Chef and Puppet Logstash Fluentd and a lot of scripting and utilities around ruby and ruby on rails application lifecycle such as Capistrano, has taken a punch in favor of python go and javascript. With the rise of popularity of these langs & frameworks, we see less and less use of ruby.

We assume that as there will always be something written in ruby but it will most definitely not be the language of choice when we are required to develop a utility / micro-app.

Groovy is an object oriented language which is based on the JVM. It is both a static and a dynamic language.It can be used as both a programming language and a scripting language. We recommend to stop using Groovy except for in tools such as Jenkins and/or gradle. The reason for this is for one, the loss of popularity for this language and this a lack of community. Also, is is a much slower language as java itself or other jvm based languages. Furthermore, there are other more popular Language based on the JVM which have a much larger community and have a much higher adoption rate. The communities are growing and the language is evolving at a high pace. All of this is not happening with Groovy…

Autonomous things, abbreviated AuT, or the Internet of autonomous things, abbreviated as IoAT, is an emerging term for the technological developments that are expected to bring computers into the physical environment as autonomous entities. Key properties of AuT include ability to act upon environmental changes without human direction or centralized processing. AuT are capable of freely moving in the physical world, interacting with humans and other AuT and objects.

Major approaches in the AuT field include:

1. Mobile Agent Architecture - AuT device send and receive mobile agents that know how to control, manage and collect data from the devices. AuT agents are capable of visiting AuT, adding behavior and data to their baseline implementation and instruct AuT device upon further processing.

2. Mobile Cloud Architecture - provides methods of accessing cloud directly from AuT device in order to report state and status, upgrade behavior, etc.

Android P Features and API Overview

It offers the following :

• new RTT WiFi and location tracking. App must have Location permission and WiFi scanning on.
• Smart notifications improvements such as smart reply inline from notification bar, adding images to the reply ! Saving the reply as drafts.
• Multi Camera Support - access 2 camera devices in the same time

ML Kit for Firebase

ML Kit is a mobile SDK that brings Google’s machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you’re new or experienced in machine learning, you can implement the functionality you need in just a few lines of code. There’s no need to have deep knowledge of neural networks or model optimization to get started. On the other hand, if you are an experienced ML developer, ML Kit provides convenient APIs that help you use your custom TensorFlow Lite models in your mobile apps.

Production-ready for common use cases

ML Kit comes with a set of ready-to-use APIs for common mobile use cases: recognizing text, detecting faces, identifying landmarks, scanning barcodes, and labeling images. Simply pass in data to the ML Kit library and it gives you the information you need.

On-device or in the cloud

ML Kit’s selection of APIs run on-device or in the cloud. Our on-device APIs can process your data quickly and work even when there’s no network connection. Our cloud-based APIs, on the other hand, leverage the power of Google Cloud Platform’s machine learning technology to give you an even higher level of accuracy.

Google Virtual Reality

Use the Google VR SDK to build apps for Daydream and Cardboard. The SDK provides native APIs for key VR features like user input, controller support, and rendering, which you can use to build new VR experiences.

ARCore

ARCore is Google’s platform for building augmented reality experiences. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information. Some of the APIs are available across Android and iOS to enable shared AR experiences.

ARCore uses three key capabilities to integrate virtual content with the real world as seen through your phone’s camera:

• Motion tracking allows the phone to understand and track its position relative to the world.
• Environmental understanding allows the phone to detect the size and location of all type of surfaces: horizontal, vertical and angled surfaces like the ground, a coffee table or walls.
• Light estimation allows the phone to estimate the environment’s current lighting conditions.

Dart

Dart is the Google’s programming language that is optimised for client side. Many rich frameworks such as Flutter are using Dart as their development language.

Approach

Dart is a light wight, object oriented language that utilise modern programming paradigms such as reactive functional programming. Dart syntax is very familiar to C#, C++, Java and Kotlin developers. Furthermore, Dart code is highly expressive

Why Dart

Dart compiles to ARM and x86 code, apps that developed in Dart can run natively on Android and iOS

Jetpack is a collection of Android software components to make it easier for you to develop great Android apps. These components help you follow best practices, free you from writing boilerplate code, and simplify complex tasks, so you can focus on the code you care about.

Jetpack comprises the androidx.* package libraries, unbundled from the platform APIs. This means that it offers backward compatibility and is updated more frequently than the Android platform, making sure you always have access to the latest and greatest versions of the Jetpack components.

Foundation

AppCompat Degrade gracefully on older versions of Android

Android KTX Write more concise, idiomatic Kotlin code

Multidex Provide support for apps with multiple DEX files

Test An Android testing framework for unit and runtime UI tests

Architecture

Data Binding Declaratively bind observable data to UI elements

Lifecycles Manage your activity and fragment lifecycles

LiveData Notify views when underlying database changes

Navigation Handle everything needed for in-app navigation

Paging Gradually load information on demand from your data source

Room Fluent SQLite database access

ViewModel Manage UI-related data in a lifecycle-conscious way

WorkManager Manage your Android background jobs

Behavior

Download manager Schedule and manage large downloads

Media & playback Backwards compatible APIs for media playback and routing (including Google Cast)

Notifications Provides a backwards-compatible notification API with support for Wear and Auto

Permissions Compatibility APIs for checking and requesting app permissions

Sharing Provides a share action suitable for an app’s action bar

Slices Create flexible UI elements that can display app data outside the app

Preferences Create interactive settings screens

UI

Animation & transitions Move widgets and transition between screens

Auto Components to help develop apps for Android Auto

Emoji Enable an up-to-date emoji font on older platforms

Fragment A basic unit of composable UI

Layout Lay out widgets using different algorithms

Palette Pull useful information out of color palettes

TV Components to help develop apps for Android TV

Wear OS by Google Components to help develop apps for Wear

MVVM Model-View-ViewModel is an architectural approach used to abstract the state and behavior of a view, which allows us to separate the development of the UI from the business logic. This is accomplished by the introduction of a ViewModel, whose responsibility is to expose the data objects of a model and handle any of the application’s logic involved in the display of a view.

Google recently released the Android Architecture Components library that makes it easy to integrate this pattern with some more useful components.

Android Things lets you build professional, mass-market products on a trusted platform, without previous knowledge of embedded system design. It reduces the large, upfront development costs and the risks inherent in getting your idea off the ground. When you’re ready to ship large quantities of devices, your costs also scale linearly and ongoing engineering and testing costs are minimized with Google-provided updates.

Android Things extends the core Android framework with additional APIs provided by the Things Support Library, which lets you integrate with new types of hardware not found on mobile devices. Developing apps for embedded devices is different from mobile in a few important ways such as:

• More flexible access to hardware peripherals and drivers than mobile devices
• System apps are not present to optimize startup and storage requirements Apps are launched automatically on startup to immerse your users in the app experience.
• Devices expose only one app to users, instead of multiple like with mobile devices.

Android, IOS/OSx or web developers!

Have you ever thought animation can be so easy? Do you want to boost your web page or mobile app with cool animation which take you product to the highest standards of user experience? Now, with Lottie by airbnb, animation has never been so easy and cool!

Lottie is a library which developed by airbnb for Android, iOS/MacOs, Web and even ReactNative apps that parses Adobe After Effects animations exported as json with Bodymovin and renders them natively on mobile and web include . For the first time, designers can create and ship beautiful animations without an engineer painstakingly recreating it by hand. They say a picture is worth 1,000 words so here are 13,000

Checkout the Lottie documentation

It looks like that Lottie is the next thing in animation frameworks in the mobile and Web world. As for Android, so far we usually used native animation framework such as Object animators or Value animators API which defined in code or in Android Resource XML, it seems that Lottie will do to animation what Retrofit did to the Http request frameworks.

So far, Lottie is already used in many apps such as GoogleHome, Target and Uber

A collection of libraries that help you design robust, testable, and maintainable apps. Start with classes for managing your UI component lifecycle and handling data persistence.

Historically, Google’s Android documentation examples lacked architecture and structure. This changes with the release of Android Architecture Components, a set of opinionated libraries that help developers create Android applications with better architecture. They address longstanding pain points of Android development: handling lifecycles; pagination; SQLite databases; and data persistence over configuration changes. The libraries don’t need to be used together — you can pick the ones you need most and integrate them into your existing project.

New lifecycle-aware components help you manage your activity and fragment lifecycles. Survive configuration changes, avoid memory leaks and easily load data into your UI using:

LiveData: - Is an observable data holder class which has lifecycle-awareness. This awareness ensures LiveData only updates app component observers that are in an active lifecycle state.

ViewModel: - This class is designed to store and manage UI-related data in a lifecycle conscious way. The ViewModel class allows data to survive configuration changes such as screen rotations.

Room: - A SQLite object mapping library. Room provides compile time checks of SQLite statements and can return RxJava, Flowable and LiveData observables.

Paging: - This library makes it easier for your app to gradually load information as needed from a local or remote data source, without overloading the device or waiting too long for a big database query.

A set of Kotlin extensions for Android app development. The goal of Android KTX is to make Android development with Kotlin more concise, pleasant, and idiomatic by leveraging the features of the language such as extension functions/properties, lambdas, named parameters, and parameter defaults. It is an explicit goal of this project to not add any new features to the existing Android APIs.

Android KTX saves a lot of boilerplate code, making development speed and productivity increase.

Android KTX is in Preview - Bugs and Stability issues may occur

Flutter is the new Google’s framework for creating high quality apps both for Android and iOS with a single codebase.

The framework is based on Dart programming language which eventually generates an Android’s APK and iOS IPA packages.

Flutter first introduced in May 2017, and currently still in beta but so far, its already has a large community around the world.

Compared to other cross platform frameworks such as ReactNative which usually relays on the platform native UI widgets, Flutter internally renders the UI widgets and draw them directly on the screen canvas. This makes flutter much faster and more reliable when it comes to backwards support.

As said above, Flutter has large community already, it has highly rich documentation and has Google’s Support.

For more details checkout about Flutter in their home page

Google Wear OS introduced on March 2014 previously known as Android Wear, is Google’s Android version for smartwatches and other wearables.

With Wear OS and embedded Google Assistance, James Bond world has never been so realistic. Check the weather or search for a restaurant by speaking to your watch. Get update with your next meetings and directions, stay connections and even pay using your watch. The future is here! and hey, Wear OS can easily connect with both Android or IPhone.

Wear OS brings new vision to the people that love to be on the edge of what the mobile technology has to offer. If you working out, listening to your music or on your business trip, Google Google Wear OS is the little thing that will take your activity to the next level. And it works both with Android and IPhone.

Check out the Google Wear Homepage

Kotlin is a statically typed programming language that runs on JVM (Java Virtual Machine). Development lead Andrey Breslav has said that Kotlin is designed to be an industrial-strength object-oriented language, and a “better language” than Java, but still be fully interoperable with Java code, allowing companies to make a gradual migration from Java to Kotlin.

At Google I/O 2017, Google announced first-class support for Kotlin on Android. As of Android Studio 3.0 (October 2017) Kotlin is a fully supported programming language by Google on the Android Operating System, and is directly included in the Android Studio 3.0 IDE package as an alternative to the standard Java compiler.

Kotlin is now an official language on Android. It’s expressive, concise, and powerful. Best of all, it’s interoperable with our existing Android languages and runtime.

Besides the stylistic advantages over Java, Kotlin has new and exciting features that are offering performance and stability improvements over Java code, such as Kotlin coroutines, Rx-like collection iteration modifiers, null-safety, operator overloading, etc.

Firebase Platform

Firebase is a powerful platform for Mobile Android, iOS and WebApps.

Firebase gives you functionality like analytics, databases, messaging and crash reporting so you can move quickly and focus on your users. Firebase is built on Google infrastructure and scales automatically.

Firebase gives you many services such as: analytics, Auth, Realtime database, Cloud messaging, Storage, Crash reporting and more.

Using One Firebase Console to control the services and data is provided. It is easy to use and setup by using Firebase SDK libraries. Usually, in order to setup most of the services there are wide range of documentation and code examples.

What is new in firebase: Cloud fireStore Cloud functinos

Gradle is a build tool. Following a build-by-convention approach, Gradle allows for declaratively modelling your problem domain using a powerful and expressive domain-specific language (DSL) implemented in Groovy instead of XML. Because Gradle is a JVM native, it allows you to write custom logic in the language you’re most comfortable with, be it Java or Groovy. it too provides a powerful dependency management.

Although still many more developers use Maven (~60%) Gradle is a powerful contender/ally which needs to be considered when making the decision on which build tool to use.

The Android Studio build system is based on Gradle, and the Android Gradle plugin adds several features that are specific to building Android apps.

ExoPlayer is an application level media player for Android. It provides an alternative to Android’s MediaPlayer API for playing audio and video both locally and over the Internet. ExoPlayer supports features not currently supported by Android’s MediaPlayer API, including DASH and SmoothStreaming adaptive playbacks. Unlike the MediaPlayer API, ExoPlayer is easy to customize and extend, and can be updated through Play Store application updates.

Repository Pattern

You design a single API that abstracts out all of the storage stuff. The repository implementation deals with all of the decision-making for where the data goes, what all has to get updated, what has to be refreshed from some remote source, and so on.

The Repository has a few key roles:

• Manages Data Storage
• Normalizes Model Objects
• Provides a Clean Reactive API
• Isolates Rest of App from Strategy Changes
• High-Level Repository Strategies
• Network + Network API Caching + Presistence (DB)

Why the Repository Pattern?

• Decouples the application from the data sources
• Provides data from multiple sources (DB, API) without clients being concerned about this
• Isolates the data layer
• Single place, centralized, consistent access to data
• Testable business logic via Unit Tests
• Easily add new sources

Dependency Injection

Dependency Injection is a design pattern which allows developers to write code that has low coupling and which can therefore be easily tested.

Dependency Injection is a technique whereby one object (or static method) supplies the dependencies of another object.

A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it.

One of the most powerful and used DI in Android is Dagger which is maintained by Square and Google engineers.

MVP - Model View Presenter

The MVP pattern allows separate the presentation layer from the logic, so that everything about how the interface works is separated from how we represent it on screen. First thing to clarify is that MVP is not an architectural pattern, it’s only responsible for the presentation layer.

The presenter - responsible to act as the middle man between view and model. It retrieves data from the model and returns it formatted to the view. But unlike the typical MVC, it also decides what happens when you interact with the view.

The View - usually implemented by an Activity/Fragment, will contain a reference to the presenter. The only thing that the view will do is calling a method from the presenter every time there is an interface action (a button click for example).

The model - would only be the gateway to the domain layer or business logic. It is enough to see it as the provider of the data we want to display in the view.

MVP makes views independent from our data source. We divide the application into at least three different layers, which let us test them independently. With MVP we are able to take most of logic out from the activities so that we can test it without using instrumentation tests.

Java is currently the most popular programming language, used for building server-side, desktop and mobile applications. It is core foundation for developing Android apps, making it favorite of many programmers. With its WORA mantra (write once, run anywhere), it is designed to be portable and run across multiple software platforms.

Also, as Java runs under JVM, Java Virtual Machine, many other languages can be complied into JVM bytecode and interoperate with Java code. In addition to that, most of the Android API, example code, documentation, blog posts, forum discussions and various open source projects are written in Java, still making it primary development platform for Android for foreseeable future.

React Native - building native apps with React.

React Native is the first and most popular paradigm for building native apps with JS. these are not the weak PhoneGap like web apps but real native applications, with real native-like performance.

RN is not dependent on any vendor in order support new native features, it is in sync with the native OS components and is partly what makes it is so strong.

It is also relatively easy to pick up, especially if you worked with React Web before.

it has a vivid community with many users and contributors.

this technology really bridges frontend and mobile in the best way I know of to date.

lastly, RN is used in React VR/AR.

refs:

• official site : https://facebook.github.io/react-native/
• popular ui framework - https://react-native-training.github.io/react-native-elements/docs/button.html
• trends overview - https://trends.google.com/trends/explore?q=react%20native,native%20base,phonegap,%2Fm%2F0gtv959,%2Fm%2F05q31

Machine learning on mobile devices is offering an array of new and unique user experiences for problems that are close to data people are working on. Experiences that were impossible before, like OCR, image recognition, machine translation, speech recognition, etc, were in the realm of Science Fiction, but they are being brought into our daily lives. Thanks to machine learning on mobile and IoT devices, universal translators from Star Trek, Babel Fish from Hitchhiker’s Guide to the Galaxy, “neural net processor, a learning computer” from Terminator 2, and more, are not a figments of writers’s imagination, but actual products that you can buy, or are in development stages.

TensorFlow is a opensource library for dataflow programming, often used in deep learning. It was developed for internal use by Google Brain team. It was released to the public, and in 2017 mobile/lite version was released that is targeting mobile and embedded/IoT platforms, such as Android, iOS, Rasberry Pi, etc.

Reactive Programming is a paradigm that changes the direction of the flow. Instead of creating threads and blocking for communications, we create pipelines where data is pass though and each layer can react to the data received. The indirection changes the code flow from blocked to async.

Over time the reactive extensions has become a standard for adding higher level functionality on the stream of data while being agnostic to the data itself. The extensions are based on the observable pattern, and allow for cancellation of data process in stream

Google Support API

Background Almost any Android application relies on third party services such as: Geo Location, database and storage, networking management, UI tools and widgets.

In addition, as Android framework evolved on every new API, it still required to support old devices which runs older API’s. For this, the Android framework offers the Google Support API libraries which provide services, UI tools and backward compatibility support.

Support Library Uses

1. Backward compatibility for newer API For example, Android Fragments introduced in API 11 (Android 3.0), if want to use Fragments on older APIS we can use the fragment support library.

2. Helper Classes The Support library provides helpers classes and tools such as Fragments, the Recycler View and the Constrain Layout

3. Testing and Debugging Tools Testing and Debugging is a fundamental principle in SW development as well in Android. The Support library offers number of classes beyond our code include the support-annotation library and Testing Support Library that help us to test and debug our app.

Google Support API

Why should you avoid using an event bus?

One common pattern in Android development is the use of event buses. Libraries like Otto and EventBus are often used to remove boilerplate listeners that require tunneling code through many layers of hierarchy. Although an event bus might seem convenient at first, they quickly devolve into a mess of tangled events that are incredibly hard to follow and even more difficult to debug.

Treating Producers Like Synchronous Getters

This is another common pattern that’s incredibly hard to undo in a larger codebase. Often times many activities or fragments will assume events are produced as soon as that component subscribes to the event bus. They’ll set a field based on the event, and then proceed to use that field in other lifecycle methods with the assumption that it is not null.

Alternatives

No library or tool will fix these problems for free, but some tools and patterns will encourage you to do things the right way.

An event bus, when used correctly, can probably avoid these issues. However, it also encourages these practices more than most tools. Using simple listeners, although they will require more code, will make things much easier to understand.

For more complex scenarios, RxJava provides a great solution

why an application needs a good architecture?

A simple answer is that everything should be organized in a proper way. So does your Android application.

If an application is developing with no pattern, the following problems are raised

• All of the codes are not covered by Unit Tests.
• It is difficult to debug a class because it contains a huge number of functions.
• You are unable to keep track of the logic inside that huge class.
• Another developer finds it so difficult to maintain and add new features to your work.

What are the benefits of a proper architecture?

• Simplicity
• Testability
• Easy maintenance

**What are the popular patterns? **

• MVP ( Model — View — Presenter)
• MVVM (Model — View — ViewModel)
• Clean Architecture

SQLite was a good way to store/retrieve large sets of Data until object relational mapping (ORM) libraries were introduced. Libraries such as GreenDao, SugarORM, Realm and more.

Today, after almost a year, google have announced the architecture component libraries - which also introduced Room. Room was built on top of Sqlite and makes the CRUD operations very intuitive and easy to use so we no longer need to use SQLite old fashion queries to insert and get the data. Room also supports RxJava and LiveData returned types which makes the code cleaner and more robust.

ListView is a view group that displays a list of scrollable items. The list items are automatically inserted to the list using an Adapter that pulls content from a source such as an array or database query and converts each item result into a view that’s placed into the list.

**Why stop using it? **

• ListView does not recycler the items by default and will cost more memory processing each view item.
• RecyclerView is better in performance and manages memory by recycling the views

lets you run Java 8 code with lambda expressions, method references and try-with-resources statements on Java 7, 6 or 5. It does this by transforming your Java 8 compiled bytecode so that it can run on an older Java runtime. After the transformation they are just a bunch of normal .class files, without any additional runtime dependencies.

Why stop using it?

• Java 8 has been released and it includes lambda expressions.
• RetroLambda is limited support for backporting default methods and static methods on interfaces.

RxJava – Reactive Extensions for the JVM

a library for composing asynchronous and event-based programs using observable sequences for the Java VM.

Why stop using it?

• RxJava 1.X does not handle back-pressure
• RxJava 1.X will be deprecated soon and developers will have to change to RxJava 2.X at some point.. rather do it now :)
• RxJava 2.X has been completely rewritten from scratch on top of the Reactive-Streams specification. The specification itself has evolved out of RxJava 1.x and provides a common baseline for reactive systems and libraries.