Opinionated perspective of "Ordered Chaos"
I have to say with “Radar Days” here @Tikal my beacons have overheated ;) …
It’s the 2nd year in a row where, were in this when I thoght I know what’s going on and to get to a point where your’e amazed Like I had no idea I could do that !
MVP time !
It’s kind of the “name of the game” nowdays, how fast can I do this thing or that, and in many cases it become’s Chaos … which like I think Iv’e argued enough, is the main theme of our radar … when we all woak up from the march into the MicroServices and Infinitay Scalable Whathavyou’s ….
If your’e a small startup with
infinite scaling potential to a
big enterprise, solutions are a blend or a mesh (which seems to be trendy word nowadays) of technologies, we also tend to use the term Polyglotisem and Chaos Engineering as it’s complementery.
In Retro on 2017-18 one of the things which stood out this year in all our Round-Tables (Our way of keeping up to speed with ideas coming back and forth between customers and our consultants) is the lack of Audit and Tracability whilst we are
Zeplin (in context of service mesh BTW) as a complementery to the
EFK, (which were also implemented by Stas, Eran, Salo & Myself), stacks which don’t suffice (which we keep squeezing to the limit with grocks and shit … ), and from a pure “DevOps prespective” we had Avi & Rafi which were talking specifically about Auditing and who did this or who did that in our customers cluster(s) - many war stories ;).
One in short: some developer with “elevated privilidges” deploys an untested version …,
Which pu a spotlight on another wind of change is this “Security First” (Salo Shp tm :) ), approch, I’ve heard from salo on signal & vova on wahts-up (or signal) this year things which I would have overlooked unless tuned in to the chatter.
Back to topic, I think we were less picky at first, when things failed as long as it was snappy … and we had to implement technologies that allow such expereimnts like Reverse Pocies, ssh tunnels which lead to Canary releases and Blue Green etc, where K8s seems like a quick win (once you learn how to deploy it / manage it kops, kubespray, tectonic, kubeadm, “comando”) consdierng all the AdHock Configuration Management is automated (DB, CA’s, Mail Relays, DNS service and the supporting infra stuff we all hate to love).
When you convinced with my very opinionated POV, it’s a wild wild west out-there, when the challanges we are facing arn’t the by the book perfectly molded microservice, It’s those “mon/micro-services” we are being fed through the legacy piepline which is a reality we all (Product, Dev, Ops and SRE’s) need to still consider … and nurture. However …
Wind of Change
In some cases you see a innovative, pashionate TL (did I mention we’re hiring?), a change which either magentizes or co-exists under the radar as an
experiment of “doing things fast” alongside the exsting operational procedures and slowley starts taking over, Or on the “other side, where there used to be a wall” ;), An Ops guy (in a way - all of us) proposing to put “Jenkins/AWX/*” in a production environment, and in 2010, it was banned out! (at least it was banned by a few companies iv’e consulted throught the years), today we see Jenkins spining up Swarm clusters (Asaphe, Itay, Itai) in ci for testing & Production, Jenkins pipelines swith opinionated pieline librereis and viseverse (also see our growing own Tikal Pileine Librery, Yoram Chen & Itai as key contributers), as a service deploying to your managed Kubernetes clusters … on GK F**g E ;) … [aya, eithan this sounds much better !]
Solutions like CodeFresh (Vova POC from client side this time), GitlabCi (Anatoly, ethos team in Fuse), which just moments ago releases a serverless / fuction framework ontop of it’s offering, Drone.io (dartanian team in Fuse), Travis, Concorse and others are the
cole behind the continuos deployment train, for microservices … in addition to
slack are putting cluters to sleep (tal-k did our “kops manged ac cluster”), it’s in the “christmas” spirit of …
Where you can’t do it with Jenkins let’s write a Function (where? - we don’t care) get s**t done ! true story, one of our customers (don’t laugh) has hundreds of lambdas filled with “copy pasta” code doing the same thing
with a twist …, It’s really OK were learning (faster) when Chaos part of your learnining process, as long as we find the time to point out the
technical dept (I have a backlog full of those).
Really i’m just trying to keep things “light” consdering this is our job which I do take seriously, but enjoy a little laugh along the way. Not in a condensending way but the same as I started this post, looking at the fact that “information is out there for the harvest” we need the ability to lift our heads up from the day to day to in many cases find somthing to help us (and our customres) with their technological challanges and in many cases help clear those technical debt.
4 years ago we took a project to run a few workshops for the IDF which turined out to be about 4 years of a project where Salo & Ramon led that effort which became to be IMO the biggest OpenStack project which wasn’t performed by one of the leading vendors such as Mirantis, RedHat or Canonincal which 2 years in were amazed of the depth of details which was put into what was widely called OSTKL,
Open Stack Tikal, Salo continues to to consult them and in the process we also met Chris which joined us earlier this year, which i also worked with personally in the same project.
Evolution fo Chaos
When Docker came out in 2015, the “big boom” was the
cool toolz which docker broght to the table arround containers but containers still have many issues, when docker came out with 0.8 and the recomendation by Docker Inc was don’t use docker in production,
8% of the market was dockering and I am not counting the
Yelp!’s and the
Netflixes they are just the first
2% (Reminder - Opinionated!!).
Down the Technical Depth backlog
Yes, all our “non-kubernetes” workloads all those
Hadoops replaced by SaaS offerings, Yaay!, consdeirng the operational overhead (becuse in the long term cost isn’t the name of the game scalbility is ! right ?).
Seems like we are looking for
creative ways (ECS/Fargate Ofer & Ilan.G) and i’ve witnessed & implemented a few ;) which try to “bend the monolith” so badly to be a microservice that .., and in hebrew we have a saying “The Temprary is the most permanent” which we tend to do due to
business justifications which put us into another technical depth (who’s counting?)
But in the “era of expermenting” I thinkg the shift is in the way we accept
trial and error much more efficintelly, than looking at things as
failure (such as builds / shitty commits/ typoes etc), Dsitributed Soruce control, Continuous Integration and Continuos Delivery as the driver of frequent change enables us (devops) to bisect with automation Tools like
Terraform (Itai, Eitan) and others, and help treat those “hunks” in our modules or playbooks as the atomic parts of infrustructure.
We see complementry tools such as
Jenkins for *,
Terragrunt which “incidently” in the same week was higlighted by Both Chen @ ClickSoftware and Lior in the “Radar Days” spirit, and of course another “jenkins for * alterntive” - like
AWX (Avi & Aya).
We also see the Locked-in or Locking-in solutions such as IBM Cloud (Tom,Amir) CodeCommit (Anatoly/Chen) and AWS apiGW and AWS Lambda (GM/Lightapp) and Google Functions and Firebase on the other.
So when were looking at the entire spectrum of solutions we as DevOps / SRE’s, we need the “know how” of the “barzel” to the “vm” to the “container” in the “pod” the “Chaos” is only in the swirl / the eye of the storm when you dive in it’s netwoking, routing, security etc “system premitives” on steroids - perhapps this is why k8s and apprentelly not ncesecarily with Docker (Mukitsu, Rkt others) is considerd the
Cloud Operating system well with Operators making k8s a PaaS and FaaS framweokrs scaling from SW frameworks such as serverless.js, claudia.js and others which correspond directly with a cloud provider or a swarm/k8s powered serverless framework - which is considered by the “bare metal” reps the way to go (better than docker, lets just skip that !), it’s really a great thing that Chaos is here becuase personally zooming in-out were having fun …
(Ordered) Better Orginzed - Chaos
Radar items this year weve embraced from start to keep such as
kops - need to go to actual radar to see but weve covered almost all of them thi year
- referene 1
- referene 2
- referene 3
Q4 fuseday shed a light for me on the amount of things wev’e done alone & together as a team such as putting kubernetes in production Rafi, Miron, Itai, Boris there are some or still doing swarm like itay ans Asaphe and ECS / Fargate as mentioned above no body’s perfreclty ever after with just Kuberners (Thank go Salo has ears)