Jenkins the software development framework for everyone [ JUC 2014 Hertzlia - overview ]

Hi all,

Before I start I would like to say it was overwhelming to see that Kohsuke Kawaguchi, Heath Dorn and all other participants who took the trip to ISRAEL inspite of the delicate situation, took the time to participate in this event. And apart of a short distruption @ 09:15 it was a quite day for the Hertzliya area ...

This blog post is to serve as my JUC summary (Worth noting: this summary wasn't sponsered / ordered by any company and reflects my personal take of Jenkins and the Jenkins user conference.)

The what (what is this guy talking about ?)

I have to say I had an epiphany this week in the JUC Hertzlia, Jenkins or Jenkins-ci as suggested in it name / domain name, isn't just a build server, its a Software Development Framework, it does much more than invoke your favorite build tool ... and give me a chance to explain ...

In past years I used to refer to Jenkins (& Hudson) as "a fancy cron job" (hope I didn't offend anyone ...), but witnessing such a vast adoption and hearing from many companies about how they use Jenkins to drive their business makes me use this term "framework" because you can literally mold Jenkins into anything you want ... and the for everyone part is to point our its not only for java as many people used to think ... it is really a multi purpose software development framework which empowers our businesses.

I have just recently completed an Audit for one of my customers @ Tikal, which are using Jenkins to deploy their product into test, pre-prod & prod utilizing tools like mcollective, chef etc, it's not all about CI anymore ! (KK I suggest a rename), Jenkins is much much more than CI!, the reason I refer to it as a framework, is that there are many more usages to Jenkins which never came to mind until you participate in these conferences - A huge thanks to Jfrog & Cloudbees for making it happen. I see companies using Jenkins to trigger backup & restore procedures, spin up servers / autoscale (in production) and many more crazy ideas. This is due to the flexibility in Jenkins's design, ease of use and extensibility - there are over 900 plugins to Jenkins. If you need to make Jenkins do something it doesn't already do -> let's write a plugin for it ... And thanks to the design and the tooling offered by the Jenkins community that makes all this possible.

The why / how - my JUC Hertzlia summary ...

The epiphany -> In the last 3 JUC's in ISRAEL (this year was the 4th), I gave a talk on Jenkins and how I use it in the many projects I participate/d in, and this year I felt I have nothing new to say ... and then without noticing and hearing the different talks I noticed I was so wrong (yes I am not always right ;)) ..., Jenkins is the enabler of many processes we implement into our workflow. In JUC 2012 presentation I referred to Jenkins as the "conveyor of business" driving our software from concept to production utilizing maven, chef and open-stack and I was happy to see that the talks we witnessed both in 2013 and 2014 are exactly where the Jenkins framework is going, you can see plugins & processes which refer to Jenkins as the business enabler, it is now a mission critical system not just a "backend" system used by development.

JUC Hertzlia a had a few great talks by quite a few companies which are all saying how great Jenkins and its ecosystem are. Creator & maintainer of Jenkins Kohsuke Kawaguchi shared the roadmap and a few plugins he was able to reach in order to talk about. As I recall KK himself said he apologizes but these are the plugins he was able to reach in order to talk about and he hopes no one was offended because he didn't talk about their plugin ..., this reminds me two years ago in 2012 JUC when I talked about the Jenkins Multijob plugin where KK said I had no idea you guys were developing it ;). These are the reasons I call Jenkins a framework is it doing so much and being used in so many ways it is hard to keep track of it.

One slide which caught my eye was the following one:

Taken from -> this link and although this slide is very Java oriented (J rebel ...), I think that it gives a clear picture of what is going on in the Java Enterprise market and can reflect on other segments, the numbers may differ give or take 5-10%, which is quite impressive.

Kohsuke Kawaguchi The Keynote speaker gave the opening and closing talks describing the main project focus which is on workflow and improving slave connectivity of both JNLP and SSH slaves basically improving they "reporting" capabilities between the master and slave, In addition "out sourcing" / sharing the Jenkins infrastructure testing, One of the pains I have had in a few projects is testing a Jenkins upgrade In one of my previous projects we came up with a set of shell scripts which did that in a very inefficient way, and I think being able to utlize the "Jenkins official integration testing suite" will drive stability into the upgrade lifecycle of Jenkins and perhaps enable us to be less conservative with LTS releases and using more and more cutting edge releases (latest).

 In continuation to that topic ("testing upgrades") there is Evgeny Zilslis who presented a different approch explaining how jenkins should become a phonix server insted of a snowflake server using chef jenkins cookbook Chef Software, Inc (fomaly opscode), the only missing part in the puzzle for me was persistence, I have been doing something similar in one of my projects but the persistence of build/deployment logs is somthing which IMO is missing from Evgeny's solution. IMO the Chaos Monkey approch is awesome in order to assure the availability of Jenkins but the persistence is important and the problem with persistance is that we need a way of getting our  phonix jenkins server up from the ashes not only with it's configuration but with it's data also.

There is actually a pattern here we can see more and more of, which was also presented by Ohad Basan from RedHad who described their entire build pipeline using Jenkins and puppet utilizing pattens from the openstack approach of how to utilize Jenkins and achieve a deployment pipeline, Ohad mentioned the build pipeline & multijob plugins and how the build pipeline plugin was more flexible considering it could trigger the same build with different parameters in the same execution.

The openstack approach should definitely be looked at when designing a CI & CD solution and it's open -> jenkins.openstack.org & ci.openstack.org/jenkins.html.

Lastly unrelated to the main theme specified above there where three other notable talks:

  1. Oren Katz from Liveperson: using Jenkins to test infrastructure and update Ganglia monitor in order to enable their support desk to find issues before they become issues, quite impressive implementation, the only thing I was not clear about was the usage of rsync instead of SCM.
  2. Nir Koren from SAP: I don't have this years presentation yet but there is his talk titled "Quality on submit" and the slides, Nir describes a process which is quite refreshing in a huge company (65,000 employees) like SAP and how Jenkins plays a key role in their development & deployment processes.
  3. Baruch Sadogursky from JFrog: using bintray to streamline your binaries, another great project (IMO) which enables you to freely store your binaries in the cloud for nothing also referred to "Github for Binaries"

To conclude, there are quite a few approaches for how to use Jenkins and I hope this post / JUC summary might introduce a new term "the Jenkins framework", Jenkins is a project to keep an eye on, for I am not sure if there is a limit to its grasp. Until next year's JUC ...

Over and out, (references below)

HP 

References:

  1. JUC Hertzlia: http://www.cloudbees.com/jenkins/juc-2014/herzliya
  2. Cloudbees: http://www.cloudbees.com/
  3. JUC 2012 presentationhttp://prezi.com/u4zb3h6s5vql/devops-4-devlopers/
  4. Jenkins / jrevel statistics: this link
  5. Phonix server http://martinfowler.com/bliki/PhoenixServer.html
  6. Snowflake server http://martinfowler.com/bliki/SnowflakeServer.html
  7. Chef Jenkins cookbook https://github.com/opscode-cookbooks/jenkins
  8. Chaos Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
  9. workflow plugin: https://github.com/jenkinsci/workflow-plugin
  10. Openstack's jenkins instance: http://jenkins.openstack.org/
  11. Openstakcs ci spec: http://ci.openstack.org/jenkins.html
  12. Delivery pipeline plugin: https://wiki.jenkins-ci.org/display/JENKINS/Delivery+Pipeline+Plugin
  13. MultiJob plugin: https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin