Fuse Day– “Jenkins As A service” Chef route – summary
In order to summarize our Fuse day I would like to take a moment and explain our end goal.
The end goal consist of:
- Amazon Instance running multiple tenants of Jenkins
- A CLI / Web interface which will eventually manage this “Multi Jenkins” instance in a way that one can create a new instance on the fly.
- At the tenant level:
- Each tenant will utilize the Jenkins scm sync plugin and in that manor maintain configuration in a git repo for disaster recovery.
- Each tenant will have it’s own custom authentication and authorization strategy.
- Each Jenkins tenant will have is own plugin-stack and we already know we might have different tenant types for example Java, Ruby etc, so we would like to have 1 base plugin- stack + a domain specific plugin-stack suitable for each tenant type.
- Each tenant will have it’s own listen port, hence we will need a reverse proxy to serve clean urls for each tenant based on his name / dns name.
We gathered around the white board [well, it was really a transparent window, but don’t be picky] and discussed our options – we cam up with two options:
- Use Opscode’s Jenkins chef cookbook with some customizations in order to support the requirements above.
- Use Docker which will, instantiate isolated Jenkins tenants on one Amazon instance.
We had a good feeling we will not be able to do all these tasks in one day, but we have to start somewhere …
So we split into two groups One group the Chef Group & the Docker group.
Each group had an initial goal running a Jenkins instance using Chef & Docker and once that is achieved start building the plugin sets.
In the Chef Group [ Schachar Ben Zeev, Yoram Micaeli, Timor Raiman & yours truly ] we did the following:
Use: https://github.com/tikalk-cookbooks/alm a chef-solo repository which manages all the dependency and can be used by us in order to develop it.
Use Vagrant in order to instantiate a local instance on Virtual Box and start iterating on that see Vagranfile on github – in this use case vagrant up ci would do the trick of running 1 instance of jenkins ci-server on our vm.
So by lunch we had a Vagrant file with a VirtualBox instance running
The way we achieved that was by creating a chef role and instructed the Vagrant provisioner to use it:
So our cookbook at this stage is empty, no need to do anything for this first instance!
The key in this is our chef.add_role, which sets some parameters used by the Jenkins cookbook:
At this stage it hit us we will need some mechanism of invoking chef on our machine and editing the run-list / node object (our “dna.json” file in the Vagrant jargon) in order to be able to run more than 1 Jenkins instance, In addition to that we had to start building our plugin-stacks.
Timor came up with an idea, which suggested we would generate a datbag per tenant with the following parameters:
- Tenant id => numerical value
- Company Name => the name of the customer
- Dns Prefix (e-mail suffix) / fqdn of the tenant
- Expire Date – considering it’s a time based tenant it will be removed / blocked if the Expire data is larger than this date.
- Purge Date – how long is the tenants grace period
- Jenkins Version – what version of Jenkins is this tenant going to run
- Sync SCM repo – the repository we will store the data in
- Plugin stacks – an Array of plugin sets selected by the customer
- Specific plugins – a plugin set outside of the stacks
We came up with this 1st version of databag:
And whilst we were using data bags we thought why not hold the plugin-sets in a database too like so:
The base databag:
A Stack (Java) specific databag:
At this stage we started to understand how we could use the chef paradigm utilizing the node object and define a service, we had to do the following:
- Collect the information needed in order to construct the databag per user – based on the hellorandm example
- Connect to the remote machine and run chef (chef solo / server is still debatable) with that json file
- Analyze the chef-run input in order to determine it’s success / failure
- Report back to the cli/ui tool.
At this stage we ran out of time, but what we are now taking offline is:
- Continue the chef-solo + data bag route where we insatiate new tenants from the Jenkins cookbook – in the current state of the cookbook it looks like we will need to write our own / expand it to facilitate more than 1 Jenkins tenant on 1 Amazon Instance.
- Writing a rest based service which will run on the amazon instance and listen on a web / cli tool for a json object. Once delivered the service will then invoke chef on the node with the new tenant. Another option I am checking out is using Mcollective in order to pass a json to the node and instantiate the run (Mcollective already has a CLI ! – so perhaps all we need to do is expand it for our needs ?)
I stumbled upon “Mcollective with chef” blog post which is worth cheking out.
- Consider combining Docker / Vagrant to do the heavy lifting: The second route in parallel just concluded and reported their finding on Docker (stay tuned for their blog post and findings) – Another “Timor search” yielded https://github.com/hw-cookbooks/lxc which is basically doing “Docker style” management of LXC instances so we could have a complete isolated instance running on LXC hosting mutipule Jenkins tenants in this case we will not need to expand the Jenkins cookbook …
- Write the CLI to interact with the service running on the instance
- Write the Web
To summarize: It was a good experience seeing different point of views and methods on how to accomplish this, we will definitely continue hacking on this in our Bi-Weekly workshops and of course following Fuse Days.
We would most appreciate your comments / suggestions / ideas we might have missed.