Jenkins, Jclouds & EC2 KickStart walk-through

As part of an experiment of utlizing EC2 instances during build, I wanted to share with you how easy it was to get Jenkins and EC2 to work togehter without needing to create custom AMI's [which is usually the most daughnting task], you might ask yourself why not use the Jenkins EC2 plugin the short answer is it does much more than the EC2 plugin the long answer - continue reading 

The breeze of configuring a Jenkins slave, thanks to the great guys @ jclouds [ see jclouds plugin page on Jenkins-ci ], what was missing  was some documentation on how to actually configure the diffrent options of the plugin and how to make your job play nicely on the newley created instance.

So I hope the following recipie will shed some light on how to go about and configuring your build to run on EC2 via Jenkins & Jclouds.

What you need:

1. Jenkins [quite obvious - tested with 1.477 (latest at time of composing this post)]
2. Jenkins Jclouds plugin [ tested with version 2.2.2 ]
3. Access to an EC2 account - I used the FreeTier for this tutorial [? what is Amazon free tier ?]

I presume you have Jenkins installed !

Adding the plugin is easy as going to http://yourjeknins/pluginManager/available and search for the Jclouds plugin [ if you are using 1.475 and up you have a search box ], install and reload jenkins [ again you might not need to in versions grater than 1.4x ].

Three steps in this walk through

1. Add a cloud & Configuring plugins
2. Adding templetes [ or not ... ]
3. Configuring a job to use jclouds

Step 1: Add a cloud

Once the plugin is installed go to Jenkins > Manage Jenkins (http://yourjenkins/configure) and "Add a new cloud"

Screen shot #1 :: Adding a cloud
Please note if you already have other "cloud provider plugins" you will have more than "Cloud (JClouds)" option.

Configuring the plugin:

See the screenshot below (i'll elaborate on it stright after it)

Screen shot #2 :: Plugin basic configuration

• Profile: call it whatever you like this is how you will choose instances when tieing jobs to this cloud [will show exmple later on in thie post]
• Provider Name: aws-ec2, please note there is a very long list of providers which I will be testing but it'q quite impressive [? see full list here ? another reason to use this plugin over the EC2 plugin :)]
• End Point URL: N/A for ec2 [ I used the same plugin for OpenStack which requires the api url to connect to ]
• Max. No. of Instances: this is a tricky one how many instances of this cloud may you have meaning can you have 10 jobs using the same cloud - limit this to the number of instances you want utlized of this cloud provider [ you can also limit it per credential !]
• Retention Time: How long, in minutes, to wait for a slave to remain idle before disconnecting and terminating it [pretty striaght forward].
• Identity: get it from: https://portal.aws.amazon.com/gp/aws/securityCredentials whilst loged in
• Credential: also can be found @: https://portal.aws.amazon.com/gp/aws/securityCredentials

RSA private Key and Public key: the plugin can generate one for you  after creation you will be able to see it your account [ considering your account credentials are correct ]:

Screen shot #4 :: EC2 keypairs

Step 2 : Adding Templates [ Or not ]:

Just under the plugin configuration you will find a button

Screen shot #5 :: Cloud instance template

You have two options here:

1. Use an actual ready made AMI [? Amazon Machine Images ?]
2. Let Jclouds locate AMI's according to a regexp 

Screen shot #6 :: specify existing AMI [ and regsion e.g: "us-east-01/ami-xxxx" ]

and

Screen shot #7 :: specify AMI regexp

I prefere the second [ at lease that is what I needed in order to test this plugin with EC2 ... ]

The specify image field requires you to have the certain image in your EC2 region and thus whilst specifying an AMI you will need to tell the plugin the regions/aminame for example i used: "us-east-1a/ami-079e5d6e" [ in order to locate and AMI you can go to: https://console.aws.amazon.com/ec2/home?region=us-east-1&#s=Images and search by keyword ].

to summarize the "adding templates section" it could look as simplae as this:

Screen shot #8 :: Cloud instance template configuration

The name and label are quite confusing here [ if you have one I wouldn't worry about this ! ].

Name: ec2tmpl1 if you recall the section above with keypairs you could see the reference of the plugin to ec2tmpl1 in the keypair name.

label: Jenkins works with labels so the node will recive this label and the Name mentioned above will be what Jenkins calls "Labels" see:

Screen shot #9 :: Jenkins - manage nodes screen

The reason this is so confusing is that the Name was given to a "node" which represents a cloud whilst the label is provided to an instance which was started in the cloud. 

• Hardware ID: see I chose t1.micro if you want your experiment to be free you can use only the micro instances [? see the free tier link on EC2 ?]

There are a few more advanced usages which I will probably utlize in a later time for exmaple the init script [ firing up Chef for example ] the coolest thing about this plugin is that it ties it to Jenkins, installs JAVA straight from Oracle etc etc ... so in a matter of seconds you have an instance in the cloud.

Step 3: Configure your job to use Jclouds

This one is the easiest, in job configuration page you now (after installing the plugin) have an option:

Screen shot #10 :: Job configuration page "tie to node"

Executing this build will fireup an instance, execute the build and stop [ if you look at screeshot #9 you see I used stop on terminate - uncheck this if you wish to terminate the instance instead of just stopping it ] the instance:

Screen shot #11 :: Job Console 

In your EC2 dashboard you now can see:

 Screen shot #12 :: Instance status on EC2

Hope this was informative and easy to use.

Here are some resources you can reference on certain topics discussed in this post:

1. what is Amazon free tier
2. Jenkins Jclouds plugin wiki on jenkins-ci
3. Jclouds "providers list" full list
4. Amazon Machine Images
5. Jenkins latest enhancements [ui & plugin management ]

I will be updating this with more advanced usages as time premits and as needs mature.

Enjoy

Haggai Philip Zagury