tags:

Building Java application over AWS Cloudformation

This blog will describe the process that I went through to build a fully auto scale AWS application over cloud formation.

The reason I am writing this blog, is the fact that I was surprised at how hard it was. There is a lot of information out there but not all of it works, and not all of it works as I assumed it would work.

So what was I trying to do?

Actually it sounds very simple. I have a spring boot application (thought it could be any application). In addition this machine is part of a cluster so I need to install zookeeper and kafka with configurations. I would like to create a cluster in amazon that can scale when needed.

For this I need to create a machine, setup all the infrastructure on the machine and then deploy my application. Of course I need all services to autostart for that I need to install supervisor.

This was the first time that I was learning about amazon, so I used my previous knowledge to ask some basic questions:

  1. Where do I get the base image?

  2. What frameworks do I need to install on the machine?

  3. How will I update the frameworks in the future?

After some reading I had a few ideas in mind. Since I choose amazon, obviously there is cloudformation. In addition there is the docker framework that knows how to package applications.

The installations come down to three types:

  1. Basic frameworks - java, supervisor…

  2. Application frameworks - kafka, zookeeper

  3. Application itself.

Each layer has it’s own life cycle and need to be updated. My main dilemma was should I use docker to encapsulate some of the frameworks (kafka + zookeeper) since they will be added to other machines, or should I put them part of the cloudformation.

Obviously another direction to look into would be ansible. But currently we did not want to create a full devops environment.

Cloud Formation

So what is cloud formation? Cloud formation is a service that amazon gives on top of EC2. You write a json file that describes what you want configured and installed on your machine. The power of cloud formation is that you define resources, ip address and other definitions, and then the service knows how to setup the machine. Depending on the resource your machine might be rebooted, and sometimes the update is immediately. The other main advantage of cloud formation, is that when you update the definition (stack in CF terminology) it will automatically update all instances that are part of the stack.

The main disadvantage of cloud formation is that it mainly configures physical aspects of the machine, and does amazon services. But if you need your own services or customized services then cloud formation does not support it.

Even when using cloud formation there are issue that it does not deal with. For instance I would have like to assign an IP address to my machine or a pool of address for my stack, but you cannot do it through cloud formation. If you are creating instances and not auto scaling groups, then you can assign an elastic IP to the instance but not with scaling groups (for more information on this topic see Attaching Elastic IP). Amazon does have a cli that we will use on the machine itself to do configurations that cannot be done via the json of the cloud formation.

To give you a feel for what cloud formation looks like, here is a snippet of my configuration:

{

  "AWSTemplateFormatVersion": "2010-09-09",
  "Description" : "MyCompany - Formation of Batch Machine",
  "Parameters" : {
     "EnvName" : {
        "Type" : "String",
        "Default" : "Batch",
        "Description" : "Enter environment name"
     },
    ...
  },

  "Mappings" : {
     "AMIMap" : {
        "us-west-2" : { "64" : "ami-1234c" }
     },
     "SubnetMap" : 
     ...
     },
     "AvailabilityZoneMap" : {
        "Staging"   : { "availabilityZone" : [ "us-west-2a", "us-west-2b", "us-west-2c" ] },
     ...
     },
     "SecurityGroupMap" : {
        "Staging"   : { "sg" : "sg-d143acbe" },
     ...
     },
     "KeyPairMap" : {
        "Staging"   : { "kp" : "MyKeyPair" },
        ...
     }
  },

  "Resources" : {
     "BatchAutoScalingGroup" : {
        "Type" : "AWS::AutoScaling::AutoScalingGroup",
        "Properties" : {
          "AvailabilityZones" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "EnvType" }, "availabilityZone"] },
          "VPCZoneIdentifier" : { "Fn::FindInMap" : [ "SubnetMap", { "Ref" : "EnvType" }, "subnet"] },
          "LaunchConfigurationName" : { "Ref" : "BatchLaunchConfiguration" },
          "MinSize" : 1,
          "MaxSize" : 1,
          "Tags" : [
           { "Key" : "Name", "Value" : { "Fn::Join" : [ "-", [ { "Ref" : "TeamName" }, { "Ref" : "EnvType" }, { "Ref" : "EnvName" }, { "Ref" : "BuildNumber" }  ]] }, "PropagateAtLaunch" : "true" },
           { "Key" : "Environment", "Value" : { "Ref" : "EnvName" }, "PropagateAtLaunch" : "true" }
           ]
        }
     },

     "BatchLaunchConfiguration" : {
        "Type" : "AWS::AutoScaling::LaunchConfiguration",
        "Properties" : {
          "ImageId" : { "Fn::FindInMap" : [ "AMIMap", { "Ref" : "AWS::Region" }, "64" ]},
          "InstanceType" : { "Ref" : "InstanceType" },
          "KeyName" : { "Fn::FindInMap" : [ "KeyPairMap", { "Ref" : "EnvType" }, "kp" ]},
          "AssociatePublicIpAddress" : "true",
          "SecurityGroups" : [ { "Fn::FindInMap" : [ "SecurityGroupMap", { "Ref" : "EnvType" }, "sg" ]} ],
          "UserData" : {
           "Fn::Base64" : {
              "Fn::Join" : ["\n", [
                 "#!/bin/bash -x",
                 "exec >& /var/log/cloud-output.log",
                 "mkdir -p /usr/local/company",
                 { "Fn::Join": [ " \\\n", [
                    "curl -s --retry 30",
                    { "Ref": "UserDataS3URL" },
                    "| type=batch",
                    { "Fn::Join": [ "=", [ "envtype", { "Ref" : "EnvType" } ] ] },
                    { "Fn::Join": [ "=", [ "envname", { "Ref" : "EnvName" } ] ] },
                    { "Fn::Join": [ "=", [ "buildnumber", { "Ref" : "BuildNumber" } ] ] }
                    "bash"
              ]]}
           ]]}}
        }
     },

     "BatchProfile" : {
        "Type" : "AWS::IAM::InstanceProfile",
        "Properties" : {
           "Path" : "/",
           "Roles" : ["BackendServer"]
        }
     }
  }
}

I will not go into all the features, but you have an option to store parameters in a 2 level map. You have an option to get external parameters. So you can setup your machines according to machine type and location via the map with parameters. There are methods to lookup information from the map like Fn::FindInMap.

So you can easily define what the base image is, what the security group and other aws parameters.

As for os configuration, cloud formation does support basic configurations on the machine for users, simple yum installations. For this you can use the init section. But for things that are just a bit more complicated, like installing java this will not work.

My Solution

The solution that I found to work best, is to use what cloudformation has to offer and then add user scripts.

So with cloud formation I assign an image that i created (the image is a basic image with a little software on it as possible. With cloud formation you can define the size of the machine, the location and all other physical aspects of the machine.

Another backdoor that cloud formation gives you is to run a user data script (a service from EC2 layer).

What is the magic in cloud formation? What it actually does is split your machine into two parts. There is the os partition and the EBS. Any time the machine goes down cloud formation will create a new machine with a clean os and then run all the scripts and upgrades from cloud formation. As for the EBS it depends if you choose to have to kept also after termination of the machine.

UserData

From the template above you can see that after the machine is booted, cloudformation will run a script that sits on S3. This script is a bash script that you can do anything you want with it. You can pass into the script parameters that the cloud formation has, so that the script can take the specific configuration into consideration.

So for example I can create an environment parameter and use it in the machine host name.

Applications and frameworks

As mentioned above I need to install frameworks and software packages. Basic installations that are part of the "os" I added to the base image - for instance supervisor. The configuration file I will add later on.

Each application (kafka …) I upload to S3 with the latest information.

I have a lot of configuration files to update per product that might change, so I have all the configuration files in a separate zip file that I also upload to S3.

We need to remember to give access to the instance to the area of S3 in the cloud formation configuration.

Then for example I will download the file from S3:

wget -O ${homedir}/ops.zip https://s3-us-west-2.amazonaws.com/my-files/ops.zip

From here I can unzip the file, copy, transform or do anything else I need to do.

IP Addresses

There are times that you might want to have a specific Static IP address for your machine. As specified above if you machine is a standalone EC2 you can do it from cloud formation. But if your machine is part of the scaling group then you cannot.

What you can do is run scripts from within the machine. What needs to be remembered is that external IP addresses are not on the actual machine. What amazon does is by using a DNS it routes the traffic to your machine. SO to get information and to update the configuration you need to use the amazon cli.

First your instance id (for more information see metadata):

INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`

The static ip that you want to set needs to be in the map area or externally via the parameters. So lets say:

ELASTIC_IP=a.b.c.d

You can now use the following command:

aws ec2 associate-address --instance-id $INSTANCE_ID --public-ip $elasticip

This will send a request to aws to associate the elastic ip with the instance.

For a more robust solution for a pool if elastic ips see ipassign script.

Java

The installation of java is more complicated. There is the option for a manual installation and to set the JAVA_HOME for the application. But this option is not the standard option. So I will share my script that will properly install java:

JAVA_MINOR=111
JAVA_VERSION=8u${JAVA_MINOR}
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/${JAVA_VERSION}-b14/jdk-${JAVA_VERSION}-linux-x64.tar.gz"
tar -zxvf jdk-${JAVA_VERSION}-linux-x64.tar.gz -C /opt
rm jdk-${JAVA_VERSION}-linux-x64.tar.gz
cd /opt/jdk1.8.0_${JAVA_MINOR}
update-alternatives --install /usr/bin/java java /opt/java/jdk1.8.${JAVA_MINOR}/bin/java 100
sudo alternatives --set java /opt/java/jdk1.8.111/bin/java
export JAVA_HOME=/opt/java/jdk1.8.${JAVA_MINOR}/
export JRE_HOME=/opt/java/jdk1.8.${JAVA_MINOR}/jre
rm /opt/jdk
ln -s /opt/jdk1.8.0_${JAVA_MINOR} /opt/jdk

Getting our latest application

Frameworks do not get updated all the time. But how do I upgrade my application on all the servers and also make sure when a switchover occurs that the machine gets the proper version. The best solution would be of course to use ansible. But we were looking for a simple solution with the devops. All our artifacts are published to our nexus machine. So all I need to do is to know the version of the jar and I can donwload it from the nexus. One option would be to add it the cloud formation. The disadvantage of this is that for every build I need to update the stack. So the solution we went with was a file on S3. Every time our build machine (jenkins) runs, it will publish a file on S3 with the current version of the jar. In addition it will run a script on our machine to download this jar from the nexus and install it.

On the other hand when we create a machine from scratch using the cloud formation, we will use the file on S3 to know what the version of the jar is.

Debugging

The hard part is how do you know what went wrong with your script?

Cloud formation itself writes a log file on the machine in the /var/log directory. But it does not write the output from the user data file.

To do this you need to do it yourself. There are some options that you can find at ec2-user-data-output. The solution is used in the end is:

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
# Echoes all commands before executing.
set -o verbose

CI Integration

We need to make sure that our solution is repeatable. So with cloud formation you can load the template from the GUI or via API.

We use jenkins as our build machine so we use the amazon plugin. This way you input the parameters value via the jenkins job and then your stack is created.

Summary

Though not as good as I thought it would be, you can build a fully automated cloud formation on amazon using the cloud formation template with userdata, on top of jenkins.