Creating self starting worker machines for fun and profit
Lately, I needed to create worker machines (Machines that have Resque workers on them).
Usually, the process is very manual and tedious, you need to deploy the code to each machine (on code change), you need to start/stop/restart the workers and more.
Since I needed A LOT of workers, I really wanted to make the process automatic.
I wanted the machine to be in charge of everything it needs.
- Pull latest git code
- Copy file to the
- Startup god (which will start the Resque processes)
I wanted the machine to be in charge of all this when it boots up, so in case I need to update the code or something, all I need is to reboot the machines and that’s it.
Also, scaling up the process is super easy, all I need is to add more machines.
The logic behind it was also being able to use spot-instances on Amazon, which are much cheaper.
So, as always, I thought I would document the process and open source it.
What are the challenges we are looking at, when we want to do something like that.
- Awareness of RVM, Ruby version
- Awareness of Bundler and Gem versions.
Those are things we need to consider.
Breaking down the process into pseudo code.
So, let’s break down the process into pseudo code
1 2 3 4 5
For the last 2 parts of the process, if you try to do just
bundle install --deployment for example, the script will fail, it does not know what bundle is, it resolves to the default Ruby installed on the machine at this point.
Since want the last 2 processes to go through RVM, we need to create a wrapper script.
Creating RVM Wrappers
You can create RVM wrappers like this:
This will generate this file: (called
1 2 3 4 5 6 7 8 9 10
I did the same for bundle which generated this file: (called
1 2 3 4 5 6 7 8 9 10
The files are created in
Now that we have the wrappers, we made sure the boot up process will be aware of RVM and we can continue.
If you want something to run when the machine starts you need to get to know the
Usually, those files are located in
/etc/ folder (may vary of course)
The files are loaded and executed in a specific order
1 2 3
If you want something to run when ALL the rest of the machine bootup process ran, you put it at the end of the rc.local file.
One gotcha here is that the file must end with
exit 0, or your script will not run at all, and good luck finding out why :P
I put this line at the end, right before the
Now that I have the wrappers, I have the line in rc.local I can finalize the process with creating the shell script.
This is what the script looks for me
1 2 3 4 5 6 7 8
The god file to start the workers is standard.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82
Now, it’s a completely automatic process, all you need is to run as many machines as you need, once the machine is booted up, it will deploy code to itself, copy files, install gems and run workers.
If you want to restart the process, add machines or anything else, you can do it with the click of a button in the AWS UI.
Please feel free