Luntbuild cluster issues
Pontis required a Luntbuild cluster for managing their builds. Can't blame them for that, its their right... There was just a tiny little problem - Luntbuild doesn't support clustering.
Now, this all happened a long time before I got to Tikal, so I'm not sure which of you geniuses thought of the idea to cluster the Quartz implementation which sits under Luntbuild instead of the Luntbuild itself, but I sure wish it was me!
Anyway, this idea worked fine and everybody was happy for a while. But you know how it is in this country, things can't stay good for long, so when Luntbuild upgraded their Quartz version to 1.6 the system stopped working. Actually, it kept working, but at any given time only one node in the entire cluster was actually working on a build.
I tried to reproduce the problem localy by configuring a luntbuild (Quartz) cluster using a clean alm installation. But the cluster I built just insisted on working fine, with concurrent builds flying all over the place...
After some digging around, I noticed a few differences between my local Luntbuild+Quartz db schemas and the one in Pontis. In a test env that was set up in Pontis, we recreated the DBs with new schemas and imported the data from the old Luntbuild database. Looks like this solved the problem, and the Pontis test servers are now running concurrent builds on different nodes.
I've added a step by step manual for replacing the Luntbuild+Quartz databases, along with sql scripts that can be used to build the new databases.
This manual can also be used for the general purpose of configuring a Luntbuild cluster in existing environments (adding more nodes to 1 existing node). Just remember to also edit the 'luntbuild.hbm.xml' file, so that the ID column in every table will be generated nativley by the database. Also, you sould configure the quartz.properties file to work in cluster mode (more information can be found here java-x.blogspot.com/2006/12/using-quartz-scheduler-in-cluster.html). All Luntbuild instances should work with one shared database (same goes for all Quartz instances).