Fighting with CruiseControl.Net...

CCNet, Continuous Integration, Geek Speak

I’ve been trying to get my code to build with CruiseControl.Net this week. It’s taken longer than I’d hoped, but I’m almost there. It became easier when I switched from assuming various parts of CruiseControl.Net would “work as I expected them to” to assuming that I’d have to delve into the source and change things…

On the whole I’m a bit disappointed in CruiseControl.Net. I’m sure it works very nicely for simple situations, such as where you pull everything out of your repository and build it with a single project, but, when you’re trying to do more complex things it seems to be a bit fragile. Part of this fragility is probably due to me; I sort of assume things should work how I’d expect them to work ;) So, for example, when I saw that there was support for combining triggers in ‘and’ and ‘or’ chains I assumed that I could do that with all of the other triggers… However, it seems that if you try and combine project triggers then you don’t really get what you want; project triggers reset themselves after they’ve ‘fired’ rather than waiting until they’re told that an integration has been completed (which is when some of the other triggers reset themselves…). So, if you have several project triggers ANDed together then you might find that you’re waiting a hell of a long time for all of them to fire at the same time… And since the “multiple trigger” trigger applies a “C programmer predictability” to how it deals with its ANDing (it short-circuits, so if the first trigger hasn’t fired yet then the rest don’t get called at all), but it doesn’t do the same to how it works out when its triggers will next change state (it scans all of the triggers and takes the lowest of the “when are you due to go off next” times), you can find that your dashboard is showing that some of your projects will next be built in the past… Oh, and project triggers only fire when a project changes state, so if you’re waiting for project A to become successful and it does then you’ll run, but if the server has just started and it’s already successful then, well, you’ll wait until it builds again…

Of course, the good thing about Open Source is that you have the source and you can change things. Well, at least I can change things… I’ve been changing things…

I admit that the way I’m trying to structure my build is probably a bit complex.

  • My code tends to be broken up into separate libraries depending on function, so I have a series of libs that I link with any exe project that I’m building.

  • Each library project depends on 0 or more other library projects.

  • Each library project has a test harness, which builds as an exe from a different project, and there also tend to be mock libraries which provide mocks for the interfaces that the library contains. A library’s test may or may not use the library’s mocks, but it generally does.

  • I don’t want to overly stress my CVS server.

  • My code is structured as CVS modules, but CVS is a bit broken with respect to doing an update on a module that is in a subdirectory of the repository if you decide to tell it to ‘create any missing directories’ with the -d option.

  • I’d like the code to build in isolation, that is, with only the dependencies that I think it should actually have.

  • I don’t want to overly stress my build machine, after all, it could be the machine I’m working on, or a virtual machine running on the machine I’m working on, etc.

What I’ve ended up with is a series of CruiseControl projects per library. Most libraries have a Build, Build Tests, Run Tests and Deploy project. There’s a central “Get Sources” project that grabs the whole of the directory structure where all of the libraries live, this prevents/works around the problem that CVS has with updating modules in isolation. Since this project gets a whole tree I then use dependent projects which trigger off of the “Get Sources” project to copy just the parts of the tree that each project needs. These use the file system source control provider.

This is where I had to make my first changes. By default, with the build of the code that I have, CruiseControl.Net 1.3.0.2918, has a file system source control provider that can only copy to your project’s working directory. So, if you want to chain a series of file system source control providers together and have them move files into a series of sub directories then you’re out of luck… This is an easy fix, just give the provider its own working directory property and ‘do the right thing’ when you decide where to copy your files. My next problem was that combining a file system source control provider with the CVS provider didn’t work quite how I wanted it to. When the CVS provider updates its tree it fiddles with the timestamps on the files in the CVS directories, the file system source control provider considers that a ‘change’ and therefore tells CruiseControl.Net that the source is modified, which triggers a build every time the CVS dump is updated… A quick hack so that the file system provider ignores anything in a directory called CVS and things start to work how I’d like them to (and yes, this can be generalised into a property with a string of ‘ignore directories’). Finally there’s the problem that, when it has decided that there are changes and is then asked to update from the repository, the file system provider just copies everything in its source directory into its destination directory. This, effectively, merges the new directory with the old; if anything has been deleted from the new directory it’s still in the updated directory… Right now the fix is to delete the target directory before updating (but after checking for changes!), this works but causes more code to be recompiled on a change; I’ve made it optional.

So, now I have a source control system that works; I pull a directory out of my repository and have a project that gets updates for this as they happen. I then have dependent projects that use file system providers to get just the bits that they need into their build directories. This enables each library to build with only the code that should be available.

At around this point the aforementioned holes in the multiple trigger and the project trigger come to light. More changes to the code leave me with a multiple trigger that, er, well, works, and a project trigger that is much more configurable. For starters I want to restrict when things try to build to when they’re likely to be able to build. This means that if project A depends on project B and C then it shouldn’t even try to build until project B and C have both built successfully. However, once they have built then changes to either should cause A to rebuild. This isn’t possible with CruiseControl.Net out of the box. A change to the project trigger so that it can be set to trigger if the project has ever built successfully means that the first part of project A’s trigger can be an ANDing together of project B and C being available. Next we can and that trigger with a trigger which looks for changes in either the ‘Get Sources’ project, A or B. This gives the required amount of control; A doesn’t ever get scheduled to build until B and C are built, but once they are it will rebuild if new source arrives or if A or B rebuilds… Oh, and then there’s the “project triggers don’t trigger on the first integration of the projects that they monitor” issue… Er, mine can, enough said. This was a bit of a killer really, it was another “if the server was shutdown before you built but all your dependencies were built then you’ll never built until they’re changed” issues…

By this stage I have some code that writes the project files based on some hard coded dependencies; I’ve hand crafted it to build some of the base libraries to make sure that things work. Once things work, and we’re nearly there, then I’ll add in the Visual Studio project parsing code and have build the CruiseControl projects from the Visual Studio projects. Since each step is quite hard to test I’ve found myself doing clean starts of the integration server quite a lot. This led to another change, the file state object requires that the directory in which you’re storing state exists; for me that’s a pain, I like to blow away everything from a single root and then run the server again and have it ‘just work’ and create all of the directories that it needs. My file state object doesn’t barf if the target directory doesn’t exist, it creates it.

It was at this point that I began to wish that the CruiseControl.Net “pluggin” architecture was a bit more “plug in” rather than “plugged in”. It seems that all of the various ‘providers’ and ’tasks’ etc are all compiled into the same assembly. Sure you can implement another, but I’d prefer to be able to do that in my own assembly so that I can, er, well, plug it in. I’d be more than happy for someone to correct me here and tell me to go and read some web page to find out how to do it…

My final change was just for convenience. My “Get Sources” project runs every 10 mins, however, I don’t really like the fact that when I start the server up it takes 10 mins before it runs this task for the first time. So, I added a property to the IntervalTrigger that allows you to specify a different ‘initial timeout’.

I’m not quite where I want to be, but I’m a lot closer. I have code that generates my CruiseControl projects, which makes changing things easy, it has already allowed me to generate different projects for different compilers and to optionally have the x64 builds build and test as well as the Win32 builds. I’ve found quite a few build bugs, as it’s becoming easier to build everything in every way each time I change anything. This is a good thing. I just wish that CruiseControl.Net had done it all straight out of the zip…