I am intending to check out some of the other build servers that people have been suggesting, but today I was too busy with real work so I just left a cut down version of my latest CruiseControl.Net configuration running on one of my boxes and fixed a few issues whilst doing proper work most of the time... This evening I decided to go and look at why CruiseControl.Net scales so poorly and the first issue that I came across is the title of this blog posting...
ProjectIntegrator class there's a
Run() method (because, of course, all projects have their own threads...) and at the bottom of the run loop there's this piece of code
Thread.Sleep(100); along with, what I assume is a wishful thinking comment... Of course with 100s of projects (each with its own thread) a 100ms sleep from each of them before they go off and do busy work is a nightmare scenario. Every thread wakes up and gets scheduled every 100ms... No wonder the system staggers around like a drunken monkey and brings even beefy development boxes to their knees.
The strange thing is that the project knows how long it is until it needs to check again (and run through the loop again). The triggers that trigger the project have a "next time to run" time set on them, the project just doesn't bother to query them and use that as the basis of how long it should go to sleep for... A five minute hack job and my version of CC.Net does this and I'm able to run my 'all compilers' build again. It's still not the kind of performance that I'd like from a system that's basically doing nothing (albeit lots of nothing) but it's considerably better than what went before...
The next target on my search for 'the bleeding obvious' (I think 'low hanging fruit' is the polite term...) is the corresponding 200ms sleep that happens during a busy loop when the project is in the integration queue... I expect that I'll fix it in a similar way for now, but, really, it should be waiting on a request to terminate the thread (and not just a boolean flag, this is what auto reset events are for!) and, likewise, waiting on the project being removed from the queue...Of couse, what I'd really like to do is remove at least 90% of the threads and most of the polling and have the completion of a project actively trigger the projects that depend on it, but I think the simplistic design of CC.Net is too ingrained for me to achieve much in a reasonable time-frame.