There's a solution to this in the shape of Windows Job objects, and, just by chance, I happen to have been reading about them this week in Richter's new release. A Job object allows you to group together processes and it knows when new child processes are started and it can kill all processes in the job in one go, or you can wait on all processes in the job to terminate, etc. Unfortunately it's not easy to add support for launching CC.Net tasks in a job as jobs currently aren't exposed by .Net (they've only been around since Windows 2000, after all...).

One of the nice thing about jobs is that the notifications from them come in via an IO completion port. This means that you need 0 more threads than you already have to monitor a job (assuming you already have at least one thread monitoring an IOCP). I took my usual approach and spent some time putting together some simple classes to make job management easier and to learn how the API works. Once I had a job object that I could use to manage a set of processes and which I implemented my IWaitable interface so that I could wait for the termination of all of the processes within the job I decided it might be nice to be able to launch and monitor a process and capture its output and error streams all without using a single extra thread in the process that was doing the management...

The standard way to capture a process' standard output and standard error is detailed pretty well in this article, but the problem that I have with that is that it uses an anonymous pipe and you can't issue overlapped reads on anonymous pipes which means you need a thread to read the pipes and that spoils my scalability. However, as the documentation for CreatePipe() states, "Anonymous pipes are implemented using a named pipe with a unique name. Therefore, you can often pass a handle to an anonymous pipe to a function that requires a handle to a named pipe." so I set about writing a new implementation of CreatePipe() that allows the pipe to be used with overlapped I/O.

The result is something like this:

CProcessOutputPipe::CProcessOutputPipe(
   const DWORD bufferSize,
   const Milliseconds timeout)
   :  m_name(GenerateName()),
      m_pipeServer(CreateServerPipe(m_name, bufferSize, timeout)),
      m_pipeClient(CreateClientPipe(m_name))
{
   Connect(timeout);
}
  
CProcessOutputPipe::CProcessOutputPipe(
   const _tstring &name,
   const DWORD bufferSize,
   const Milliseconds timeout)
   :  m_name(name),
      m_pipeServer(CreateServerPipe(name, bufferSize, timeout)),
      m_pipeClient(CreateClientPipe(name))
{
   Connect(timeout);
}
  
void CProcessOutputPipe::Connect(
   const Milliseconds timeout)
{
   COverlappedWithEvent overlapped;

   if (0 == ::ConnectNamedPipe(m_pipeServer, &overlapped))
   {
      const DWORD lastError = ::GetLastError();
      
      if (lastError == ERROR_IO_PENDING)
      {
         if (!overlapped.Wait(timeout))
         {
            throw CException(
               _T("CProcessOutputPipe::CProcessOutputPipe()"), 
               _T("Failed to connect client and server ends of the pipe."));
         }
      }
      else if (lastError != ERROR_PIPE_CONNECTED)
      {
         throw CWin32Exception(
            _T("CProcessOutputPipe::CProcessOutputPipe() - ConnectNamedPipe"), 
            lastError);
      }
   }
}
  
CProcessOutputPipe::~CProcessOutputPipe()
{
}
  
HANDLE CProcessOutputPipe::DetachWriteHandle()
{
   return m_pipeClient.Detach();
}
  
HANDLE CProcessOutputPipe::DetachReadHandle()
{
   return m_pipeServer.Detach();
}
  
static _tstring GenerateName()
{
   return CreateGUIDAsString();
}
  
static HANDLE CreateServerPipe(
   const _tstring &name,
   const DWORD bufferSize,
   const Milliseconds defaultTimeout)
{
   const _tstring fullName = _T("\\\\.\\pipe\\") + name;
  
   CSecurityAttributes securityAttributes(false);
  
   CSmartHandle handle(::CreateNamedPipe(
      fullName.c_str(),
      PIPE_ACCESS_INBOUND |      // data flow from client to server
      FILE_FLAG_OVERLAPPED,      // overlapped mode 
      PIPE_TYPE_BYTE |           // byte-type pipe 
      PIPE_READMODE_BYTE |       // byte-read mode 
      PIPE_WAIT,                 // blocking mode 
      1,                         // number of instances 
      0,                         // output buffer size - no data flows this way
      bufferSize,                // input buffer size 
      defaultTimeout,            // client time-out 
      &securityAttributes));     // dont allow handle inheritance
  
   if (handle == INVALID_HANDLE_VALUE)
   {
      const DWORD lastError = ::GetLastError();
  
      throw CWin32Exception(
         _T("CProcessOutputPipe::CreateServerPipe()"), 
         lastError);
   }
  
   return handle.Detach();
}
  
static HANDLE CreateClientPipe(
   const _tstring &name)
{
   static const _tstring server = _T(".");
  
   const _tstring fullName = _T("\\\\") + server + _T("\\pipe\\") + name;
  
   CSecurityAttributes securityAttributes(true);
  
   CSmartHandle handle(::CreateFile( 
         fullName.c_str(),   
         GENERIC_WRITE, 
         0,                      // no sharing 
         &securityAttributes,    // allow handle inheritance
         OPEN_EXISTING,          // opens existing pipe 
         FILE_FLAG_OVERLAPPED,   // default attributes 
         NULL));                 // no template file 
  
   if (handle == INVALID_HANDLE_VALUE)
   {
      const DWORD lastError = ::GetLastError();
  
      throw CWin32Exception(
         _T("CProcessOutputPipe::CreateClientPipe() - CreateFile"), 
         lastError);
   }
  
   DWORD dwMode = PIPE_READMODE_BYTE; 
  
   if (0 == ::SetNamedPipeHandleState( 
      handle,   // pipe handle 
      &dwMode,  // new pipe mode 
      NULL,     // don't set maximum bytes 
      NULL))    // don't set maximum time 
   {
      const DWORD lastError = ::GetLastError();
  
      throw CWin32Exception(
         _T("CProcessOutputPipe::CreateClientPipe() - SetNamedPipeHandleState"), 
         lastError);
   }
  
   return handle.Detach();
}

Once the write end of this pipe is wired up to a process' standard output or error stream and an instance of CAsyncFileReader() is plugged in on the read end I can monitor the process and read the pipe using the threads in my IOCP's pool. Firing off a process and capturing/processing its output is now reliable (I can terminate it and any processes it creates) and scalable (I don't need any new threads in my process to monitor a new process tree).

These patches are 'supporting patches' to the main ProjectTrigger and integrator changes, I thought I'd start with the simple ones... None of these patches should change existing functionality, all require new, optional, properties to be specified to activate the new code. All patches include updated and or new tests.

1) IntervalTrigger - Added initialIntervalSeconds.

initialIntervalSeconds is a new property that controls when the trigger fires the first time. The idea being that you might want to have a trigger that fires every 'X' seconds but you dont want to have to wait a full 'X' seconds for it to fire when the server first starts up. If initialIntervalSeconds is set then the trigger will first fire after this number of seconds and will then switch to firing every intervalSeconds as normal.

I find this useful when running ccnet on machines which aren't dedicated build machines (i.e. machines that don't have ccnet running all the time) for projects that I don't want to run very often but, when I start ccnet, I want them to be checked...

ccnet-3607-IntervalTrigger-InitialInterval-Patch.txt

2) IFileSystem changes - Added some more functionality to support the tests for the changes in patches 3 and 4.

Added CreateDirectory() and DeleteDirectory().

ccnet-3607-IFileSystem-Patch.txt

3) CVS source control - allow non existent working directory to be created.

Changes to DoesCvsDirectoryExist() and NewProcessInfoWithArgs() to allow for creation of working directory if required for checkout or update only.

ccnet-3607-CVS-CreateWorkingDirectory-Patch.txt

4) FileSourceControl - allow working directory to be specified and added a property to force deletion of remove files during GetSources.

I've added a working directory property so that you can have a FileSourceControl source control block that checks files out to a sub directory of the project's working directory. This is useful if you are accumulating files from multiple source control blocks into a directory tree and want to place some of the files in subdirectories.

The second change forces the working directory to be removed before an update occurs. This is a brute force method of removing files which no longer exists in the repository from the working copy.

ccnet-3607-FileSourceControl-Patch.txt

5) Visual C++ version 6 "MsDev" build task.

Added a new build task that works with Visual studio 6.

ccnet-3607-MsDevTask.cs
ccnet-3607-MsDevTaskTest.cs

6) Allow you to specify which version of Visual Studio is used if mutliple versions are installed without having to know the executable path. Although you can currently select the version of visual studio that is used you need to do it by providing the full path to the executable this patch allows you to simply say "I'd like Vs2003 please" and then used the registry query code to work out where it is installed. This is enabled by specifying a Version property with a value of one of the following: "9.0", "8.0", "7.1", "7.0", "VS2008", VS2005", "VS2003" or "VS2002". I considered, and rejected, having the DevenvTask read the solution file and work out the appropriate version as unecessarilly complex.

This removes the need to embed potentially machine specific information in a config file just to select which version of VS you want to use; though, unfortunately it doesn't help me on Vista x64 as the InstallDir value isnt present in the registry :(

ccnet-3607-DevEnvTask-VersionSelect-Patch.txt

With any luck these will be approved and incorporated into the subversion ccnet repository soon, but, if they're not you can get them from here.

I'm currently working on tidying up the ProjectTrigger changes that are mentioned here and these will be released later. The Sleep() changes to the ProjectIntegrator were more complex than expected (mainly due to the design of the integration thread), although my hacked version works for me it's not good for prime-time so I'm working on a better fix.

In the ProjectIntegrator class there's a Run() method (because, of course, all projects have their own threads...) and at the bottom of the run loop there's this piece of code Thread.Sleep(100); along with, what I assume is a wishful thinking comment... Of course with 100s of projects (each with its own thread) a 100ms sleep from each of them before they go off and do busy work is a nightmare scenario. Every thread wakes up and gets scheduled every 100ms... No wonder the system staggers around like a drunken monkey and brings even beefy development boxes to their knees.

The strange thing is that the project knows how long it is until it needs to check again (and run through the loop again). The triggers that trigger the project have a "next time to run" time set on them, the project just doesn't bother to query them and use that as the basis of how long it should go to sleep for... A five minute hack job and my version of CC.Net does this and I'm able to run my 'all compilers' build again. It's still not the kind of performance that I'd like from a system that's basically doing nothing (albeit lots of nothing) but it's considerably better than what went before...

The next target on my search for 'the bleeding obvious' (I think 'low hanging fruit' is the polite term...) is the corresponding 200ms sleep that happens during a busy loop when the project is in the integration queue... I expect that I'll fix it in a similar way for now, but, really, it should be waiting on a request to terminate the thread (and not just a boolean flag, this is what auto reset events are for!) and, likewise, waiting on the project being removed from the queue...

The good news is that I now have an integration system that works, it can build, test and deploy libraries in such a way that dependent libraries and applications are rebuilt as new versions of deployed artifacts become available. The config file generator can generate files for all of the build machines that I'm currently using and builds projects per machine, per compiler and per branch so that I can have several compilers building several branches on several machines and can see what's what from the CCTray application (it would be nice if that allowed you to group projects...).

The bad news is that CruiseControl.Net just isn't really designed for this kind of scalability... I have lots of projects (each library consists of 4 projects (build, build tests, run tests and deploy)), my most basic server application needs 6 libraries and the exe projects, so that's almost 30 CC.Net projects (per branch, per compiler, and (optionally) per architecture). Adding a server that doesn't use any new libraries means 2 more projects (servers have build and test projects)...

It's better than not having an integration system but on my old dev box which does the builds for VC6, VS2002, VS2003 and VS2005 I currently have a CC.Net server running with 190 threads which is thrashing the box with pointless busy work as it polls itself to determine if the projects need to be rebuilt. My main x64 dev box is almost as bad, it's building for VS2005 and VS2008 for both x86 and x64 (so total projects X 4 for each branch that I want to build)... And this is with the whole system building using two queues, so there's only ever two concurrent actions in progress (a build and a test run can run at the same time).

On the whole I'm a bit disappointed in CruiseControl.Net. I'm sure it works very nicely for simple situations, such as where you pull everything out of your repository and build it with a single project, but, when you're trying to do more complex things it seems to be a bit fragile. Part of this fragility is probably due to me; I sort of assume things should work how I'd expect them to work ;) So, for example, when I saw that there was support for combining triggers in 'and' and 'or' chains I assumed that I could do that with all of the other triggers... However, it seems that if you try and combine project triggers then you don't really get what you want; project triggers reset themselves after they've 'fired' rather than waiting until they're told that an integration has been completed (which is when some of the other triggers reset themselves...). So, if you have several project triggers ANDed together then you might find that you're waiting a hell of a long time for all of them to fire at the same time... And since the "multiple trigger" trigger applies a "C programmer predictability" to how it deals with its ANDing (it short-circuits, so if the first trigger hasn't fired yet then the rest don't get called at all), but it doesn't do the same to how it works out when its triggers will next change state (it scans all of the triggers and takes the lowest of the "when are you due to go off next" times), you can find that your dashboard is showing that some of your projects will next be built in the past... Oh, and project triggers only fire when a project changes state, so if you're waiting for project A to become successful and it does then you'll run, but if the server has just started and it's already successful then, well, you'll wait until it builds again...

• My code tends to be broken up into separate libraries depending on function, so I have a series of libs that I link with any exe project that I'm building.
• Each library project depends on 0 or more other library projects.
• Each library project has a test harness, which builds as an exe from a different project, and there also tend to be mock libraries which provide mocks for the interfaces that the library contains. A library's test may or may not use the library's mocks, but it generally does.
• I don't want to overly stress my CVS server.
• My code is structured as CVS modules, but CVS is a bit broken with respect to doing an update on a module that is in a subdirectory of the repository if you decide to tell it to 'create any missing directories' with the -d option.
• I'd like the code to build in isolation, that is, with only the dependencies that I think it should actually have.
• I don't want to overly stress my build machine, after all, it could be the machine I'm working on, or a virtual machine running on the machine I'm working on, etc.

What I've ended up with is a series of CruiseControl projects per library. Most libraries have a Build, Build Tests, Run Tests and Deploy project. There's a central "Get Sources" project that grabs the whole of the directory structure where all of the libraries live, this prevents/works around the problem that CVS has with updating modules in isolation. Since this project gets a whole tree I then use dependent projects which trigger off of the "Get Sources" project to copy just the parts of the tree that each project needs. These use the file system source control provider.

This is where I had to make my first changes. By default, with the build of the code that I have, CruiseControl.Net 1.3.0.2918, has a file system source control provider that can only copy to your project's working directory. So, if you want to chain a series of file system source control providers together and have them move files into a series of sub directories then you're out of luck... This is an easy fix, just give the provider its own working directory property and 'do the right thing' when you decide where to copy your files. My next problem was that combining a file system source control provider with the CVS provider didn't work quite how I wanted it to. When the CVS provider updates its tree it fiddles with the timestamps on the files in the CVS directories, the file system source control provider considers that a 'change' and therefore tells CruiseControl.Net that the source is modified, which triggers a build every time the CVS dump is updated... A quick hack so that the file system provider ignores anything in a directory called CVS and things start to work how I'd like them to (and yes, this can be generalised into a property with a string of 'ignore directories'). Finally there's the problem that, when it has decided that there are changes and is then asked to update from the repository, the file system provider just copies everything in its source directory into its destination directory. This, effectively, merges the new directory with the old; if anything has been deleted from the new directory it's still in the updated directory... Right now the fix is to delete the target directory before updating (but after checking for changes!), this works but causes more code to be recompiled on a change; I've made it optional.

So, now I have a source control system that works; I pull a directory out of my repository and have a project that gets updates for this as they happen. I then have dependent projects that use file system providers to get just the bits that they need into their build directories. This enables each library to build with only the code that should be available.

At around this point the aforementioned holes in the multiple trigger and the project trigger come to light. More changes to the code leave me with a multiple trigger that, er, well, works, and a project trigger that is much more configurable. For starters I want to restrict when things try to build to when they're likely to be able to build. This means that if project A depends on project B and C then it shouldn't even try to build until project B and C have both built successfully. However, once they have built then changes to either should cause A to rebuild. This isn't possible with CruiseControl.Net out of the box. A change to the project trigger so that it can be set to trigger if the project has ever built successfully means that the first part of project A's trigger can be an ANDing together of project B and C being available. Next we can and that trigger with a trigger which looks for changes in either the 'Get Sources' project, A or B. This gives the required amount of control; A doesn't ever get scheduled to build until B and C are built, but once they are it will rebuild if new source arrives or if A or B rebuilds... Oh, and then there's the "project triggers don't trigger on the first integration of the projects that they monitor" issue... Er, mine can, enough said. This was a bit of a killer really, it was another "if the server was shutdown before you built but all your dependencies were built then you'll never built until they're changed" issues...

By this stage I have some code that writes the project files based on some hard coded dependencies; I've hand crafted it to build some of the base libraries to make sure that things work. Once things work, and we're nearly there, then I'll add in the Visual Studio project parsing code and have build the CruiseControl projects from the Visual Studio projects. Since each step is quite hard to test I've found myself doing clean starts of the integration server quite a lot. This led to another change, the file state object requires that the directory in which you're storing state exists; for me that's a pain, I like to blow away everything from a single root and then run the server again and have it 'just work' and create all of the directories that it needs. My file state object doesn't barf if the target directory doesn't exist, it creates it.

It was at this point that I began to wish that the CruiseControl.Net "pluggin" architecture was a bit more "plug in" rather than "plugged in". It seems that all of the various 'providers' and 'tasks' etc are all compiled into the same assembly. Sure you can implement another, but I'd prefer to be able to do that in my own assembly so that I can, er, well, plug it in. I'd be more than happy for someone to correct me here and tell me to go and read some web page to find out how to do it...

My final change was just for convenience. My "Get Sources" project runs every 10 mins, however, I don't really like the fact that when I start the server up it takes 10 mins before it runs this task for the first time. So, I added a property to the IntervalTrigger that allows you to specify a different 'initial timeout'.

I'm not quite where I want to be, but I'm a lot closer. I have code that generates my CruiseControl projects, which makes changing things easy, it has already allowed me to generate different projects for different compilers and to optionally have the x64 builds build and test as well as the Win32 builds. I've found quite a few build bugs, as it's becoming easier to build everything in every way each time I change anything. This is a good thing. I just wish that CruiseControl.Net had done it all straight out of the zip...

I've been playing with CruiseControl.Net and it works reasonably well and since the source is available I've been able to patch up a couple of things that didn't work quite how I needed... I can see that I'm going to need to write something that writes a config file for me as most of the examples will have similar project definitions and I figure I should manage the duplication in one place... It's a pity that some nodes can't logically 'cascade' down the XML tree...

So far my project structure has been giving me a few problems but I've fixed them by hacking the cruise control code (oh the double edged sword of open source software)... The first issue was that the CVS source countrol provider required that the directory that you wanted to check out into (its working directory) existed. Probably not a problem most of the time but I wanted to run multiple CVS source control providers and have them check out different modules into different directories. Next the file system source code provider would only copy into the main project working directory; likewirse I needed to chain multiple ones of these together and so wanted each one to copy into a directory that I could specify...