/* Rambling comments... */

DevPartner Studio 10 - unmanaged disappointments

The story so far, I've owned DevPartner Studio for several years and been on a support contract the whole time. The support situation went a bit flakey when Compuware sold the product to Micro Focus, I expect that either I'm simply not a big enough client for them to care or their post sales and support is just no where near as good as the Compuware offering. With Compuware I'd be contacted when new versions came out and I had a named sales contact that I could also use for questions, etc. What's more the sales guy was good, he knew I wouldn't be buying more licenses, he remembered the issues I'd raised and he helped chase up responses etc.

Anyway, I whinged about how DevPartner lagged behind Visual Studio releases and was told, via a comment from someone at Micro Focus, that DevPartner 10 shipped at the same time as Visual Studio 2010; it would have been nice if my support contract meant that they'd bother to tell me about this since I was entitled to the latest version; heh, here's an idea, the software itself could work it out for me somehow (nah, nobody would do that)... Anyway, it seems that the way I'm supposed to work out that a new version that I'm entitled to under my support contract is available is to log on to the Micro Focus support site and 'request a new version'. Presumably I either poll this site every couple of months and request current version + 1 or I'm supposed to track when they release new versions... Interestingly I get a list of patches for 9.1 displayed but no indication that a 10 is available...

So, I fill in the form and wait a few days, no response, I chase them and eventually get an email with some download links. I download and install DevPartner 10 and run it up in "Error detection" mode on an unmanaged VS2010 solution which contains a relatively simple multi-threaded project...

During the build I get several compiler crashes and whilst it IS the Microsoft C++ compiler that is crashing here it has never crashed with this code if DevPartner instrumentation is NOT enabled... The errors are all similar:

1>e:\source\practicaltesting\jetbytetools\testtools\testmonitor.cpp(748): fatal error C1001: An internal error has occurred in the compiler.

1>  (compiler file 'f:\dd\vctools\compiler\utc\src\p2\main.c[0x524EDFBA:0x00000004]', line 183)

1>   To work around this problem, try simplifying or changing the program near the locations listed above.

1>  Please choose the Technical Support command on the Visual C++ 

1>   Help menu, or open the Technical Support help file for more information

And they all occur in these kinds of situations...

   }

   catch(const CSEHException &e)

   {

      cout << "Exception during test timeout processing: " << CStringConverter::TtoA(e.GetWhere() + _T(" - ") + e.GetMessage()) << endl;

   }

   JETBYTE_TESTS_CATCH_ALL_AT_THREAD_BOUNDARY_IF_ENABLED

   {

      cout << "Unecpedted exception during test timeout processing" << endl;

   }

Where the macro evaluates to catch(...) in this particular build.

So, my first impression of the new version is that it's just about as good as the first impression I had of 9.0, i.e. it simply doesn't work on the latest compiler with simple code. I don't know if I'm alone in thinking that when software that you're supposed to rely on for software quality assurance work gives this kind of first impression you tend to worry if it actually does what it needs to at all, rather than simply being a rushed to market mess which underwent too little testing before release so that it could be shipped at the same time as the new compiler without actually supporting it...

Anyway, I thought, perhaps, that by default there were too many error detection options turned on (I tend to use a small subset of the functionality as I've found that the more interesting stuff is prone to not work (API parameter checking is great until you realise that some of the (admittedly many) APIs that it covers are configured incorrectly and report problems that don't exist and you can't change them and have to wait for a service release...). Anyway, off I go to the options screen and find that I can't change the error detection options as "Error Detection settings are unavailable until you select a valid start up project"...

At this point I switched to VS2008 to see how things were back in the legacy compiler environment. This time everything builds OK but when I go to run the program under BoundsChecker I get a message telling me that DevPartner is unable to test 64bit projects. Fair enough; we're still in 'tool lag' country, DevPartner 9.1 could also only 'run on' x64 and not instrument x64 code; then the tool would also try and instrument the x64 builds but the link failure messages were fairly clear in what was wrong. It's a pity that the DevPartner 10 user interface isn't consistent on this, the VS2010 options screen obviously knows it's not supposed to let you work with x64 projects (though could be more helpful) yet the tool bars and build process let you try to build x64 projects rather than, more helpfully, preventing you...

Rebuilding the 32bit version of the code in VS2010 worked OK, but what a wonderful first impression...

Now there seem to be some new weird problem reports inside of the VS2010 STL code which I need to dig my way through to determine why they're OK to ignore...

And then I suppose I need to go and dig up all of the same information that they already have about my development machine so that I can ask them to send me a new license file...

More thoughts on invasive/intrusive containers and STL allocators

I'm still considering my options with regards to intrusive containers to replace the STL maps that I'm using in my timer queue implementation. I think I may be able to use the boost::intrusive sets in place of a true map in most of the situations that I need (I just have convince myself that a) it will work with all the compilers that I need to support, b) that adding a dependency to part of boost is a good idea for me and my clients and c) that even though my gut reaction on seeing the code is that it's pointlessly clever and bound to be a bitch to debug it's probably better than rolling my own).

I can't help but think that this whole issue would have never existed if the standard STL allocators had taken the value type as an argument to allocate(), this is likely not possible with things other than std::set and std::map BUT if these took the value type as a parameter then a custom allocator could simply locate the intrusive link structure in the value type and return that to the container. The same underlying map template could then work with either an allocator that allocates its node structure or an allocator that obtains the intrusive structure from the value type... Since it would have been this simple we'd have had the option of intrusive STL containers from day 0...

It's very tempting to hack this kind of change into a copy of the map/set/tree templates used in STLPort rather than writing my own...

CLR Hosting - A flexible, managed plugin system, part 2

Last time I explained how the managed side of my flexible hosting server architecture was structured. I dealt with the IDL required to generate the COM interfaces which were used to communicate between unmanaged code and managed code and detailed how the custom AppDomainManager object allowed me to load and execute managed code in such a way that it was easy to update the managed applications whilst the server was running.

This time I'll cover the unmanaged side of things. This is actually where much of the complexity lives. The code that actually manages the creation and destruction of the application domains in which the managed applications run and which deals with wiring them up to the network endpoints that are accepting connections for them.

Continue reading "CLR Hosting - A flexible, managed plugin system, part 2"

Useful Visual Studio retirement matrix

Here's a useful matrix which shows when each version of Visual Studio will become unsupported by Microsoft. I'm posting the link here as I'm sure I'll not be able to find it the next time I need it...

Practical Testing: 28 - Finishing the timer wheel

Previously on "Practical Testing"... I'm writing a timer wheel which matches the interface used by my timer queue. This new implementation is designed for a particular usage scenario with the intention of trading space for speed and improving performance of some reliable UDP code.

Over the last four entries I've implemented various parts of the timer wheel and adjusted the test code so that I could reuse the tests that I already had for the other implementations with my timer wheel. The tests needed to be tweaked quite a bit to take into account the different behavioural characteristics of the wheel and the queues, this was accomplished using traits which determine the detail of how the class under test interacts with its service providers (mainly a tick count provider).

Today we finally get to the point where we have a working timer wheel that is compatible with the interface used by the two timer queues. We can then look at the results of the performance tests and work out where we need to go next.

Continue reading "Practical Testing: 28 - Finishing the timer wheel"

Invasive containers

Rather than immediately dive into the fun of writing my own invasive alternative for std::map I decided to take a look at what has been done before, as expected boost contains something that might work in the shape of the "intrusive containers library".

Of course, being part of boost I first have to work out exactly how much more of boost it will require me to depend on and then I have to work out how I can use it to replace my current std::map usage. It seems quite clever (no surprise there) and allows for a type to be included in multiple intrusive containers by allowing an object to have multiple links embedded in it and allowing you to specify the link to use for a particular container. It seems slightly over engineered for my needs (again, no surprise there) and it's a pity that it doesn't simply provide obvious drop in replacements for the equivalent STL containers; yes it's nice that there are two kinds of base tree structure (red black and AVL) so you can optimise for your specific usage patterns but it would be nicer if you could simply switch from std::map to boost::intrusive::map and add an appropriate data member to the class you want to store in it...

Still, I guess I should spend some more time learning and understanding... One thing that it doesn't do for me, as far as I can see, is provide a multimap from which I can remove all elements at a specific key in one go...

STL allocators, hmm...

As I mentioned a while ago, I have some code which needs to perform better than it currently does and one of the areas that could be improved upon is the amount of contention for the heap that's occurring. The fact that I'm using an STL map for my collection means that the class has a 'big C' contention value of C(n threads using the heap) rather than C(n threads using the object). Of course, the fact that allocations need to be done at all is an unfortunate feature of std::map but rather than immediately replace the container with an invasive collection I decided to look into replacing the STL allocator that's being used with one that uses a private heap so that I could reduce the contention value of the allocations to C(n threads using the object). Doing this has required that I take a look at STL allocators and my initial thought is "hmm..."

Continue reading "STL allocators, hmm..."

Practical Testing: 27 - Fixing things...

Previously on "Practical Testing"... To deal with some specific usage scenarios of a piece of general purpose code I'm in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. Last time I left myself with a failing test. The problem is that setting a new timer on the timer wheel sets a timer that's relative to the time that timer wheel thinks is 'now' and the timer wheel's view of the current time could be slightly behind reality; see the previous entry for a diagram that explains the problem.

Continue reading "Practical Testing: 27 - Fixing things..."

.Net 4.0 different AppDomain managers for different AppDomains

Whilst looking through the latest documentation for the AppDomainSetup class I see that you can now specify a new AppDomainManager object for each application domain that you create. This removes the duality of the pre 4.0 AppDomainManager as the manager that you specify when you start the CLR can now deal solely with the requirements of the default application domain and it can set a new AppDomainManager object for each application domain that it creates.

CLR Hosting - A flexible, managed plugin system, part 1

I'm working on some prototype code right now to improve the "deployment characteristics" of a socket server that I wrote for a client which uses CLR hosting to provide multiple managed applications within a single unmanaged host. The client wants to be able to start, stop and restart individual managed applications within the server so that during development or when a managed application is updated they don't need to restart the whole unmanaged server process to use a new version of a managed application. This is all quite easy to do using separate application domains for each managed "application" within the unmanaged host but, as always, the devil is in the detail. Their existing server is already using an application domain per application but no thought was given to being able to dynamically unload and reload applications whilst the unmanaged host was running.

I've got to the point where I have a nice new server framework example server that demonstrates all of the functionality I need and so it feels like the right time to write about how I got here.

Continue reading "CLR Hosting - A flexible, managed plugin system, part 1"

CLR Hosting - .Net 4.0, .Net 2.0, take your pick

I've recently been adjusting my CLR hosting code as a client wanted to be able to host the .Net 4.0 runtime. Previously they were hosting the 2.0 runtime and, as I mentioned a while back, the hosting API has changed with 4.0.

Switching to hosting 4.0 was easy enough but being able to fall back to hosting 2.0 on a machine where 4.0 wasn't installed is slightly more complex. It's reasonably obvious (you need to make sure that you call GetProcAddress() to bind to CLRCreateInstance() rather than linking to it at build time), but Brad Wilson has a nice list of steps to show how it's done here on his blog.

There still seems to be some complexity with regards to selecting the most appropriate version of the CLR for a given assembly as whilst ICLRMetaHost::GetVersionFromFile() will, for example, give you "v1.0.3705" for a CLR 1.0 assembly and "v1.1.4322" for a CLR 1.1 assembly which is fine but ICLRMetaHost::GetRuntime() wont apply any machine policies to give you a v2.0 or whatever CLR if you ask it for v1.0.3705 and it's not installed.

I'm probably missing something in the API somewhere and it's not really that much of a problem for me right now, I already have a version munging stage which allows things like "v2.0" to mean 'the latest greatest v2.0 you have please (purely for when the next version comes out and the build is changing with the betas...) and which can be told to default to the latest CLR available if the specified one isn't found, or to fail the request, etc.

Tool lag

One of the problems of having a collection of tools that interoperate is that there's often a lag between when a tool will interoperate with the latest version of another tool. I'm hardly a bleeding-edge tool junky, I wait until RTM before I start using the latest Visual Studio on a daily basis, and in the case of VC 6 I stuck with it (as did most of my clients) until VS2005 came out and actually improved life for unmanaged C++ development... However, it seems that some tools take a long long time to catch up with Visual Studio. Take DevPartner Studio, for example, it seems that it's always lagging by just enough that I spend more time using it in my "legacy" compiler builds than I do in my day to day builds. It's cool that it works on x64 now but it's less cool that I have to use VS2005 or VS2008 to use it...

The problem with tools that lag is that no matter how useful they are the lag and the lack of daily use affects the usefulness of the tool and that always gets evaluated when I have to pay more money for another year of 'support'...

When a tool isn't in the tool box that you use daily you tend to use it less often than you should... Yes, the reason for this rant is that I've just found a couple of leaks with DevPartner that lived a little longer than they should because I'm developing the code in VS2010 and DevPartner isn't there in the toolbar nagging at me to use it. And, yes I know that I'm the one at fault for not using the tool anyway even though it's easier to ignore right now...

This kind of thing usually isn't actually too much of a problem for me as using these kinds of tools is on my release check-list but as I get clients that only want VS2010 builds it gets more likely that the tool lag will bite.

So, in summary, a version of DevPartner Studio for Visual Studio 2010 would be nice and, even better, if you can't ship with the RTM of the compiler then a roadmap that gives a hint about when you might ship, or even a hand wavey indication of an intention to ship, would be nice...

Practical Testing: 26 - More functionality, more refactoring and a new bug

Previously on "Practical Testing"... To deal with some specific usage scenarios of a piece of general purpose code I'm in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. The timer wheel trades space for speed and so far the development has gone well as I've been able to use the tests that I had already developed for the previous implementations to guide the new development.

By the end of last time we'd got to the point where we had four functions left to implement...

Continue reading "Practical Testing: 26 - More functionality, more refactoring and a new bug"

Amusing bug in GetTempPath()

Yesterday I had a bug report from a client who had a user that was getting an exception report from their software which simply said "GetTempPath() - Failed to get temp path". Now as error messages go this isn't the best but it comes from some code that has been in my tools libraries for around 10 years and which has never failed before, it has no tests, we're probably lucky that the message didn't just read "TODO" as I'm pretty sure that it's the first time that anyone has ever seen it apart from by reading my source code or running strings on an exe that includes my source code... Anyway...

The offending code was a wrapper around the operating system function GetTempPath(). I always tend to wrap these kinds of things so that I can convert errors to exceptions (I did a good job of that here didn't I) and so that I can convert types to the kind of types that I prefer to use. In this case I also dealt with the fact that you need to call the API without a buffer to determine how big a buffer it requires and then call it again with the correct sized buffer to obtain the path.

GetTempPath() is a simple function and the documentation is pretty clear on how it should be used. My original wrapper was fairly crap.

_tstring GetTempPath()

{

   const DWORD spaceRequired = ::GetTempPath(0, 0);      // 1

  

   _tstring directory;

  

   directory.resize(spaceRequired - 1);     // 2

  

   const DWORD spaceUsed = 1 + ::GetTempPath(spaceRequired, const_cast(directory.c_str()));    // 3

  

   if (spaceRequired != spaceUsed)   // 4

   {

      throw CException(_T("GetTempPath()"), _T("Failed to get temp path"));

   }

  

   return directory;

}

First I called the function to determine the required size, note that the required "size is the size of the buffer in TCHARs" needed, which includes the terminating null character. Next I created a string that was the correct size (note that I'm removing 1 because the required size includes space for the null and I'm resizing the data area of the string and the null is added on for me. Thirdly I call the API again with my buffer. I then check that the second call uses all of the space that I allocated in the first call and fail if it doesn't.

There's quite a lot wrong with this code but mostly it will work... The check at stage 4 is overly restrictive but shouldn't pose a problem, except that it does... If your temp path contains an unnecessary double backslash; C:\\temp for example rather that C:\temp then the first call will return the length including the double backslash and the second call will return the path with the unnecessary backslash removed. This means that for the example path shown above, the second call returns spaceRequired - 2 rather than the expected spaceRequired - 1 and the code fails with the rather useless error message.

The fixed version of the code doesn't suffer from this problem.

_tstring GetTempPath()

{

   const DWORD spaceRequired = ::GetTempPath(0, 0);

  

   if (spaceRequired == 0)

   {

      const DWORD lastError = GetLastError();

  

      throw CWin32Exception(_T("GetTempPath()"), lastError);

   }

  

   _tstring directory;

  

   directory.resize(spaceRequired - 1);

  

   const DWORD spaceUsed = ::GetTempPath(spaceRequired, const_cast(directory.c_str()));

  

   if (spaceUsed == 0)

   {

      const DWORD lastError = GetLastError();

  

      throw CWin32Exception(_T("GetTempPath()"), lastError);

   }

  

   if (spaceUsed >= spaceRequired)

   {

      throw CException(

         _T("GetTempPath()"), 

         _T("Failed to get temp path, second call needed more space ") + ToString(spaceUsed + 1) + 

         _T(" than first call allocated ") + ToString(spaceRequired));

   }

  

   directory.resize(spaceUsed);

  

   return directory;

}

Whilst there are still, no doubt, style issues, this version checks for errors in a better way (I'd first thought that the failure was some kind of permissions thing) and reports these errors in a clearer manner. Hopefully it will be another 10 years at least before I get another error report for this piece of code.

This fix will be included in version 6.3 of the server framework code which currently has no scheduled release date.

Practical Testing: 25 - Nothing is free

I'm in the process of implementing a timer wheel that matches the interface to the timer queue that I previously developed in this series of articles. The idea being that for certain specific usage scenarios the timer wheel will perform better than the timer queues. Last time I refactored the tests that I was using for the timer queues to remove duplication and I now have a set of failing tests for the new timer wheel.

As soon as I started to look at making some of the failing tests pass I realised that having a heap of failing tests wasn't such a good idea, at least with my home brew test framework. I had stubbed out the timer wheel's interface and had decided to throw exceptions from the functions that weren't implemented yet. Those exceptions caused the tests that used that functionality to fail, so far so good. Unfortunately there was no differentiation between tests that I knew would fail and tests that just happened to be failing; I discovered this when I realised that some of the failing tests were for the timer queues and not the wheel... Switching the exception thrown to one of my testing exceptions, a "test skipped" exception means that I now have a load of timer wheel tests that are skipped due to lack of implementation code and the test failures are clearly failures. Once the real failures were fixed I could move on with the new code.

Continue reading "Practical Testing: 25 - Nothing is free"