This entry's really just so that I don't forget where I found the link...

Oh, and this link is really useful to, though I've already been bitten by and fixed most of the issues that are mentioned.

There's a lock taken out in CSocketServer::Socket::IsValid() that isn't really required and which can cause a deadlock if you have your own lock in your derived class which you lock in OnConnectionReset() or other server callbacks and which is also locked when you call into the framework via Write() or other calls. The deadlock is due to the lock sequencing; when calling into the framework your lock is locked and then the socket's lock is locked when IsValid() is called, when the framework's calling in to you it does so with the socket's lock locked and so your lock is acquired after the socket's lock.

The fix is to remove the lock acquisition in IsValid() so that the socket's lock is not locked when you call into the framework to Write() or to issue a Read().

Since the CLR can't be stopped by a host without terminating the host process (and that's by design...) you need to be aware that any of the code that you have plugged into the CLR by way of the hosting interfaces can be called into during process shutdown. Since most of the code that you plug in will be in the form of COM objects you can rely on the COM reference counting to deal with cleaning up the objects (if the CLR could be relied on to release the references at any point...) but since you're now writing code that out-lives main() you have to be intimately aware of how this code uses other code that might rely on module level static variables, and other nastiness. Sure you wouldn't want to be relying on module level statics and the order of creation and destruction of those statics (which, as we know, is undefined across multiple modules) but if you're using the version of STL that ships with VS2005 in any of your hosting COM objects then you are...

My implementation if IHostTaskManager uses a std::map to keep track of its tasks. During process shutdown the CLR calls into my task manager to put a task to sleep, to do this I need to locate the task in my map and doing that relies on iterating the STL map and that, if _HAS_ITERATOR_DEBUGGING is enabled, seems to manipulate a global lock... Unfortunately the global lock has been destroyed by the time the CLR decides to shutdown ... Fortunately the global lock is kept alive by a reference counting class called _Init_locks which, although apparently undocumented, can be used to keep the locks that you need to use alive until the object that needs them is destroyed... So, by sticking an instance of _Init_locks into my task manager class I can make sure that it can iterate its collection after main() has returned and the process is shutting down...

Anyway, this week I needed to run up the tool to track down this newly reported problem and I found that I didn't have a build that compiled...

At that time I had another problem that looked suited to a custom debugger with API hooking and I wrote a new tool to allow me to control the way an application viewed time. Given the way I structure my code, most of it ends up being pushed into libraries for reuse, it was easy for me to build this and other tools from the same codebase. This new time shifting tool was considerably simpler and I slapped a simple GUI onto it. The GUI components were coming on but I still didn't have everything that I needed to put the GUI on the deadlock tool.

I found myself bogged down in the GUI and noticed that I was making less and less progress. I decided to put a halt to the GUI for a while and, instead, work on some of the meatier items from my list of things to do. Top of the list was making the debug engine work with .Net (CLR) processes. At this time my debug engine was simply using CreateProcess() to create a suspended process and then injecting a DLL and hooking the APIs that it wanted to hook and then resuming the process. This worked fine for "normal" exes and failed dismally with .Net exes. For some strange reason CreateProcess() appears to fail to suspend the correct threads in a .Net process and a suspended process simply runs....

The solution was to switch to using the Win32 Debugging API which gave me enough control over the process to be able to work out how to correctly start and safely suspend a .Net process. Once I was using the Win32 Debugging API I decided that I may as well cross another item off of my list by supporting multiple process debugging; my debug engine could now debug a process that started a child process and also debug the child process at the same time.

Then Christmas arrived and I was caught up finishing some work for a client, I integrated the new debug engine into the time shifter tool and updated the GUI, then we skied some and then my world ended.

So there I was, not really in the mood to code and with a sudden need for a working version of the deadlock tool. Due to some sloppiness on my part, I didn't have one (I'd not bothered to tag the code of the last working build) and to get one either meant restoring a backup or knocking another item off of my list by updating the deadlock detection tool to work with the new debugger engine. I decided that I needed to get back into coding so opted for the later and spent two days getting my head back into the code and updating the deadlock detection tool to work in the same way as the time shifting tool, they both now use the new multi-process, CLR friendly, debug engine.

The tool found the problem by mistake. The actual bug that the client was reporting wasn't actually a deadlock but my thrash testing of their server under the deadlock detection tool managed to reproduce the problem and the tool captured some useful information that helped me to track it down.

The problem turned out to be a subtle race condition. The server is a gateway, it accepts incoming connections and routes them to an outbound connection. The race condition was that the "connector" object could be deleted due to a connection closure whilst it was still being used due to data flow. The solution was to reference count the connector so that although the connection closure would release a reference the data flow would still be holding a reference and so destruction would be delayed until the connector was no longer in use.

Locating the problem identified an issue with the deadlock detection tool. It uses masses of memory when running for a long period on processes that perform lots of manipulations on lots of locks. I'm collecting all of the data (in all of the formats) as the manipulation happens. This means storing some of the information multiple times - I track details about which threads do what to each lock, I track the order in which each thread acquires lock sequences, I track everything a particular thread does with any lock, I track all of the operations that occur on any thread in the order that they happen, etc. All of this is useful information but much of it could be worked out after the event. At present we could derive much of this information at the point that we want to create the report, all from a single list of operations... When displaying information on a GUI some of the stuff we currently store needs to be derived for display, but not all of it, and not all of the time, and even if it does then the data structures used for holding the data for display will probably duplicate much of what we're already collecting.

There was no real plan to how we'd collect the data. It just grew as I added more interesting reporting options. I'd decided not to summarise as I went along as this restricted the ways that I could slice and dice the data but now I was doing the opposite and collecting all manner of pre sliced data... Now that I know I'm collecting too much I need to go back and revisit the data collection and work out what is the minimum that I require...

1) Why project delivery bonuses suck - from Kevin Barnes over at "Code Craft". Kevin writes a lot of really good stuff and this one really hits the nail on the head. I've worked in a lot of investment banks over the years and they use these kinds of bonuses a lot; I've never seen a good outcome from their use, the project always gets twisted to achieve the date and the future usability and maintainability is always the thing that gets sacrificed. In the end they're giving bonuses to write shoddier software that will cost more to fix and extend in the future...

2) Three reasons to reinvent the wheel - another excellent piece from Kevin Barnes. Sometimes there are good reasons to build it yourself from scratch.

3) In Defense of Verbosity - Jeff Atwood at "Coding Horror" on why saving keystokes whilst writing code is a fool's economy. As always Jeff's spot on with his analysis and backs up everything he says with references to Code Complete. Personally, I'm quite happy with the conciseness of } given my style of coding (I don't nest too deeply and I use lots of really small functions for stuff so it's always easy to know what the current } applies to; but his point applies to all aspects of coding. Write for the reader, not the compiler!

4) Manage Dependencies not Aesthetics - David Chelimsky over at "But Uncle Bob .com". A nice piece on design and the Single Responsibility Principle.

5) Classic Technical Lead Blunder - Jeremy D. Miller over at "CodeBetter.com" (note that you can subscribe to Jeremy's feed directly rather than subscribing to the whole crowd at CodeBetter.com, a much better option IMHO). A good piece for all who are leading other coders; I've probably made all of these mistakes a couple of times. Some are easy to stop yourself from doing, others less so; the "Detailed work assignments" mistake is spot on and an easy mistake to make.

6) What's so great about Inversion of Control? - Jeremy D. Miller again. Excellent piece on designing for testing which could be used to counter the next posting...

7) Why unit tests are disappearing - Cedric over at "Otaku, Cedric's Weblog". Cedric disagrees with Michael Feather's "rules" on what constitutes a unit test. I disagree with Cedric ;) and feel that he's missing the key point of Michael's rules; "Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes." If you try and stick to the rules then you'll end up breaking the file access, or database access, etc, out into quite a small and simple class that does just that. The rest of the system can deal with these classes via interfaces and can therefore be tested without using the database or file system but instead from a mock object. If you disregard the rules you're less likely to do this breaking apart, your classes violate the SRP and you end up mixing business object to file format type code in with file access code which makes it harder for you to switch to alternative persistence methods in the future. So, stick to Michael's rules and read 6) for a much better explanation of why.

8) Ferrari versus Land Rover - Mark Pearce (who lives near me, apparently) over on "Fighting the Plumbing since 1979" (great name for a blog and not just because I spent the first 10 years of my working life doing just that... ;) ) - Although this piece is a response to my Assert is Evil posting from last week, and, whilst I've responded to that aspect of it there's more here that I feel I need to respond to. The whole Ferrari verses Land Rover idea and the suggestion that most C++ behaves like a Ferrari and most VB like a Land Rover... I think, it depends. Some kinds of program can safely act like a Land Rover and some must act like a Ferrari. Often the programs that can be like a Land Rover can be written in VB...

9) Multi-platform Images - Mark Russinovich over at "SysInternals". Mark shows us how he packages the SysInternals tools as a single executable file that can run natively on multiple platforms (32bit & 64bit Windows) by embedding the 64bit executable inside the 32bit exe as a resource. I've embedded dll's in exes in the past, we used it for the simple downloadable driver stuff for my WebRemote Lego Mindstorms over Netmeeting code (3 dlls embedded in a single exe which you downloaded and ran to unpack and register the dlls). The embedded driver technique that Mark mentions is mentioned in the RootKits book that I'm currently reading. Oh, and don't miss the link to James Finnegan's article; well worth reading if you're starting out with driver development...

10) There's no feeling like releasing software... - Robert over at "Software is too expensive to build cheaply..." A nice piece on releases and agility; I have to agree that you can tell how likely a project is to succeed by the "pulse" of its production releases. I've worked on systems where the release schedule is so slow that the team almost forgets that's what they're working towards and on projects where we released twice a week every week. I know which one was better and I know which one had the most polished release procedures and dealt best with release problems, rollbacks and hotfixes. The less often you release the harder a release is because you're less used to doing them and you are less likely to have automated all the things you need to have automated. When you're releasing twice a week to a trading desk you make damn sure that everything that can be automated is automated and that you have lots of regression tests in place.