WOW64 Win32 DebugAPI and managed code

It seems that I’ve located the “issues” in my Debug Tools library. This library is used in my TickShifter (time control) tool and my native Win32 Deadlock Detection tool. Due to how I wanted to control the debugged processes start up and how I needed to halt the debugged process at a particular time there were some hoops that I found I had to jump through. Those hoops have changed shape; either because of differences between Vista and XP or due to x64 and the WOW64 layer, I’m not sure which yet.

Anyway, when starting a native process under the Debug API it was always easy enough to set a break point in the process’ entry point and halt the debugger at that point so that I could inject code and fiddle around. On XP I found that managed processes never hit the native entry point that’s written into the PE file and so I had to spot that the process was managed and shove a break point into _CorExeMain in mscoree.dll as that was the nearest alternative. Under x64 using .Net 2.0 I still have to do this (and what’s more the entry point in the PE isn’t in writable memory, so perhaps it’s an offset to a managed entry point…). However, when running with .Net 1.0 or .Net 1.1 on x64 I end up with the WOW64 layer coming to life (which complicates matters a little) and the native entry point IS hit as you’d expect… An added complication is that I can’t just use the code that assumes that the native entry point wont be hit and ignore it if it is because under WOW64 I start getting notifications from other threads before my managed entry point is hit and I need to gain control before other threads are running…

Fun, fun, fun… The good news is that I know what’s going on now and I expect I can build in some checks that will work out how to take control of the processes that are being started up. I don’t really want to look at the managed debug api just yet as that’s not really what I need…

Now I just need to work out how to programatically disable the “program has stopped working, dialog” which pops up when my application under test exits due to an exception as my debugger is in control and I don’t want windows trying to do ‘clever’ things for me… And, of course, it’s all different under a Win32 build on x64…