Rambling Comments

WebSockets - I miss the TCP half close...

2011-07-08T15:25:59Z

As I mentioned here, the WebSockets protocol is, at this point, a bit of a mess due to the evolution of the protocol and the fact that it's being pulled in various directions by various interested parties. I'm just ranting about some of the things that I find annoying...

The WebSockets protocol includes a way for the endpoints to shut down the connection.


   If an endpoint receives a Close frame and that endpoint did not
   previously send a Close frame, the endpoint MUST send a Close frame
   in response.  It SHOULD do so as soon as is practical.  An endpoint
   MAY delay sending a close frame until its current message is sent
   (for instance, if the majority of a fragmented message is already
   sent, an endpoint MAY send the remaining fragments before sending a
   Close frame).  However, there is no guarantee that the endpoint which
   has already sent a Close frame will continue to process data.

   After both sending and receiving a close message, an endpoint
   considers the WebSocket connection closed, and MUST close the
   underlying TCP connection.  The server MUST close the underlying TCP
   connection immediately; the client SHOULD wait for the server to
   close the connection but MAY close the connection at any time after
   sending and receiving a close message, e.g. if it has not received a
   TCP close from the server in a reasonable time period.

Unfortunately it's a full close of the connection in both directions rather than an indication that the endpoint sending it has nothing more to send. In effect it's a close() rather than a shutdown(). This means that if you ever want to indicate that you're done sending but that you'd like the other end to finish up and then close you have to implement it yourself...]]> TCP allows you to close each direction of the connection independently. The first peer to shut down their sending side is deemed to have issued the active close. Whoever issues the active close ends up in TIME_WAIT state (see here for more details on that). The WebSockets protocol provides a protocol level close message which is intended to do the following:


   The closing handshake is intended to complement the TCP closing
   handshake (FIN/ACK), on the basis that the TCP closing handshake is
   not always reliable end-to-end, especially in the presence of man-in-
   the-middle proxies and other intermediaries.

   By sending a close frame and waiting for a close frame in response,
   certain cases are avoided where data may be unnecessarily lost.  For
   instance, on some platforms, if a socket is closed with data in the
   receive queue, a RST packet is sent, which will then cause recv() to
   fail for the party that received the RST, even if there was data
   waiting to be read.

Which is all well and good but the ability to perform a half-close would be very useful and its unfortunate omission will no doubt lead to applications inventing their own...

WebSockets - The deflate-stream extension is broken and badly designed

2011-07-08T15:00:00Z

The WebSockets protocol is designed to be extended, which is all well and good. Extensions can, at present, be formally specified by RFCs or be "private use" extensions with names that are prefixed with an "x-". So far the only "official" extension is the deflate-stream extension that's detailed in the WebSockets protocol RFC itself.

Unfortunately, the deflate-stream extension isn't really an ideal example of how future extensions should work. A far better example would be the alternative deflate-application-data extension that's detailed here.

So, what's wrong with deflate-stream?]]> The WebSockets protocol is very open to extension, possibly too open.


4.8.  Extensibility

   The protocol is designed to allow for extensions, which will add
   capabilities to the base protocols.  The endpoints of a connection
   MUST negotiate the use of any extensions during the opening
   handshake.  This specification provides opcodes 0x3 through 0x7 and
   0xB through 0xF, the extension data field, and the frame-rsv1, frame-
   rsv2, and frame-rsv3 bits of the frame header for use by extensions.

   The negotiation of extensions is discussed in further detail in
   Section 9.1.  Below are some anticipated uses of extensions.  This
   list is neither complete nor proscriptive.

   o  Extension data may be placed in the payload data before the
      application data.

   o  Reserved bits can be allocated for per-frame needs.

   o  Reserved opcode values can be defined.

   o  Reserved bits can be allocated to the opcode field if more opcode
      values are needed.

   o  A reserved bit or an "extension" opcode can be defined which
      allocates additional bits out of the payload data to define larger
      opcodes or more per-frame bits.

But even with all of this available the deflate-stream extension goes further. The deflate-stream extension completely replaces the wire format of the WebSockets protocol with a stream of compressed data. That is, all framing and headers are included in the compression and you can't decompress a portion of the stream without decompressing all of it. It means that any form of proxy that wants to inspect the contents of WebSocket traffic has to decompress the whole stream to look at the individual frames; it can't even simply use the header information to skip the data portions of the frames as the headers are also compressed. The WebSocket protocol doesn't really even hint at this being the intended use of the extension mechanism, in fact it seems to suggest that whilst extensions can fiddle with the header contents and the frame contents they can't simply replace the entire wire format. There are those who think the inclusion of "connection-level" extensions is a bad idea and I agree with them. After all, by following the example of the deflate-stream extension surely the obvious solution to the rather broken message based aspect of the protocol is simply to write an extension that replaces the entire WebSocket wire format with one that pleases you better; x-raw-tcp-stream anyone?

If the fact that the deflate-stream extension isn't exactly an ideal example of how extensions should be built isn't enough of a reason to remove it from the RFC then surely the fact that it doesn't actually work as expected might be? It doesn't seem likely. The problem is that now that WebSockets requires client to server data frames to be masked the data being sent from client to server is sufficiently random that compression can't be applied. So, at best, deflate-stream, simply burns server-side CPU when dealing with inbound WebSocket data and at worst it makes the data stream from client to server bigger than it need be. Ideally you'd simply not mask if you were going to use deflate-stream but that doesn't seem to be an option; there appear to be concerns that the extremely unlikely situation that client to server masking is supposed to protect against could occur using unmasked compressed data. Of course the attacker would have to know exactly how the data was going to be deflated and could therefore somehow come up with some payload data that results in a useful stream of deflated data; but really...

Ripping deflate-stream out of the RFC entirely and/or replacing it with deflate-application-data would seem to be the obvious course of action, but there seems to be resistance to this. Making it possible to enable deflate-stream in one direction only, i.e. from server to client, would also seem to be a potential win as you'd get your deflated stream in the direction that it can work and not waste time on data that can't be meaningfully compressed.

WebSockets - Why do we need to mask data from client to server?

2011-07-06T14:52:04Z

   The client MUST mask all frames sent to the server.  A server MUST
   close the connection upon receiving a frame with the MASK bit set to
   0.  In this case, a server MAY send a close frame with a status code
   of 1002 (protocol error) as defined in Section 7.4.1.

Fair enough, but why?

]]> The masking in itself is relatively painless to achieve but it interacts poorly with the albeit badly designed deflate-stream extension and forcing the client to mask zero length frames seems unnecessary - though I agree that special cases don't really seem warranted.

The RFC doesn't explain why masking from client to server is considered essential but a search through the discussion list brings up plenty of hits. The best descriptions of why can be found here and here.

Masking of WebSocket traffic from client to server is required because of the unlikely chance that malicious code could cause some broken proxies to do the wrong thing and use this as an attack of some kind. Nobody has proved that this could actually happen, but since the fact that it could happen was reason enough for browser vendors to get twitchy, masking was added to remove the possibility of it being used as an attack.

The idea being that since the API level code generating the WebSocket frame gets to select a masking key and mask the data supplied by the application code the application code cannot in any meaningful way dictate the data that ends up passing through the potentially broken intermediaries and therefore can't cause trouble. Since the masking key is in the frame intermediaries can be written to understand and unmask the data to perform some form of clever inspection if they want to.

Now, would it be so hard to add something that explains this to the RFC?

WebSockets - Why differentiate between text and binary frames?

2011-07-06T13:14:49Z

Back when binary frames were mentioned in the WebSocket protocol specification as a slightly hand wavy "something for the future" and only text frames were actually possible to send and receive using the clients at the time then there MAY (in the strictest RFC meaning of the word) have been a need to differentiate between text and binary frames. Especially since text frames were 0xFF terminated rather than length prefixed. Now, however, it seems pointless to have a protocol level differentiation between text and binary frame types.

This would be, perhaps, less pointless if a general purpose protocol handler could reliably deliver complete messages to an application, but as we've seen before, that's not possible. Although backtracking to the last complete character in the UTF-8 stream is relatively straight forward (see here) there seems to be little value in converting a partial message to a wide string form only to have the have the application layer have to store and concatenate to obtain a complete message...

Robert Gezelter probably says it far better than I can here on his blog; the differentiation between binary and text should occur at the application level.

WebSockets is a stream, not a message based protocol...

2011-07-06T12:34:17Z

The first thing to realise about the WebSockets protocol is that it isn't really message based at all despite what the RFC claims.

   Clients and servers, after a successful handshake, transfer data back
   and forth in conceptual units referred to in this specification as
   "messages".  A message is a complete unit of data at an application
   level, with the expectation that many or most applications
   implementing this protocol (such as web user agents) provide APIs in
   terms of sending and receiving messages.  The WebSocket message does
   not necessarily correspond to a particular network layer framing, as
   a fragmented message may be coalesced, or vice versa, e.g. by an
   intermediary.

and...

   The WebSocket protocol uses this framing so that specifications that
   use the WebSocket protocol can expose such connections using an
   event-based mechanism instead of requiring users of those
   specifications to implement buffering and piecing together of
   messages manually.

Suggest that a message based, event driven design which presents complete messages to the application layer would be a sensible design. Unfortunately once you realise exactly how a message is made up it becomes impossible to provide an interface where you ONLY deliver messages as complete units to the application layer.

]]> WebSocket messages consist of one or more frames. A frame can be either a complete frame or a fragmented frame. Messages themselves do not have any length indication built into the protocol, only frames do. Frames can have a payload length of up to 9,223,372,036,854,775,807 bytes (due to the fact that the protocol allows for a 63bit length indicator) and finally...

   The primary purpose of fragmentation is to allow sending a message
   that is of unknown size when the message is started without having to
   buffer that message.  If messages couldn't be fragmented, then an
   endpoint would have to buffer the entire message so its length could
   be counted before first byte is sent.  With fragmentation, a server
   or intermediary may choose a reasonable size buffer, and when the
   buffer is full write a fragment to the network.

So a single WebSocket "message" can consist of an unlimited number of 9,223,372,036,854,775,807 byte fragments. This makes it impossible for a general purpose Websocket protocol parser to only present complete messages to the application layer in such a way that the application doesn't need to do some form of "buffering and piecing together of messages manually". At best a general purpose parser could present Websocket data as a 'sequence of streams', given that each "message" is in fact simply a potentially infinite stream of bytes with a message terminator (the FIN bit in the frame header) at the end. It could do this by passing the application layer an interface that allowed the application to pull data from the Websocket "message" until it was complete, and that's less than ideal if you are used to working with asynchronous, push, APIs, or trying to avoid unnecessary memory copies...

Even if the maximum frame size was reduced, as some propose, the problem would still be present due to the fact that a single message can consist of an infinite number of fragments. Likewise a protocol parser can not take the easy route and simply disallow fragmented frames since the RFC states that...

   o  Clients and servers MUST support receiving both fragmented and
      unfragmented messages.

   o  An intermediary MUST NOT change the fragmentation of a message if
      any reserved bit values are used and the meaning of these values
      is not known to the intermediary.

   o  An intermediary MUST NOT change the fragmentation of any message
      in the context of a connection where extensions have been
      negotiated and the intermediary is not aware of the semantics of
      the negotiated extensions.

Which means that although the application that you've written may send and receive WebSocket messages of an application restricted maximum size you may still find that you receive fragments because an intermediary has decided to fragment your frames. Unless, of course, you subvert the protocol extensions functionality by negotiating "x-{My own private GUID}" extension between your client and server which would neatly prevent any intermediaries (except ones that you'd written yourself) from changing the fragmentation of your frames... Then, of course, the intermediaries may simply decide to remove the client's request for your unknown extension from its initial handshake request to prevent it being negotiated. Or, perhaps more likely, close your connection with a 1004 close code...

There's resistance to proposals to allow the maximum frame size to be negotiated during the handshake phase but there's a standard close code for "frame too big"... Should an application just guess how large it's allowed to go?

The view of some on the discussion list seems to be that "A server (or client) which exposes the frames as its primary API is doing it wrong." but it seems to me that to write a flexible and general purpose protocol parser which can be used by both push and pull APIs you have no option but to expose details of the message framing. The reason for this is that a general purpose parser cannot buffer complete messages, or even complete frames and so must deliver the data either as a stream of bytes at an application level or as a sequence of partial frames and allow the application to decide how to accumulate the frames into messages. By hiding all of the framing from the application the application developer cannot take advantage of knowledge that he has of the message structure of his particular messages. This is especially useful with asynchronous APIs where the application might be pushed buffers with data in them as the data arrives - which is how most of The Server Framework happens to operate and how I/O Completion Port centric designs would tend to work. If an application works in terms of, say, messages that could be at most 4096 bytes and is dealing with buffers that can contain complete messages then it could use the details of the data framing to allow it to efficiently accumulate the data into a single buffer and then dispatch the complete message for processing when it receives the final frame. The alternative is to add complexity to the protocol parser by allowing it to accumulate 'messages' up to a configurable size and present complete messages via one callback and incomplete messages via another, or to provide only a stream based pull API which requires the application to needlessly copy data from the protocol parser's buffers into its own.

The 63bit fragment size and fragmentation in general appear to come from a requirement for streaming data from one end of the connection to the other, see here where a Unixy design idea of simply telling the application to "read x amount of data from this file handle" seems perfectly sensible... Of course this design will fail as soon as you need to send a stream that's bigger than the 63bit frame size will allow and it'll also fail if the frame is fragmented as then the API becomes "read x amount of data from this file handle, but you're not done yet, wait and I'll call you again with more"... At which point and given the possibility of intermediaries that fragment your large fragment down to, lets be generous, 1024 byte fragments anyway, you may as well simply limit the maximum size of frames to something more manageable... But I suppose "nobody will ever need more than 63bits of data length"...

Unfortunately, large frame sizes also open the protocol up to lazy application design. Lets say we're sending a file, we open the file and read some, send a fragmented frame header for the total file length and then simply start sending data. Cool, we don't need to worry about the protocol any more, no need to build messages, just pull the data from disk and send it to the other side. This works fine until you have a read failure, or any other reason to terminate the connection. Since you're in the middle of sending a single huge frame you can't send an application level frame that informs the other side of the problem. You can't even send a WebSocket close frame to shut down the connection cleanly, all you can do is abort the connection...

So, WebSockets presents a sequence of infinitely long byte streams with a termination indicator (the FIN bit in the frame header) and not a message based interface as you might initially believe. Given that a general purpose protocol handler can only work in terms of partial frames, we effectively have a stream based protocol with lots of added complexity to provide the illusion of a message based protocol that can actually only ever be dealt with as a stream of bytes.

Movable Type upgrade annoyance

2011-06-15T07:12:37Z

I've just upgraded Movable Type from 5.03 to 5.11. The upgrade itself went smoothly except for one thing. A recent fix to MT to remove an obscure HTML standard violation that Firefox was causing problems with means that permalinks with runs of dashes in them have been changed. You can see the full details here on the MT forums where I posted the bug report.

This is more of an issue for me as I did the upgrade and then launched www.lockexplorer.com which uses the same MT installation as www.serverframework.com, www.jetbyte.com and www.lenholgate.com. The new site now has the new format urls which would be broken if the guys at MT revert back to the old format and the older sites have the old format urls which are broken (if I republish) if they don't revert.

I'm glad I'm up to speed on .htaccess url rewrites as I can't see that I can get around needing to work some magic no matter what the guys at MT do to fix the problem.

Permalink. There's a hint in the word...

Things I hate about Visual Studio 2010 - part "today"

2010-09-24T20:51:12Z

I tend to work with lots of solutions at once. I'm often building code for clients, building and testing new example servers for The Server Framework and running lots of copies of various versions of Visual Studio at once.

Now, if I happen to try and open two VS2010 solutions at around the same time (and given the time it takes for VS2010 to start up that's not that hard to do), then one of them wins and gets my normal settings and one (probably the one that gets a sharing violation whilst trying to read my normal settings) ends up with some default madness setup that isn't what I want at all.

The great thing is, hmm, irony, that the one with the madness thinks it should update the change it has made and so from then on every time I launch VS2010 I get the 'new' madness settings. Great. Not.

It's things like this that stop me customising things... I run with a pretty vanilla OS install just because nobody else has my custom setup, so it's easier for me to be productive if I don't do custom setups; learn to be productive in the default environment... But I like my solution explorer on the left, and I like not having those useless tabs on the right... Grrrrr.....

DevPartner Studio 10 - unmanaged disappointments

2010-08-15T10:06:21Z

The story so far, I've owned DevPartner Studio for several years and been on a support contract the whole time. The support situation went a bit flakey when Compuware sold the product to Micro Focus, I expect that either I'm simply not a big enough client for them to care or their post sales and support is just no where near as good as the Compuware offering. With Compuware I'd be contacted when new versions came out and I had a named sales contact that I could also use for questions, etc. What's more the sales guy was good, he knew I wouldn't be buying more licenses, he remembered the issues I'd raised and he helped chase up responses etc.

Anyway, I whinged about how DevPartner lagged behind Visual Studio releases and was told, via a comment from someone at Micro Focus, that DevPartner 10 shipped at the same time as Visual Studio 2010; it would have been nice if my support contract meant that they'd bother to tell me about this since I was entitled to the latest version; heh, here's an idea, the software itself could work it out for me somehow (nah, nobody would do that)... Anyway, it seems that the way I'm supposed to work out that a new version that I'm entitled to under my support contract is available is to log on to the Micro Focus support site and 'request a new version'. Presumably I either poll this site every couple of months and request current version + 1 or I'm supposed to track when they release new versions... Interestingly I get a list of patches for 9.1 displayed but no indication that a 10 is available...

So, I fill in the form and wait a few days, no response, I chase them and eventually get an email with some download links. I download and install DevPartner 10 and run it up in "Error detection" mode on an unmanaged VS2010 solution which contains a relatively simple multi-threaded project...

During the build I get several compiler crashes and whilst it IS the Microsoft C++ compiler that is crashing here it has never crashed with this code if DevPartner instrumentation is NOT enabled... The errors are all similar:

1>e:\source\practicaltesting\jetbytetools\testtools\testmonitor.cpp(748): fatal error C1001: An internal error has occurred in the compiler.
1>  (compiler file 'f:\dd\vctools\compiler\utc\src\p2\main.c[0x524EDFBA:0x00000004]', line 183)
1>   To work around this problem, try simplifying or changing the program near the locations listed above.
1>  Please choose the Technical Support command on the Visual C++
1>   Help menu, or open the Technical Support help file for more information

And they all occur in these kinds of situations...

   }
   catch(const CSEHException &e)
   {
      cout << "Exception during test timeout processing: " << CStringConverter::TtoA(e.GetWhere() + _T(" - ") + e.GetMessage()) << endl;
   }
   JETBYTE_TESTS_CATCH_ALL_AT_THREAD_BOUNDARY_IF_ENABLED
   {
      cout << "Unecpedted exception during test timeout processing" << endl;
   }

Where the macro evaluates to catch(...) in this particular build.

So, my first impression of the new version is that it's just about as good as the first impression I had of 9.0, i.e. it simply doesn't work on the latest compiler with simple code. I don't know if I'm alone in thinking that when software that you're supposed to rely on for software quality assurance work gives this kind of first impression you tend to worry if it actually does what it needs to at all, rather than simply being a rushed to market mess which underwent too little testing before release so that it could be shipped at the same time as the new compiler without actually supporting it...

Anyway, I thought, perhaps, that by default there were too many error detection options turned on (I tend to use a small subset of the functionality as I've found that the more interesting stuff is prone to not work (API parameter checking is great until you realise that some of the (admittedly many) APIs that it covers are configured incorrectly and report problems that don't exist and you can't change them and have to wait for a service release...). Anyway, off I go to the options screen and find that I can't change the error detection options as "Error Detection settings are unavailable until you select a valid start up project"...

At this point I switched to VS2008 to see how things were back in the legacy compiler environment. This time everything builds OK but when I go to run the program under BoundsChecker I get a message telling me that DevPartner is unable to test 64bit projects. Fair enough; we're still in 'tool lag' country, DevPartner 9.1 could also only 'run on' x64 and not instrument x64 code; then the tool would also try and instrument the x64 builds but the link failure messages were fairly clear in what was wrong. It's a pity that the DevPartner 10 user interface isn't consistent on this, the VS2010 options screen obviously knows it's not supposed to let you work with x64 projects (though could be more helpful) yet the tool bars and build process let you try to build x64 projects rather than, more helpfully, preventing you...

Rebuilding the 32bit version of the code in VS2010 worked OK, but what a wonderful first impression...

Now there seem to be some new weird problem reports inside of the VS2010 STL code which I need to dig my way through to determine why they're OK to ignore...

And then I suppose I need to go and dig up all of the same information that they already have about my development machine so that I can ask them to send me a new license file...

Tool lag

2010-07-27T10:02:06Z

One of the problems of having a collection of tools that interoperate is that there's often a lag between when a tool will interoperate with the latest version of another tool. I'm hardly a bleeding-edge tool junky, I wait until RTM before I start using the latest Visual Studio on a daily basis, and in the case of VC 6 I stuck with it (as did most of my clients) until VS2005 came out and actually improved life for unmanaged C++ development... However, it seems that some tools take a long long time to catch up with Visual Studio. Take DevPartner Studio, for example, it seems that it's always lagging by just enough that I spend more time using it in my "legacy" compiler builds than I do in my day to day builds. It's cool that it works on x64 now but it's less cool that I have to use VS2005 or VS2008 to use it...

The problem with tools that lag is that no matter how useful they are the lag and the lack of daily use affects the usefulness of the tool and that always gets evaluated when I have to pay more money for another year of 'support'...

When a tool isn't in the tool box that you use daily you tend to use it less often than you should... Yes, the reason for this rant is that I've just found a couple of leaks with DevPartner that lived a little longer than they should because I'm developing the code in VS2010 and DevPartner isn't there in the toolbar nagging at me to use it. And, yes I know that I'm the one at fault for not using the tool anyway even though it's easier to ignore right now...

This kind of thing usually isn't actually too much of a problem for me as using these kinds of tools is on my release check-list but as I get clients that only want VS2010 builds it gets more likely that the tool lag will bite.

So, in summary, a version of DevPartner Studio for Visual Studio 2010 would be nice and, even better, if you can't ship with the RTM of the compiler then a roadmap that gives a hint about when you might ship, or even a hand wavey indication of an intention to ship, would be nice...

Things I hate about Visual Studio 2010 - part 1

2010-04-22T16:42:38Z

So far I've found Visual Studio 2010 quite a compelling upgrade from Visual Studio 2008 for native C++ development. Intellisense is better, the build experience seems smoother and faster, editing on a decent development box seems OK, the various profiling and concurrency tools look interesting and, well, it mostly works quite nicely.

It's a pity that there are some rough edges. I've been seeing crashes when I update project and solution files whilst it has them open (a common thing to do if you're working with source control). I haven't yet been able to get a repeatable reproduction, but my gut tells me it's intellisense (isn't it ALWAYS intellisense??).

The ipch directory is a pain. Which bright spark decided to cut the "and change the property dialog so that you can specify where it lives" work item from the schedule? Just dumping a directory in the middle of my source tree is a crap thing to do! Likewise what's with the mess of tlog files? What intern designed that solution? Surely a single file with all this information in would be better? Surely that should also have had a "and this is where you specify where it lives and what it's called" property sheet entry?

But the worst thing so far is the "find and replace" dialog. It always has been a bit of a stupid thing, I can see why it might seem like a good idea to have it jumping around all over the place so that it doesn't obstruct the found items but, really, has anyone in Microsoft actually USED it in anger? The jumping has always been a pain, especially in scenarios where you basically want to click "replace all" but instead click "replace" so that you can eyeball each of the replacements for the one that shouldn't be done... Dragging it off to one side used to be fine but now it rarely seems to stay where you put it and sometimes ends up off the screen (!). You'd think that the fact that you could dock it would help, but no, everywhere I've tried to dock it I end up with it failing to scale itself to the space that's given to it; so some buttons aren't available...

I know it's a huge product and I know I'm nit picking, but, really, these things are important, if only because they give people like me a reason to bitch about what otherwise is a cracking development environment...

Edit: And then, of course, it stops jumping around for some unknown reason and acts like a sane user interface control... I don't care why it's now behaving itself but I hope that all my other gripes can be fixed by simply ranting about them!

Visual Studio 2010 Beta 2 installation failure

2009-10-22T09:45:01Z

I've been having problems installing VS2010 Beta 2 onto a selection of x64 VMWare machines (fully updated and service packed XP and Windows 7). In fact I was having problems installing it onto any of them. The error that I was getting was this:

[10/22/09,07:28:42] Microsoft Application Error Reporting: [2] CMsiComponent::Install() expects the setup file for Microsoft Application Error Reporting, but the file failed verification.

And whilst a google showed that others were also getting the same error it was few and far between and nobody had been given any useful advice.

I'd tried with both the premium and the ultimate iso and from both mounted iso files and physical discs burnt and verified from the iso (so I was pretty sure that it wasn't a corrupted download or some issue with VMWare iso mounting). I then tried copying all of the dvd to a folder on a hard disk and at that point I got lots of 'failed to copy, can't read file' errors.

I decided to download the iso images again and this time, I used Internet Explorer rather than Chrome, my default browser. IE popped up the usual Microsoft File Transfer Manager app and began the download at a reasonable speed. When I had downloaded with Chrome I'd got a "normal" http download and I had assumed that MSDN were hosting the files differently as they were expecting the rush that they got with Windows 7, etc. Anyway... It seems that the Chrome downloads (or, presumably, any downloads that don't use the MS FTM) are broken.

The iso that I downloaded using the file transfer managed is installing just fine...

Grrr!

uuid.lib(unknwn_i.obj) : fatal error LNK1103: debugging information corrupt; recompile module

2009-08-11T07:49:55Z

Back in October 2005 some of my clients started to complain that the latest version of the Platform SDK (the April 2005 version) broke their builds. The culprit was uuid.lib which had been built with debugging information that the VC6 linker didn't understand and which caused a link failure. The end result was that the last version of the Platform SDK that worked with VC6 was the February 2003 version. This all seemed quite unnecessary to me at the time; as if the latest Platform SDK was deliberately unusable for VC6 which had (in September 2005) just become unsupported.

Well, it's happening again... With version 7.0 of the Windows SDK, uuid.lib now gives a similar error when used with Visual Studio 2005.

Of course I may be doing something wrong, but...

Either the v7.0 Windows SDK was never tested with Visual Studio 2005 building applications that need to link with (at least) uuid.lib or it's considered OK for this version of the Windows SDK to explicitly not support this compiler; in fact, if this is the case then it's considered OK for this version of the Windows SDK to only support VS2008 (VS2010 isn't in production yet).

This isn't the only way that Platform/Windows SDKs have been made incompatible with versions of the compiler. VS2005 included sal.h in its compiler header directory and versions of the Platform SDK after VS2005 used sal.h but didn't include it themselves. This meant that these versions of the Platform SDK no longer supported VC6, VS.Net (2002) and VS2003. Simply copying sal.h into the old compiler's include directories fixed this... Then there was the _WSPIAPI_COUNTOF macro change. Version 6.0a or later of the Windows SDK use a new version of _WSPIAPI_COUNTOF which is a bit more clever and templatey than the old version. As such it fails on compilers before VS2005. The new version is no more functional than the old as can be seen by the fact that you can test for the compiler version and the SDK version and simply define the old version and everything works fine...

IMHO, this kind of thing doesn't encourage people to move from one version of the compiler to another. There are lots of reasons that people can't always use the latest and greatest version of the compiler. It simply means that some people wont use the latest version of the SDK and therefore, if anything, it slows the move towards using the latest features in the operating system. Of course some people, like me, are too stubborn to let this kind of rubbish put me off and we've been supporting most compilers with most SDKs well past their best before dates; we have to, our clients demand it.

So, if uuid.lib (and, of course, whatever other libs that may have been recompiled with incompatible debug information present) really needed features from the new linker and Visual Studio is supported then I guess a hotfix for the VS2005 linker will become available... Alternatively the Windows SDK could ship a new linker in its bin directory in the same way that it ships new versions of MIDL, etc. Either way there's no reason that I can think of that this problem should exist except for making it harder to build software for Windows 7 with Visual Studio 2005...

Luckily, for now, the fix is pretty easy. You can use the version of uuid.lib that comes with v6.1 of the Windows SDK for Visual 2005 builds and, if you find that you need some new GUIDs that happen to be included in the v7.0 version you can always define them yourself. It will be interesting to see if there are other libs that are incompatible.

Later... interestingly it seems that the linkers supplied with VS.Net (2002) and VS2003 work fine with the new uuid.lib file... It's only VS2005 that seems to have problems...

Later... seems like this http://support.microsoft.com/default.aspx?scid=kb;en-us;969443&sd=rss&spid=3041 hotfix may fix things...

Even later... Thanks to a comment by Soo Wei Tan, it seems that this http://support.microsoft.com/kb/949009 is the hotfix that you need....

The new Microsoft File Transfer Manager appears to suck...

2009-01-08T15:09:39Z

I'm downloading the Windows 7 Beta from MSDN Subscriber downloads right now and the new Microsoft File Transfer Manager appears to be very very crap. The MSDN site insisted on updating the file transfer component before I could download and this version seems to stop every few minutes with errors about web server headers being missing. I have no proxies of my own that could be causing problems so I assume it's a wider problem than just me... Anyway, it's a pain that I have to keep hitting the Resume button ever few minutes...

The perils of precompiled headers

2008-09-03T10:44:03Z

I'm working on some client code today and it's fairly typical of most 'Visual Studio Wizard generated code' in that it uses precompiled headers in a pretty poor way. The default wizard generated code seems to encourage people down a slippery slope of just including everything in stdafx.cpp and not worrying about the consequences. Of course, by the time the project is large enough for this to be a problem the damage is already done and it's a nightmare to try and unpick the resulting problems.

I've mentioned before that I use precompiled headers in the "one true way" (see here for details). The advantage of my way is that I can always build without using precompiled headers if I want. Right now, with this client code, I want to...

I'm currently working on two cpp files and refactoring by moving code between two classes. The problem is that every time I change a header (and I'm doing that a lot) the whole project needs to be rebuilt. This leads to me having a lot of downtime during the whole refactor - compile - test cycle as the compile stage takes far too long. I've noticed that I've been browsing the web more, I'm shoving CDs into a drive to rip them and keep thinking that I should wander off and make tea. It's quite a difference to how things are when I'm working on code that uses precompiled headers the right way where the change - compile - test cycle is very fast when I'm developing and using the 'no precompiled headers' build configuration that I always include...

Ah well, I guess I get to catch up with some blogs this week and I doubt my hands will ache as much at the end of the day...

Flame on

2008-09-02T09:02:50Z

There's an interesting post over on "Blogging Considered Harmful" about why flaming someone for incorrect technical spouting is a good thing; I tend to agree. I actually WANT people to shout down my stupidity, if they're right and I'm wrong then hopefully I'll learn from the exchange. If I'm right and they're wrong then perhaps they'll learn from it. Either way it should make both sides think; though perhaps I'm being a little naive...

Take this recent exchange in the comments for one of my articles over on CodeProject. Someone, ostensibly called Alex Gris, with no other posting history on CodeProject looks at some of my code, does a 'code review', decides he's found bugs and reports them in a slightly aggressive manner. Fair enough, I admit that my first reaction was along the lines of WTF, you're wrong... However since I've been exposing myself to this kind of thing for a long time now and I've made plenty of stupid mistakes in the past and I've also sometimes defended stupid mistakes without double checking that I am actually right I tend to try and take a more careful line these days... So, the first thing I did was try and ignore the tone and deal with the content. My gut reaction was still "you're wrong, I'm right" but I loaded up the code in question and took a look at each of the issues... It took a little time and reviewed the areas where Alex had been critical. The end result seems to be one known and fixed bug, one incorrect problem report, one question of a potentially dubious design decision and one suggestion for an improvement that was theoretical and hand-wavy and that wasn't then followed up with some actual working code. If actual working code IS forthcoming from Alex and it IS a better design than what I have at present then my code will have been improved by placing it on the net in public, and that's good. If it doesn't then Alex has just made himself look a little silly and over-eager to shout down my code...

Of course I prefer this kind of critique to be packaged in a reasoned and friendly style (such as the problem report in the comments to this piece) but I realise that sometimes the people that know enough to be able to provide the technical feedback that I need are busy and have seen these kinds of mistakes far to many times before, or they're just gnarly old curmudgeons who don't care what I think of them but do want to publically correct technical errors... Either way I want the bug reports and the feedback that makes me think and makes me try and improve what I'm doing more than I want them to be packaged in a nice and fluffy way.

What really bugs me are people who really don't like to be disagreed with at all... I've no time for the blogging fan boys who like to set themselves up as authorities on things and then don't like to have to defend the positions that they take, some of them even don't allow comments... I'm sorry but "Don't you know who I am" just doesn't cut it ;) If you believe in what you're saying then argue your corner, don't just expect that the peons reading it should take it as gospel.

Of course there's always the danger of not spotting the point when these discussions get to the point where they're actually becoming circular and neither side is willing to accept that they might be wrong... Sometimes you just have to agree to disagree and move on, but I prefer to try and convince the other side that they're wrong, and sometimes the extended investigation and documentation required to do that actually proves that they're right...

So, flame on, it's good for us.