Custom client/server request/response protocols

Quite often my customers use The Server Framework for both ends of their communication channels. The Server Framework has fully supported developing clients as well as servers for a long time now and although many of my customers build either servers or clients with the framework some build both. One of the things that often comes up in discussions with these customers is how to develop a custom request/response protocol for their servers and clients to communicate with. Now, of course, there’s often a perfectly good existing, standard, protocol that they can use. But sometimes, for whatever reason, there isn’t, and I sketch out a simple protocol that I’ve been using for some time for simple request/response situations. Since the framework has always grown by harvesting code that I find I reuse a lot, it’s time for an implementation of this simple request/response protocol to find its way into the examples that ship with the framework.

The request response protocol was originally designed to provide the kind of thing that DCOM was being used for by one of my clients but in a cross platform, less bulky, no objects kind of way. In fact, although the client was using DCOM at the time, the replacement protocol was much more like DCE/RPC than DCOM. The main functionality that was required was the following:

  • synchronous request/response (client code looks like a normal, local, function call).

  • asynchronous requests (fire and forget).

  • asynchronous responses (server can push data to the client).

Subsequent customer projects have also added the following additional requirements:

  • asynchronous request/response (client gets a callback when the response arrives).

  • polled asynchronous request/response (client fires off a request and later comes back to see if a response has arrived).

Of course the protocol should be easy to implement in an efficient manner on the server and should allow for multiple synchronous calls from the client to the server to be in progress at any one time on the same connection (i.e. we should be able to call from the client to the server in a synchronous manner from multiple threads at once without having to open a new connection for each thread).

Due to the asynchronous nature of the framework itself the synchronous calls were the hardest requirement to implement. However, once the request/response matching code is implemented the synchronous calls are simply asynchronous request/response messages with a particular callback into another part of the protocol handling library.

Both requests and responses share the same message structure. To aid message parsing all messages start with a 4 byte message length indicator, this allows the server or client to accumulate complete messages before processing them. To enable the client to match responses to requests all messages contain a 4 byte invocation id. The value of this id is irrelevant to the server, but it must be echoed back in any response. For asynchronous messages the invocation id should be 0. For synchronous requests the client should set the invocation id to be something other than zero and then wait a response to arrive with a matching invocation id before returning to its caller. Everything else is the message body. The length indicator in the 8 byte message header includes the size of the header and the length of the message body.

With this simple protocol in place we can then build whatever command and response system we want on top of it. For most projects the message body would first contain a command code which is then used within the server to determine what operation the client wishes to perform. Following the command code we have the command specific data which has been marshalled into a form that can be transmitted to the server. This is where DCE/RPC or DCOM have an advantage as they have their own compilers that build the custom marshalling code required from descriptions of the calls and responses specified in an IDL file. Whilst many of our customers have implemented their marshalling explicitly there’s nothing to stop you using something like Google’s Protocol Buffers to ease the development of your command messages and responses and their marshalling.

I’ll be harvesting the request/response protocol code over the next few days. Once complete some example clients and servers will be available for download from the usual place. I expect that these examples will run with the 6.0 release of the framework, but if I decide to make any framework changes whilst I’m harvesting then that may prompt the release of all of the currently accumulating 6.1 features.