More Socket Server Refactoring

| 2 Comments
I'm currently working on a simple auction server for a client using The Server Framework. You can think of it as a specialised chat server, of sorts. One of the things it must do is broadcast messages from one user to a set of users. This is relatively easy to implement in a crude way in The Server Framework but it's not nice. Time to refactor towards niceness...

The lack of niceness in the quick and dirty implementation of broadcast messages is the fact that the message must be duplicated for each client that will receive the broadcast. This sucks for many reasons; it's not especially elegant, it's not especially necessary, it involves doing work that needn't be done and it hogs memory. Unfortunately our last bout of server framework refactoring stopped short of where we needed to be to be able to fix these issues.

The Server Framework uses fixed sized buffers that are managed by a buffer allocator (which can pool them for later use and pre-allocate a set number of buffers, etc). The buffer consists of a blob of memory that's used as the actual data buffer plus an OVERLAPPED structure for using the buffer in overlapped IO operations and a WSABUF structure for using the buffer in socket operations... When we wish to broadcast a buffer we want to reuse the memory blob and, possibly, the WSABUF but we need a unique OVERLAPPED structure so that each overlapped write operation has its own workspace... The framework could allocate its own OVERLAPPED and WSABUF structures and pool them, etc, but during the original design phase I decided to limit the number of memory allocations required during message processing by incorporating the buffer and all of its supporting structures in a single allocation - seemed like a good idea at the time and it's stood the test of time pretty well; until now.

What we'd really like to be able to do is attach multiple OVERLAPPED (and just to be on the safe side, WSABUF) structures to a single blob of memory. This would allow us to receive a message in a buffer and then transmit that buffer to multiple clients without needing to copy the message from the input buffer into each of the output buffers.

As usual the reason the existing design isn't quite flexible enough is because of the use of concrete classes rather than interfaces. The server deals in terms of CBuffer objects rather than in terms of an IBuffer interface. So, this morning, I slipped in an interface to decouple the users of the buffer class from its current implementation. That was reasonably painless, so we then added a 'read only buffer handle' class that implemented IBuffer and provided its own OVERLAPPED and WSABUF structures but references the original data in the original IBuffer implementation. This worked nicely and greatly reduced the memory usage of the server and improved the performance of the broadcasting.

The next step is to generalise the IAllocateBuffers interface so that we can share the pooling that we use for the real buffers. Once that's done it would be good to completely decouple the book-keeping parts of the buffer from the data parts, then, perhaps we can support buffers that consist of multiple memory buffers and that present arrays of WSABUFs to the socket layer...

2 Comments

Len,

I am thinking of using your socket server framework to develop something similar to a chat server as well. The requirements call for real-time broadcasting (so no polling the server for new data) and the server needs to initially handle 10,000 simultaneous connections.

Do you have any insight into how to maximize the number of simultaneous connections that one server can handle? It will be on a Windows 2003 server machine, with 4 GBs of RAM and one CPU. Assume that bandwidth will not be an issue. Obviously memory and the number of open file handles allowed by the OS will be an issue...

Use IOCP and overlapped IO to minimise the number of threads that you need. Issue 0 byte overlapped IO reads and then when they complete issue a sync read for the amount of data available; this helps with lots of concurrent connections as it prevents your server from using up non-paged pool memory with buffers for pending reads.

Leave a comment