The life of a stream socket connection

As I mentioned here I’ve recently adjusted how socket callbacks are dispatched in The Server Framework.

Once you’ve written a TCP server or client you will find that you spend a lot of time dealing with the lifecycle of the connections that are created. You get to deal with a number of events which take place during a connections lifetime and, due to the way that the framework works, you can select just the events that interest you and ignore the rest. As we describe in the “Callbacks how to” article, you deal with connection events by providing the framework with an implementation of a callback interface and then being having it call you back when things happen. Note that we ignore connection filtering and the additional callbacks that it entails in this discussion.

There are two different kinds of connection that you will encounter and the events that occur for them are the same for the whole of the connection’s lifetime except for how the connection is established and how much control you have during this time. Inbound connections are the connections that clients establish with your server and outbound connections are the connections that your client establishes with a server. For an inbound connection a client connects to your server and you accept this connection and deal with it until the connection ends. For an outbound connection you initiate a connection to a server and deal with it until the connection ends.

Outbound connection establishment

Outbound connections give you slightly more control over the connection that you are creating. The first event that occurs depends on whether you have a connection limiter installed or not, if you do and you reach the limit of allowable connections then you will receive an OnMaximumConnections() event. If you are allowed to initiate a new connection then the first event that occurs on an outbound connection is OnSocketCreated() this gives you a chance to access the raw SOCKET that is associated with the connection. You should use this SOCKET with care. The purpose of this callback is to allow you to customise the connection in ways that the framework doesn’t explicitly support. You may wish to set certain socket options, for example. Next OnPreOutgoingConnect() is called. This provides you with the socket object’s “user data”, the address that you’re connecting to and a pointer to any “user data” that was passed to the call to Connect() or its equivalents. This is the point where you may wish to store any per-connection user data that was supplied by the code that is calling connect, in the socket. You would do this by adding “opaque user data” to the socket and then storing your per-connection data in the user data “slot” that you allocated. You can then access the user data from other callbacks. The attempt to connect to the remote server is now made. This can either be synchronous or asynchronous and it whichever it is it doesn’t affect the callbacks that you will receive. If the connection is successfully established then you will get an OnConnectionEstablished() event and if the connection fails you will get an OnOutgoingConnectionFailed() event. Depending on whether the connection attempt was successful you will now either become active or simply move to the connection destruction callback.

Inbound connection establishment

If you’re writing a server then you’re dealing with inbound connections. For an inbound connection the first thing you know about the connection is when OnConnectionEstablished() is called. The callback is the same as for outbound connections and for servers this would be where you might allocate and store any per-connection data that you have. You can see an example of this in the “Thread Pool Large Packet Echo Server” example. The connection is now active.

Active connection callbacks

Once a connection is established it remains active for a period of time. You will most likely interact with the connection by either reading from it, writing to it, or both. As soon as you either call Read() or get a response of true from TryRead() then you will receive, at any time thereafter, either a successful read completion, OnReadCompleted(), or a read completion error, OnReadCompletionError(). Likewise when you call Write() or get a response of true from TryWrite() then you will receive, at any time thereafter, either a successful write completion, OnWriteCompleted(), or a write completion error, OnWriteCompletionError(). Completion errors usually occur when there are insufficient system resources to complete the requested operation. You can’t usually do much in these situations and the best way of dealing with them is to make sure that you never have to deal with them. To that end you should always install a connection limiter and also be aware of how many outstanding read and write requests you have on a connection at any one time, possibly even installing a CFlowControlStreamSocketConnectionFilter if appropriate. Once you have issued a successful read or write you may also receive any of the connection termination or error callbacks.

Connection termination callbacks

At any time after the first read or write is issued you might get a connection termination event. There are two of these and which one you get depends on how the connection is terminated. If the remote end of the connection gracefully closes the send side of the connection from their side then you will get an OnConnectionClientClose() event. This occurs when a read returns successfully with zero bytes. You will only ever receive this callback once and once you have received it you wont get any more read completions or read completion errors no matter how many reads you have outstanding. Note that you can continue to send data to the remote end of the connection, the connection is now in the TCP Half Close state and you can continue to send, but not receive, until you either shutdown your send side of the connection or the connection is fully closed by the remote end of the connection. If the connection is reset for any reason (so not just a RST, any error that means the connection is no longer in a usable state) then you will receive the OnConnectionReset() event. This means that the connection is no longer usable, any pending reads or writes will complete in error and attempts to initiate any new reads or writes will fail. You can cause a connection to be reset by calling AbortConnection().

The socket closure callback

The lifecycle of a connection can be different to the lifecycle of the underlying SOCKET. The connection exists until both ends of it are shutdown. The socket exists until it is closed. Closing a socket causes the connection to be shutdown. There is no explicit way for a user of the framework to close a socket; when the number of pending reads and the number of pending writes and the number of externally held references to a socket all reach zero the socket is automatically closed. If you wish to cleanly shut down the TCP connection you can use Shutdown() and if you wish to reset the connection you can use AbortConnection(). When a socket is closed you receive notification via the OnConnectionClosed() event. This event is useful to help track active connections, perhaps for use in performance monitoring counters. This event is guaranteed to occur once for each connection for which there was a corresponding OnConnectionEstablished() event.

The connection destruction callback

When a connection is shutdown and no references to the connection’s socket object are held then OnSocketReleased() is called. This event is guaranteed to be the last event that you receive on a given connection. This is the ONLY place where you should clean up any per-connection user data that you may have stored in the socket as this is guaranteed to be the last event that will occur and when this event does occur you are guaranteed that no other events are still being processed and that no other code holds a reference to the connection. Note that you only get passed the user data and not the socket. Also note that you should not assume that you can use the address of the user data object for anything. Due to how the socket object’s inheritance works this particular interface is different to the instance of IIndexedOpaqueUserData that you can cast to from the IStreamSocket interface that you’re given in all of the other callbacks and the address of this user data will be different to the address of the user data passed in OnPreOutgoingConnect() for the same socket. Note that you can, of course, access the socket’s user data store via the interface, you just can’t use the address of the interface for anything useful.

Sequencing of callbacks

Since the framework is usually configured with a pool of multiple threads to service I/O requests the only guarantees that can be made about the order in which you will receive connection events are as follows: For both inbound and outbound connections you WILL receive the OnSocketReleased() event last and it will only occur when all other events have completed. You will also never receive a read or write completion or read or write error completion unless you have issued a read or a write call. For write calls you will ALWAYS receive either 1 write completion OR 1 error completion for each write that you issue. For read calls you will receive 1 read completion OR 1 error completion OR an OnConnectionClientClose() event. Once you have received an OnConnectionClientClose() event you will NOT receive any more completions for any outstanding reads that you might have, this is by design. For outbound connections you will recieve one call to OnSocketCreated() followed by one call to

OnPreOutgoingConnect(). Note that as soon as you have issued a successful read or write call then it is possible that you could receive error, client disconnect, connection reset or connection closed events at any time. It is therefore advisable to allocate and attach any per-connection user data before issuing the first read or write call and only delete it in OnSocketReleased().

Locks held during callbacks

In the past it’s been pretty easy to deadlock The Server Framework in complex servers if you don’t abide by the rules. Unfortunately, the rules weren’t documented until now and, although I knew them, it’s probably more accurate to say “I knew of them”. In more complex server developments I often went through a phase where I’d put some code in, in the wrong place, and deadlock the server under certain circumstances. Whilst documenting the rules I changed some code so that it’s less easy to deadlock The Server Framework…

The reason that it was easy to deadlock The Server Framework is that each socket has a lock to protect its internal state and keep it sane in the presence of multiple threads accessing it at the same time. This is a Good Thing. However, often, in more complex servers, the business logic of the server also manages locks. Due to the way the callbacks work there are two directions you can call into the framework. The first is the obvious one. You get an event callback and you make a call on the supplied socket, or the server or connection manager from within this callback. The second is that you take and hold a reference to a socket; by calling AddRef() and then call into the socket at a later time, perhaps because another connection has done something that your held socket needs to know about or perhaps because of an external event. Either way if you have your own lock, A, and the socket has its own lock, B, then when you call into the framework because of an external event you may lock A and then B and when the framework calls into you it may cause you to lock B and then A. From such lock inversions are deadlocks born.

The following callbacks DO NOT hold a lock on the socket whilst they call you:

  • OnSocketCreated()

  • OnPreOutgoingConnect()

  • OnConnectionEstablished()

  • OnOutgoingConnectionFailed()

  • OnMaximumConnections()

  • OnReadCompleted()

  • OnWriteCompleted()

  • OnConnectionClientClose()

  • OnConnectionReset()

  • OnConnectionClosed()

  • OnSocketReleased()

The following callbacks MAY hold a lock on the socket whilst they call you, so you should assume that they DO:

  • OnError()

  • OnWriteCompletionError() - only for sequenced sockets

  • OnReadCompletionError() - only for sequenced sockets

Up until version 5.2.3 of the framework the following callbacks either DID hold or MIGHT have held a lock:

  • OnConnectionClientClose() - only for sequenced sockets

  • OnConnectionReset()

  • OnConnectionClosed()

Updated: 6/8/2008 - previously we were being too cautious about race conditions with OnConnectionEstablished().