I've written an article for Overload, one of the the ACCU's journals. It's based on my Efficient Multi-Threading blog post from a few weeks ago. Chris Oldwood mentioned to me about how the object described in Efficient Multi-Threading was similar to an Active Object which steals a calling thread to do its work rather than using one of its own and I agreed, he then suggested that I write it up for the ACCU journal.You can read the article here.
- Bug fixes and potential fixes to all code that used function level non-trivial static objects. These have all been replaced with alternative designs as they were not thread safe and could, in very unlikely scenarios, cause problems. See here for more details.
- Bug fix for
JetByteTools::Win32::IQueueTimers::SetTimerWithRefCountedUserData(). We now wrap the
SetTimer()call in a try/catch block and
Release()the reference we created with
AddRef()if there's an exception.
- Bug fix to
JetByteTools::IO::CBufferChain::AddDataToBufferChain()to allow for passing in an empty list of buffers to add.
- Bug fix to
JetByteTools::IO::CBufferChain::ConsoliateData()to ensure that buffers that we attempt to consolidate are writeable.
- Bug fix in
JetByteTools::Socket::TConnectionManagerBase::ReleaseSocket()to ensure exceptions are trapped appropriately.
- Bug fix in
JetByteTools::Service::CShutdownHandler::Run(). Changed the way we respond to shutdown and pause events (if enabled) so that we only respond to each event once rather than repeatedly routing shutdown events to the service whilst the event is set. This was causing non-paged pool exhaustion in some situations when the service failed to shutdown.
JetByteTools::Win32::CActivatableObject. See here for details.
JetByteTools::Win32::CalculateRequiredPrecision()which takes a double value and returns the optimum precision required to format the value for display.
- Added Intrusive containers. A set, map and multi-map which can be used to replace STL containers with ones which do not need to do memory operations during insertion and removal.
JetByteTools::IO::IBufferBase::GetSpace()which returns the available space in a buffer, size - used.
JetByteTools::IO::IBuffer::GetTotalSpace()which returns all available space in a buffer, this is
GetSpace()plus any space that is currently unused at the front of the buffer due to calls to
JetByteTools::IO::IBuffer::RemoveSpaceAtFront()which compacts a buffer by removing any unused space at the front and copying any data back towards the front of the buffer.
JetByteTools::IO::CSequencedBufferCollectionwhich is a collection based on
JetByteTools::Win32::TIntrusiveRedBlackTreewhich stores the buffers in sequence number order.
JetByteTools::Socket::CFilterDataBasea base class for filter data that is based on
JetByteTools::Socket::CReadOnlyFilterDataas base classes for filters which only work on one side of the connection.
JetByteTools::Socket::CStreamSocketConnectionFilterBasea base class for filters which use
- Added lots of functionality to
JetByteTools::Socket::CFilteringStreamSocketConnectionManagerBaseto allow for more complete filtering.
- Added a Datagram flow control filter,
JetByteTools::Socket::CFlowControlDatagramSocketConnectionFilter, see here for more details.
- Completely redesigned the OpenSSL and SChannel filters so that they use the new
JetByteTools::Socket::CFilterDataBasefilter data base classes.
- Changed all versions of
JetByteTools::Win32::ToString()which take a double or long double to also accept a precision value. This value defaults to 6, which is the default precision used if precision isn't specified.
JetByteTools::Win32::CreateDirectoryIfRequired()to return a bool which indicates if the directory was created.
JetByteTools::Win32::CreateDirectoriesIfRequired()to return a count of the number of directories created.
JetByteTools::Win32::CThreadPool::OnThreadPoolThreadStopped()so that it calls
JetByteTools::Win32::IMonitorThreadPool::OnThreadPoolThreadStopped()before deleting the thread and decrementing the counter. This makes it possible to clean up the monitor when the final thread is deleted (and the wait on the counter returns).
JetByteTools::Win32::TLockableObjectso that it only uses a SRWL if we're building for Windows 7 or later. This is so that the
TryEnter()API is available for the SRWL.
- We now use
JetByteTools::Win32::CombinePath()where possible rather than combining paths by hand.
JetbyteTools::Win32::CCallbackTimerWheelto use the new intrusive containers.
JetByteTools::IO::IBuffer::MakeSpaceAtFront()so that it takes an optional value which indicates how much space must be available at the rear of the buffer. This allows us to be able to create space at the front only if there's enough space in the buffer for the space we need at the front and rear of the buffer.
JetByteTools::IO::CBufferChainStoresNullsso that you can pass it an instance of
JetByteTools::IO::IAllocateBufferin its constructor and it can then use that to allocate a new buffer if the number of 'null buffers' stored becomes too great. This effectively removes the limit on the number of 'null buffers' that can be stored between non-null buffers.
JetByteTools::IO::CSortedBufferChainso that it caches whether you can get the next buffer or not. This reduces the work that needs to be done to find out if you can get the next buffer.
- Changed the implementation of
JetByteTools::Socket::CCompressingStreamSocketConnectionFilterso that it uses the new filter base classes.
- Changed the implementation of
JetByteTools::Socket::CFlowControlStreamSocketConnectionFilterso that it uses the new filter base classes.
I don't believe that UDP should require any flow control in the sending application. After all, it's unreliable and it should be quite OK for any stage of the route from one peer to another to decide to drop a datagram for any reason. However, it seems that, on Window's at least, no datagrams will be dropped between the application and the network interface card (NIC) driver, no matter how heavily you load the system.
Unfortunately most NIC drivers also prefer not to drop datagrams, even if they're overloaded (see here for details of how UDP checksum offloading can cause a NIC to make non-paged pool usage considerably higher). This can lead to situations where a user mode application can bring a box down due to non-paged pool exhaustion simply by sending as many datagrams as it can as fast as it can. It's likely that it's actually poorly implemented device drivers that are at fault here; by failing to gracefully handle situations where non-paged pool allocations fail, but it's the application that is putting these drivers into a situation where they could fail in such a catastrophic manner.Since the NIC driver and the operating system will not drop datagrams it's down to the application itself to do so if it senses that it's overloading the NIC. I've recently added code to The Server Framework to allow you to configure behaviour like this so that an application can prevent itself from exhausting non-paged pool due to pending datagram writes.
Performance is always important for users of The Server Framework and I often spend time profiling the code and thinking about ways to improve performance. Hardware has changed considerably since I first designed The Server Framework back in 2001 and some things that worked well enough back then are now serious impediments to further performance gains. That's not to say that the performance of The Server Framework today is bad, it's not, it's just that in some situations and on some hardware it could be even better.
One of the things that I changed last year was how we dispatch operations on a connection. Before the changes multiple I/O threads could block each other whilst operations were dispatched for a single connection. This was unnecessary and could reduce the throughput on all connections when one connection had several operation completions pending. I fixed this by adding a per-connection operation queue and ensuring that only one I/O thread at a time was ever processing operations for a given connection; other I/O threads would simply add operations for that connection to its operation queue and only process them if no other thread was already processing for that connection.
Whilst those changes deal with serialising multiple concurrent I/O completion events on a connection there's still the potential for multiple threads accessing a given connection's state if non-I/O threads are issuing read and write operations on the connection. Not all connections need to worry about the interaction of a read request and a completion from a previous request, but things like SSL engines may care. We have several areas in The Server Framework, and more in complex server's built on top of the framework, where it's important that only a single operation is being processed at a time. Locks are currently used to ensure that only a single operation is processed at a time, but this blocks other threads. What's more most of the code that cares about this also cares about calling out to user callback interfaces without holding locks. Holding locks when calling back into user code is an easy way to create unexpected lock inversions which can deadlock a system.
In the diagram above, given the threads 1, 2, 3 & 4 with four work items A, B, C & D for a single object, the threads are serialised and block until each can process its own operation on the object. This is bad for performance, bad for contention and bad for locality of reference as the data object can be bounced back and forth between different CPU caches.The code that I describe in the rest of this article forms the basis of a new "single threaded processor" object which enables efficient command queuing from multiple threads where commands are only ever processed on one thread at a time, minimal thread blocking occurs, no locks are held whilst processing and an object tends to stay on the same thread whilst there's work to be done.
There are a couple of undocumented Visual Studio compiler switches which can be useful occasionally:
- which produces a dump of the in memory layout of a given class
- /d1reportAllClassLayout - which does the same for ALL classes
Recently, whilst continuing to improve the performance of various aspects of The Server Framework, I reached a point where I found I needed to replace some STL containers with intrusive containers which didn't need to perform memory allocations during data insertion and removal. I had been toying with the idea of implementing custom containers for some time but have only recently had a pressing need for them.The STL provides some marvellously usable code which has changed the way people view container classes in C++. The result is that you hardly ever need to write your own containers as the STL provides containers with known performance characteristics which, thanks to templates, work with any data. The downside of this is that you hardly ever need to write your own containers and so you get out of the habit of doing so...
Previously on "Practical Testing"... I'm in the process of replacing STL containers with custom intrusive containers in the timer system that I have been developing in this series of articles. The idea is that the intrusive containers do not require memory operations for insertion or deletion as the book-keeping data required to store the data in the container has been added to the data explicitly. This reduces the potential contention between threads in the application and, hopefully, improves overall performance.In the last instalment I showed how my usage of
std::setin the timer wheel code could be replaced by an intrusive set. This time I'll also replace the
std::mapfrom the timer queue. These are quite major internal changes but since we have a large set of unit tests it's reasonably easy to make these changes without having to worry about breaking the external interface.
Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests.
In 2010 I needed to improve the performance of the code for a specific use case. The resulting changes saw the creation of a Timer Wheel version of the code which has the same interface and uses the same tests. I also ruminated on contention, invented a crazy notation to help me talk about contention and experimented with custom STL allocators and private heaps in an attempt to reduce it. In the end I decided that the best approach was probably to use custom containers rather than the STL. The custom containers would be intrusive (or invasive as I called it at the time) and would avoid the need for memory allocation on insertion and removal of items.I didn't have a pressing need to improve the performance further and so the intrusive container idea slipped down my list of things to do. Recently I've had a need for an intrusive map for another part of The Server Framework and have implemented some custom containers, set, map, multi-map, based on an intrusive red-black tree that I've written. It then seemed appropriate to integrate them into the timer queue and see what difference they make. That's what I'm doing in this instalment. As usual the existing tests support these, quite major, internal changes to the code.
test byte ptr ds:[5C87D0h],1 jne JetByteTools::IO::CBufferChainStoresNulls::InternalAddToFront+77h (0473967h) or dword ptr ds:[5C87D0h],1 xor eax,eax mov dword ptr [esp+1Ch],eax push 2Dh push 56DFE8h mov ecx,5C87B4h mov dword ptr ds:[5C87CCh],7 mov dword ptr ds:[005C87C8h],eax mov word ptr ds:[005C87B8h],ax call std::basic_string
,std::allocator >::assign (0404130h) push 55C8B0h call atexit (047E190h) add esp,4 mov dword ptr [esp+1Ch],0FFFFFFFFh void CBufferChainStoresNulls::InternalAddToFront( mov edi,dword ptr [esi+8]
TIME_WAITstate. It can be downloaded from the links later in this post.
TIME_WAITstate (see here for why this can be a problem). This was affecting their connectivity. Rather surprisingly there seemed to be no way to track the number of sockets in
TIME_WAITusing perfmon as there didn't seem to be a counter exposed. A counter would have been useful so that we could track the
TIME_WAIT; connections over time along with all of the other metrics that we track for their system. Anyway, I put together a quick and dirty tool for the client, this worked like a special version of
netstatwhich totalled up the number of sockets in each state by polling the system using
CLOSE_WAIT, etc. There's an x86 and and x64 build available and the services come as a single exe which automatically deploy and install the required counter dlls when you run them with /install.
- Unzip. You will end up with either
TCPStatsPerfCounters64.exe. All of the rest of the instructions apply equally to either version of the program.
/?command line switch to see help.
/installcommand line switch to install the service. You may be prompted to elevate your credentials if you are not an admin.
- To run as a service, open the services applet and locate the "JetByte TCP Stats Perfmon Counters" service, start it and, optionally set its start-up type to automatic.
- To run
TCPStatsPerfCounters.exeas an exe rather than as a service you still need to have installed it as a service but you can run from the command line with the
/runcommand line switch.
- You can get
TCPStatsPerfCounters.exeto produce a log file - this may be useful if you want to track socket states outside of perfmon. Either
/runwith the additional
/createLogcommand line switch. By default the log is created in the same directory as the exe, you can change this with the addition of the
/logPathcommand line switch.
- By default the program polls the system every second, you can change this by using the
/pollcommand line switch (again supply this either with
/run. The polling interval is specified in seconds.
CountersThere are counters for all of the socket states for both IPv4 and IPv6. Sockets transition through some of the states very quickly and so you may not see many of the states. There are also maximum counters for
TIME_WAITwhich show the highest value seen. In addition there are counters for the
TcpTimedWaitDelayregistry keys. I found it useful to have these values clearly visible for situations where machines happened to be configured in an unusual way and nobody had thought to check the registry. These values are set once on program start up and will be zero if the registry key is not set and the operating system's default value is in operation.