Two quite different approaches to multi-threading

| 0 Comments

I've been working on some code for a client recently that needs to run in a multi-threaded environment. Unfortunately it was never really written with that requirement in mind, or, more likely, the people who wrote the code knew that it needed to be accessed from multiple threads but didn't really understand quite what that meant. As such I'm doing some fairly hairy refactoring for them. It's high risk as there are no unit tests and the budget doesn't really extend to completing the work, let alone "spending extra time" writing unit tests... The code is written in quite a 'C' style, most things are simple structs and most data is public and as such member functions fall where they are most conveniently written. Adding correct locking is complicated due to lots of mutable global data, lack of encapsulation and the fact that, at its heart, it's multi-user server software where one user gets to poke at all the others.


Step zero was to realise that no locking is bad but correct locking was a long way away so I slipped in a single "global lock" to make the system work. It's stable and not too slow but it doesn't really ever use more than one thread at a time as almost everything requires that the executing thread holds the "global lock".


Step one is to add some encapsulation and move things around so that we can one day dream of adding in more fine grained locking - unfortunately this step is a big one.


I'm finding it interesting comparing this work to the work I was doing immediately before. I've been working on the next big release of The Server Framework and this release will put an Activatable Object at the heart of each connection. This removes the need for locking from the socket object and lots of stuff that's built on top of it. It means that masses of complex code can be pulled out and thrown away. It's not a new idea, the socket object is now simply driven by a message queue. Locks are needed to put work items into the queue but during processing we can run without locking, safe in the knowledge that only one thread can ever be processing at a time.


The problem is that making the kind of change that I'm making right now is a massive change. It has led to a complete rewrite of the entire TCP side of The Server Framework. The general pattern is removing locking, because it's no longer needed and simplifying code because the ways in which it can be called are simpler. The unit tests that I had for the previous version of the code have helped to guide me; yes new tests have been written, but the old ones told me what I needed to test even if they needed to be changed a lot to work with the new code.


With The Server Framework work it was scary but the various layers of tests supported the work and, in general, the code was being made simpler and easier to understand. With the client work there are no tests, huge amounts of the application simply didn't compile until the changes were complete and, whilst some things are being simplified, others are being made more complex to accommodate changes to make the code thread safe. I expect that with enough iterations the client's code could shift to using Activatable objects and some of the locking would go away again but I'm not sure we'll have the luxury of those extra iterations.


How to build a GCC Cross-Compiler

| 0 Comments
This article over on Preshing on Programming looks useful. It gives a step by step guide for building GCC cross-compilers, I expect it will save me lots of time at some point in the future.

Dropping support for Visual Studio 2005 and 2008

| 0 Comments
The 7.0 release of The Server Framework, which is likely to be released early next year, will no longer support Visual Studio 2005 or Visual Studio 2008.

The 6.6.x releases will be the last to support these compilers.

Please get in touch immediately if this will be a problem for you.

Dropping support for Windows XP and Server 2003

| 2 Comments
The 7.0 release of The Server Framework, which is likely to be released early next year, will no longer support Windows XP or Windows Server 2003.

The 6.6.x releases will be the last to support these operating systems. Release 6.6.3, is due shortly and is a minor bug fixing release. We may release subsequent 6.6.x bug fix releases but no new development will occur on the 6.6 branch.

Removal of support for these operating systems allows us to clean up the code considerably and to remove lots of code that's required purely to work around 'interesting' twists in various Windows APIs pre-Vista.

New option pack: Streaming Media

| 0 Comments
We have a new Option Pack, The Streaming Media Option Pack. This allows you to easily add streaming of H.264 and MPEG audio and video to your clients and servers using RTSP, RTP and RTCP.

With more and more Internet Of Things devices supporting rich media streaming for remote monitoring it's becoming essential to have the ability to manage these media streams within your device management servers and clients. Whether it's recording device streams for later analysis or arbitrating between multiple clients and devices, manipulating streaming media is becoming more and more important.

As always, this Option Pack integrates seamlessly with The Server Framework's Core Framework and other options and allows you to quickly and easily add rich media support.

Surprising Slim Reader/Writer Lock thread exit issues.

| 1 Comment

I've been noticing a strange thing for a while on Windows 8/8.1 and the equivalent server versions. The issue occurs when I'm using a Slim Reader/Writer Lock (SRWL) exclusively in exclusive mode (as a replacement for critical sections). What happens is, when a thread that has just unlocked a SRWL exits cleanly, immediately after unlocking the lock, sometimes threads that are waiting on the lock do not get woken and none of them acquire the lock.

At first I spent ages thinking that this was some kind of subtle bug in my LockExplorer tool as initially the problem only manifested itself during test runs that were using LockExplorer to detect potential deadlocks. Recently, however, I've been seeing the identical problem in normal programs being run with no clever lock instrumentation going on.

I still think it's more likely my bug than the operating system's but I recently found this Knowledge Base article, #2582203, "A process that is being terminated stops responding in Windows 7 or in Windows Server 2008 R2" which says "This issue occurs because the main thread that terminates the process tries to reactivate another thread that was terminated when the thread released an SRW lock. This race condition causes the process to stop responding.". This sounds suspiciously like the problem that I'm seeing, though not exactly.

The problem I see is that other threads waiting on the SWRL don't get notified that the lock is now unlocked. This locks my program up during shutdown purely because I have code that waits for these threads to exit cleanly and they can't as they're waiting on an unlocked SRWL. It's not a deadlocked attempt at recursive SWRL acquisition as I have debug code in my wrapper class which detects such behaviour. It's not an orphaned locked SRWL as breaking the hung process into the debugger and immediately continuing it causes one of the threads that's blocking on the lock to immediately acquire it and continue.

I haven't applied the hotfix yet, partly because my systems are Win 8 rather than Win 7 and partly because I'm not yet convinced that this is the issue.

As a work around I've added a call to Sleep at the point where my thread class allows a thread to exit. This seems to reduce the instances of the problem, which leads me to believe it's a race condition of some kind.

New client profile: Eonic Gaming - Turf Battles

| 0 Comments
We have a new client profile available here for a new client who is using The Server Framework to power their MMORPG game server.

Eonic Gaming - Turf Battles Triumphus - Server development

We're pleased to be working with Eonic Gaming in the development of the server for their Turf Battles Triumphus 3D MMORPG.

Eonic have selected us to replace their existing networking code with The Server Framework to improve the stability and performance of their server.

Activatable Object

| 0 Comments

I've written an article for Overload, one of the the ACCU's journals. It's based on my Efficient Multi-Threading blog post from a few weeks ago. Chris Oldwood mentioned to me about how the object described in Efficient Multi-Threading was similar to an Active Object which steals a calling thread to do its work rather than using one of its own and I agreed, he then suggested that I write it up for the ACCU journal.

You can read the article here.

Latest release of The Server Framework: 6.6.2

| 0 Comments
Version 6.6.2 of The Server Framework was released today.

This release is a bug fix and internal feature release which results in lots of change to the OpenSSL, SChannel, Compression and TCP flow control filters. See the release notes here, for full details of all changes.

All clients using earlier versions of both the OpenSSL Option Pack and the SChannel Option Pack are advised to upgrade to this release.

Bug fixes:

  • Bug fixes and potential fixes to all code that used function level non-trivial static objects. These have all been replaced with alternative designs as they were not thread safe and could, in very unlikely scenarios, cause problems. See here for more details.
  • Bug fix for JetByteTools::Win32::IQueueTimers::SetTimerWithRefCountedUserData(). We now wrap the SetTimer() call in a try/catch block and Release() the reference we created with AddRef() if there's an exception.
  • Bug fix to JetByteTools::IO::CBufferChain::AddDataToBufferChain() to allow for passing in an empty list of buffers to add.
  • Bug fix to JetByteTools::IO::CBufferChain::ConsoliateData() to ensure that buffers that we attempt to consolidate are writeable.
  • Bug fix in JetByteTools::Socket::TConnectionManagerBase::ReleaseSocket() to ensure exceptions are trapped appropriately.
  • Bug fix in JetByteTools::Service::CShutdownHandler::Run(). Changed the way we respond to shutdown and pause events (if enabled) so that we only respond to each event once rather than repeatedly routing shutdown events to the service whilst the event is set. This was causing non-paged pool exhaustion in some situations when the service failed to shutdown.

Additions:

  • Added JetByteTools::Win32::CRecursiveDirectorySearch.
  • Added JetByteTools::Win32::CRegistryKey::TryDeleteValue().
  • Added JetByteTools::Win32::TReusableIdManager::TryFree().
  • Added JetByteTools::Win32::TThreadSafeReusableIdManager::TryFree().
  • Added JetByteTools::Win32::CRingBuffer::GetSize().
  • Added JetByteTools::Win32::CActivatableObject. See here for details.
  • Added JetByteTools::Win32::CalculateRequiredPrecision() which takes a double value and returns the optimum precision required to format the value for display.
  • Added Intrusive containers. A set, map and multi-map which can be used to replace STL containers with ones which do not need to do memory operations during insertion and removal.
  • Added JetByteTools::IO::IBufferBase::GetSpace() which returns the available space in a buffer, size - used.
  • Added JetByteTools::IO::IBuffer::GetTotalSpace() which returns all available space in a buffer, this is GetSpace() plus any space that is currently unused at the front of the buffer due to calls to Consume().
  • Added JetByteTools::IO::IBuffer::RemoveSpaceAtFront() which compacts a buffer by removing any unused space at the front and copying any data back towards the front of the buffer.
  • Added JetByteTools::IO::CSequencedBufferCollection which is a collection based on JetByteTools::Win32::TIntrusiveRedBlackTree which stores the buffers in sequence number order.
  • Added JetByteTools::Socket::CFilterDataBase a base class for filter data that is based on JetByteTools::Win32::CActivatableObject.
  • Added JetByteTools::Socket::CWriteOnlyFilterData and JetByteTools::Socket::CReadOnlyFilterData as base classes for filters which only work on one side of the connection.
  • Added JetByteTools::Socket::CStreamSocketConnectionFilterBase a base class for filters which use JetByteTools::Socket::CFilterDataBase.
  • Added lots of functionality to JetByteTools::Socket::CFilteringStreamSocketConnectionManagerBase to allow for more complete filtering.
  • Added a Datagram flow control filter, JetByteTools::Socket::CFlowControlDatagramSocketConnectionFilter, see here for more details.

Changes:

  • Completely redesigned the OpenSSL and SChannel filters so that they use the new JetByteTools::Socket::CFilterDataBase filter data base classes.
  • Changed all versions of JetByteTools::Win32::ToString() which take a double or long double to also accept a precision value. This value defaults to 6, which is the default precision used if precision isn't specified.
  • Changed JetByteTools::Win32::CreateDirectoryIfRequired() to return a bool which indicates if the directory was created.
  • Changed JetByteTools::Win32::CreateDirectoriesIfRequired() to return a count of the number of directories created.
  • Changed JetByteTools::Win32::CThreadPool::OnThreadPoolThreadStopped() so that it calls JetByteTools::Win32::IMonitorThreadPool::OnThreadPoolThreadStopped() before deleting the thread and decrementing the counter. This makes it possible to clean up the monitor when the final thread is deleted (and the wait on the counter returns).
  • Changed JetByteTools::Win32::TLockableObject so that it only uses a SRWL if we're building for Windows 7 or later. This is so that the TryEnter() API is available for the SRWL.
  • We now use JetByteTools::Win32::CombinePath() where possible rather than combining paths by hand.
  • Changed JetbyteTools::Win32::CCallbackTimerQueue and JetbyteTools::Win32::CCallbackTimerWheel to use the new intrusive containers.
  • Changed JetByteTools::IO::IBuffer::MakeSpaceAtFront() so that it takes an optional value which indicates how much space must be available at the rear of the buffer. This allows us to be able to create space at the front only if there's enough space in the buffer for the space we need at the front and rear of the buffer.
  • Changed JetByteTools::IO::CBufferChainStoresNulls so that you can pass it an instance of JetByteTools::IO::IAllocateBuffer in its constructor and it can then use that to allocate a new buffer if the number of 'null buffers' stored becomes too great. This effectively removes the limit on the number of 'null buffers' that can be stored between non-null buffers.
  • Changed JetByteTools::IO::CSortedBufferChain so that it caches whether you can get the next buffer or not. This reduces the work that needs to be done to find out if you can get the next buffer.
  • Changed the implementation of JetByteTools::Socket::CCompressingStreamSocketConnectionFilter so that it uses the new filter base classes.
  • Changed the implementation of JetByteTools::Socket::CFlowControlStreamSocketConnectionFilter so that it uses the new filter base classes.

About this Blog

I usually write about C++ development on Windows platforms, but I often ramble on about other less technical stuff...

This page contains recent content. Look in the archives to find all content.

I have other blogs...

Subscribe to feed The Server Framework - high performance server development
Subscribe to feed Lock Explorer - deadlock detection and multi-threaded performance tools
Subscribe to feed l'Hexapod - embedded electronics and robotics
Subscribe to feed MegèveSki - skiing