Tracking down reference counting bugs

The Server Framework uses reference counted data buffers and socket objects. These are passed around as part of the asynchronous I/O system in the framework and as they’re placed in queues or passed from one thread to another their reference counts are adjusted so that they are automatically cleaned up when they’re finished with; but not before.

If you’re writing a server or client using The Server Framework then you may need to manipulate these reference counts yourself; as with all reference counted objects, there are two simple rules: If you need to hold on to a reference to an object than call AddRef() on it; when you’re done with your reference call Release(). Sounds pretty simple but over the years various clients have had problems with it. Even I sometimes have problems with it and today I finally decided to make it easier for everyone to debug the leaks or memory corruption based crashes that happen when you get it wrong.

The next release of the framework, 6.1, will have an optional reference tracking system that can be enabled for sockets and buffers. Once enabled the system will keep track of the call stacks to all calls to AddRef() and Release() and if an over-release occurs (that is Release() is called when the reference count is already 0) then these call stacks will be dumped out to the debug trace. Something like this:

6060: CBufferReferenceTracker::DumpCalls - Over release
6060: Tracked data: 0000000000B7DAB0
6060: Reference: 1
6060: Location:
6060: e:\source\jetbytetools\iotools\bufferallocator.cpp (227): JetByteTools::IO::CBufferAllocator::Allocate
6060: e:\source\jetbytetools\iotools\test\buffertest.cpp (172): JetByteTools::IO::Test::CBufferTest::TestReferenceTrackingBufferOverRelease
6060: Reference: 0
6060: Location:
6060: e:\source\jetbytetools\iotools\buffer.cpp (314): JetByteTools::IO::CBuffer::Release
6060: e:\source\jetbytetools\iotools\test\buffertest.cpp (174): JetByteTools::IO::Test::CBufferTest::TestReferenceTrackingBufferOverRelease
6060: Reference: 0
6060: Location:
6060: e:\source\jetbytetools\iotools\buffer.cpp (299): JetByteTools::IO::CBuffer::Release
6060: e:\source\jetbytetools\iotools\test\buffertest.cpp (176): JetByteTools::IO::Test::CBufferTest::TestReferenceTrackingBufferOverRelease

If when the allocator is flushed at the end of the program there are still allocated buffers (or sockets) present then the associated call stacks will be dumped to the debug trace:

6060: CBufferReferenceTracker::DumpCalls - Buffer leak
6060: Tracked data: 0000000000B7DAB0
6060: Reference: 1
6060: Location:
6060: e:\source\jetbytetools\iotools\bufferallocator.cpp (227): JetByteTools::IO::CBufferAllocator::Allocate
6060: e:\source\jetbytetools\iotools\test\buffertest.cpp (153): JetByteTools::IO::Test::CBufferTest::TestReferenceTrackingBufferLeak
6060: Reference: 2
6060: Location:
6060: e:\source\jetbytetools\iotools\buffer.cpp (290): JetByteTools::IO::CBuffer::AddRef
6060: e:\source\jetbytetools\iotools\test\buffertest.cpp (155): JetByteTools::IO::Test::CBufferTest::TestReferenceTrackingBufferLeak
6060: Reference: 1
6060: Location:
6060: e:\source\jetbytetools\iotools\buffer.cpp (314): JetByteTools::IO::CBuffer::Release
6060: e:\source\jetbytetools\win32tools\referencecountedsmartpointer.h (259): JetByteTools::Win32::TReferenceCountedSmartPointer<JetByteTools::IO:IBuffer>::SafeRelease
6060: e:\source\jetbytetools\win32tools\referencecountedsmartpointer.h (146): JetByteTools::Win32::TReferenceCountedSmartPointer<JetByteTools::IO:IBuffer>::~TReferenceCountedSmartPointer<JetByteTools::IO:IBuffer>
6060: e:\source\jetbytetools\iotools\test\buffertest.cpp (161): JetByteTools::IO::Test::CBufferTest::TestReferenceTrackingBufferLeak

Hopefully this will speed up the process of finding out where and when the reference count was manipulated and where the bug is. Obviously the over-release tracing relies on the fact that often the memory that was used for the object is not altered after it has been deleted and thus the dangling pointer to the object can know that the reference count is already zero. This wont always be true, but in cases where the existing exception is firing to warn you of this problem you’ll now also get a reference count manipulation dump.

Note that enabling these changes is completely unintrusive. You don’t need to change any code, you simply add a define to your Config.h file and rebuild.