I've built a small Windows Service which exposes perfmon counters to track sockets in TIME_WAIT state. It can be downloaded from the links later in this post.

Back in 2011 I was helping a client look for issues in their systems caused by having too many sockets in a TIME_WAIT state (see here for why this can be a problem). This was affecting their connectivit. Rather surprisingly there seemed to be no way to track the number of sockets in TIME_WAIT using perfmon as there didn't seem to be a counter exposed. A counter would have been useful so that we could track the TIME_WAIT; connections over time along with all of the other metrics that we track for their system. Anyway, I put together a quick and dirty tool for the client, this worked like a special version of netstat which totalled up the number of sockets in each state by polling the system using GetExtendedTcpTable() (see here).

At the time I suggested that it wouldn't take much to build a small service which did this polling and exposed the results as a perfmon counter so that we COULD track this metric in the usual way. Well, I've finally got around to doing that (thanks to the encouragement of another client who had a similar issue). The result is TCPStatsPerfCounters which is a service that you can install and which provides counters for all of the states in the TCP table for both IPv4 and IPv6; ESTABLISHED, TIME_WAIT, CLOSE_WAIT, etc. There's an x86 and and x64 build available and the services come as a single exe which automatically deploy and install the required counter dlls when you run them with /install.

• Download the x86 version of TCPStatsPerfCounters from here.
• Download the x64 version of TCPStatsPerfCounters from here.

Note that the x64 version will install an x86 and an x64 counter dll for maximum interoperability. The x86 version only installs an x86 counter dll and so will not integrate with perfmon on an x64 box. You should only use the x86 version on x86 machines.

Usage

• Unzip. You will end up with either TCPStatsPerfCounters.exe or TCPStatsPerfCounters64.exe. All of the rest of the instructions apply equally to either version of the program.
• Run TCPStatsPerfCounters.exe with the /? command line switch to see help.
• Run TCPStatsPerfCounters.exe with the /install command line switch to install the service. You may be prompted to elevate your credentials if you are not an admin.
• To run as a service, open the services applet and locate the "JetByte TCP Stats Perfmon Counters" service, start it and, optionally set its start-up type to automatic.
• To run TCPStatsPerfCounters.exe as an exe rather than as a service you still need to have installed it as a service but you can run from the command line with the /run command line switch.
• You can get TCPStatsPerfCounters.exe to produce a log file - this may be useful if you want to track socket states outside of perfmon. Either /install or /run with the additional /createLog command line switch. By default the log is created in the same directory as the exe, you can change this with the addition of the /logPath command line switch.
• By default the program polls the system every second, you can change this by using the /poll command line switch (again supply this either with /install or /run. The polling interval is specified in seconds.

Counters

There are counters for all of the socket states for both IPv4 and IPv6. Sockets transition through some of the states very quickly and so you may not see many of the states. There are also maximum counters for TIME_WAIT which show the highest value seen. In addition there are counters for the MaxUserPort and TcpTimedWaitDelay registry keys. I found it useful to have these values clearly visible for situations where machines happened to be configured in an unusual way and nobody had thought to check the registry. These values are set once on program start up and will be zero if the registry key is not set and the operating system's default value is in operation.

We're pleased to be working with VEXIS Systems Inc. again to extend the high performance pluggable server platform that we built for them back in 2008 using The Server Framework. Later, we extended the server to support the hosting of managed plugins with our CLR Hosting Option Pack and some custom development.

This time around we're adding more new functionality to make it easier for them to integrate with external systems.

Version 6.6.1 of The Server Framework was released today.

This release is a minor bug fix release mainly for several bugs in the 6.6 version's OpenSSL code, see the release notes here, for full details of all changes.

All clients using version 6.6 of the OpenSSL Option Pack are advised to upgrade to this release.

• Bug fix to JetByteTools::OpenSSL::CAsyncConnector so that we spot failures in the SSL allocation functions due to low memory situations and throw exceptions if we fail to create the SSL objects.
• Bug fix to CAsyncConnector and CStreamSocketConnectionFilter so that we notify the connector when an outbound connection fails. This allows the connector to correctly release the socket reference that it takes in its constructor.
• Bug fix to CAsyncConnector::Read() to prevent leak of buffers.
• Bug fix to the allocation and creation of JetByteTools::IO::CNonPooledBuffer which was failing to force the correct alignment. See the JETBYTE_IO_IBUFFER_ALIGNMENT_REQUIREMENT changes in version 6.6 for more details
• Bug fix to TLockableObjectTracksLockingThread::IsLockedByThisThread() to correct a casting bug that prevents some configurations from building.
• Bug fix to CTimeChangeNotificationMonitor which fires the timer callback immediately if the delay timer cannot be set and which also prevents the timer callback from being fired continually once it has been fired once (assuming a delay timer is used).
• Bug fix to CDirectoryChangeMonitor() to ensure that multiple shutdown attempts do not access deleted data.

• Added IO::CTimeChangeAwareRotatingAsyncFileLog, which is a rotating async file log which includes a CTimeChangeNotificationMonitor.
• Added IStreamSocket::SetKeepAlive() and IStreamSocket::DisableKeepAlive()
• Added IFilterDatagramSocketConnections::FilterConnectionClosed() and IFilterStreamSocketConnections::FilterConnectionClosed() which are called, when expected, after a call to FilterConnectionClosure().
• Added a conditional overload constructor to Socket::CException which can provide more detail on the socket that generated the exception if socket identifiers are enabled.
• Added an overload to Win32::CException() constructor that takes std::string.
• Added TExpandableBuffer::Swap().
• Added IConfiguration::GetUnsignedShortValue().
• Added IQueueTimers::SetTimerWithRefCountedUserData(), CancelTimerWithRefCountedUserData(), and DestroyTimerWithRefCountedUserData() which all implement the operations using the correct 'AddRef()/Release()' pattern.

• We now use IQueueTimers::SetTimerWithRefCountedUserData() and the other helper functions for reference counted user data where appropriate.
• CStreamSocketBroadcastableConnectionCollection::Broadcast() is now const.

I've just found and fixed a bug in version 6.6 of the OpenSSL Option Pack which relates to how we deal with outbound connection establishment errors. Changes in 6.6 mean that the OpenSSL 'connector' holds a reference to the socket whilst the SSL connection is active. The bug prevents connections which fail to be established from closing and causes a socket leak.

Note that this is ONLY for outbound (client) connections that you establish with The Server Framework. Inbound connections to servers are unaffected.

This fix will be included in release 6.6.1 which will be released later this month. If you need the fix sooner then please get in touch.

We're pleased to be extending the M2M server that we developed for this client back in February last year.

They're adding lots of interesting new functionality. Should be fun!

2014 already. Where does the time go?

We're still very busy with work for our Industrial Control Client but since they needed to get us SC level security clearance for this project if we told you any more we'd have to kill you...

Our work for online gaming clients is diversifying a bit as the project we've been working on for our secretive Online Gaming Company matures into a solid cloud-based SAS gaming server and we start to work more with the recent influx of Korean games companies that are now using The Server Framework.

All of our clients are always great for driving changes into The Server Framework and now that we have the big release from 2013 bedded in we're expecting to put out several smaller releases this year. The focus is still on adding functionality and improving performance so that it's easier than ever to build high performance clients and servers.

And, of course, we're still very interested in getting involved with new client projects and building even more high performance server systems.

We've released new versions of both our Lock Inversion Detector, LID and our Lock Inversion Analyser, LIA today.

This release fixes a problem when running LIA against an executable which has been compiled with Frame Pointer Omission enabled. This causes some call stacks to be reported as zero length and we were failing to treat these correctly which led to a vast number of duplicate lock sequences being created. Also a bug has been fixed in LIA which caused it to fail to use symbol paths supplied on the command line when looking up symbols.

We've also added several new command line switches to LIA to enable you to ignore inversions that occur in specific dlls, limit the number of lock inversions reported, ignore stack frames which are shorter than a configurable length and force slower but more reliable call stack collection to be used.

All customers are being contacted via email with details of how to obtain the latest release of LIA and you can download the latest version of LID from here (if you've already registered for the updates mailing list then simply reply to the email notification and we'll send you the latest version).

Do continue to get in touch with comments and suggestions and any problems that you have.

We have a new client profile available here for a client who uses The Server Framework in their Printer presence and print job transfer system.

They use both the OpenSSL Option Pack and the SSPI Option Pack in their product. The ease of adding security and authentication which can communicate with a disparate selection of clients is one of the strong points of developing with The Server Framework.