Assume makes an ass out of u and me

| 9 Comments

But mostly me. ;)

During yesterday's investigations into handling lots (30,000+) of socket connections to a server built with The Server Framework I took a few things for granted. I should have been a bit more thorough rather than just assuming I knew what I was doing.

Today I did some more tests.

I was a little surprised that flicking the switch on my server framework so that it posted zero byte reads had no affect on the limit of connections that I could support. I've never actually needed the zero byte read functionality but a user requested it, so I added it and that was that. I realised that I now wasn't 100% sure of what it was supposed to fix, just that it helps when you have lots of connections that don't send data that often. I looked it up in Network Programming for Microsoft Windows and the reason for using zero byte reads is that there is a fixed amount of memory that Windows will allow to be locked at any one time. This limit, the number of locked pages, is separate to the non-paged pool limit. I was getting the two confused. Issuing zero byte reads doesn't help reduce non-paged pool usage. It works to prevent you exceeding the I/O page lock limit. The way it works is that when you issue an overlapped read or write request it is probable that the data buffers involved will be locked. The locking granularity is the page size of the system, and since there's a limit on the number of pages that you can have locked at any one time if you have lots of connections and each has a large number of pages locked due to pending read requests then you can run our of resources and be unable to issue any more reads or writes. The zero byte read "trick" works around this by issuing reads with a zero length buffer. This means that a read is pending but there are no locked memory pages. This allows you to have many connections with pending reads and no page locks. When data arrives on a connection you get a read completion and you can either issue a synchronous read to read the data into a real buffer or you can do what I do which is post another overlapped read request with a real buffer. If you have lots of connections and each connection only receives data infrequently then this technique can help you avoid the I/O page lock limit. Overlapped operations return WSAENOBUFS when the limit is reached.

My problems at ~32,000 connections yesterday were not due to the I/O page lock limit so switching on zero byte reads had no effect.
To prove to myself that the zero byte read code did actually do something I increased the size of the buffers used for each IO operation and ran my test again. The I/O page lock limitation could be clearly seen. The server continued to accept connections but the reads that it was issuing on the connections were failing with WSAENOBUFS. This is another issue to address in the framework as it would be useful for the server to deal with this situation a little better than it currently does, but it's not a real problem for any of my clients right now so it has gone on the list of things to do. It's also, obviously, not the factor that's limiting the number of connections that the server can support.

The next potential limit is the one that I was talking about yesterday; the non-paged pool limit. The problem with me assuming that my server was hitting this limit was that I never bothered to look too closely at how much non-paged pool was actually being consumed. Like duh! Today I did that and I was surprised to see that the server wasn't anywhere near to the non-paged pool limit. A standard Windows XP box with 1GB of ram has 256MB of non-paged pool. The figures used in Network Programming for Microsoft Windows suggest that a connected socket uses around 2KB of non-paged pool, an accepted socket uses 1.5KB (less due to the fact that it only needs to store the remote address) and each pending IO operation uses around 500 bytes. This morning, whilst looking for ways to adjust the non-paged pool size (if I reduced it I should get less connections) I discovered an article on that mentioned two registry keys that affect non-paged pool usage. Both values like under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

The first, NonPagedPoolSize, lets you change the size of the non-paged pool and the second, NonPagedPoolQuota, relates to a per process quota that the system imposes on each process. Not wanting to play with these values just yet I decided to test my theory that the limitation wasn't related to non-paged pool use by having each connection post several overlapped read requests when the connection was established, rather than just one. This resulted in the server really running out of non-paged pool (possibly hitting the process quota) after using around 75MB of non-paged pool (based on task manager's display of the process) with a total of 208MB of non-paged pool consumed in total (I assume the rest is in the networking subsystem or a rogue LSP)...

So, that's where I am at the end of today. I understand the potential problems more and yet I understand the actual problem less. Next on the list are running the tests on one of my server boxes that doesn't have any wonderful LSPs installed, running in a clean VMWare machine and installing some more memory to see if the problem changes when I have 2GB rather than just the one...

9 Comments

From
http://technet2.microsoft.com/WindowsServer/en/Library/ef2a2ba7-f434-4c6c-abaf-0a8c766c862b1033.mspx

For NonPagedPoolQuota
Range: 1–128 (megabytes)

So an app can have a max of 128 MB of NP Pool, no matter how much physical RAM you have? That allows for a theoretical max of 64000 connections assuming that no overalapped requests are pending on them. I find myself posting 4KB worth of read + 4 KB worth of write on a single connection, thats 8 + 2 = 10 KB of NP Pool per connection. Going by that, the max I'll be able to handle is 12800 concurrent connections!

A far cry from the 50,000 or so mentioned in James and Ohlund. Perhaps I should look into the strategy of posting 0 byte WSARecvs.

I dont think that can be correct. Or if it IS correct then I can only think that when set to 0 the system will allow more that 128.

I've had well over 64000 active connections.

Could you comment on the amount of NP Pool used by your process at 60000 active connections or more?

Thanks!

I could if I can get the time to set up the appropriate virtual machine and run the appropriate test clients against it. I don't think that's likely in the next few weeks as I'm pretty busy.

If you do get around to it / remember figures at other multiples of 10K connections, please post!

Thanks

Hey

I wonder if you could share the source (or its already online) of that client, you use for testing servers, how many connections/traffic they can handle?

Thanks a lot
Regards

The source is available if you license the latest version of the server framework. Contact me via email for details if you'd like to do that.

What do you think about using VirtualAlloc to allocate page-aligned memory for the receive buffers? Would it reduce the number of locked pages? I mean if you allocate a buffer of 'page size' using 'new[]' it is very likely that the buffer uses 2 pages instead of 1.

It certainly has potential. Probably pretty easy to test with a replacement buffer allocator. The buffer itself would need to be page size - sizeof class but that probably wouldnt be a problem....

Leave a comment