Using OpenSSL with Asynchronous Sockets

| 22 Comments

OpenSSL is an open source implementation of the SSL and TLS protocols. Unfortunately it doesn't play well with windows style asynchronous sockets. This article - previously published in Windows Developer Magazine and now available on the Dr. Dobbs site - provides a simple connector that enables you to use OpenSSL asynchronously.

Note: If you need high performance SSL over TCP/IP then this is available as an optional part of The Server Framework; see here for details.

Originally published in Windows Developer Magazine, Copyright 2002.

The Secure Sockets Layer (SSL) is used by every commercial web browser and server to secure web traffic. Every time you use a URL that starts https://, you're using SSL. Microsoft provides an implementation of SSL via the SChannel API, but it is a low-level API that is quite complex to use. OpenSSL is a higher level, open-source alternative that allows you to easily add the security of an SSL connection to TCP/IP clients and servers. However, since the OpenSSL code has a UNIX heritage, most of the examples available assume a UNIX-style sockets architecture. The preferred Windows architecture of asynchronous sockets doesn't fit well with this, and this article presents an "asynchronous connector" for OpenSSL that allows you to use OpenSSL in a way that is more appropriate on the Windows platform. The resulting connector can be used with MFC's CAsyncSocket for client-side use and integrates well with server code that uses IO Completion Ports. Due to the original design of the OpenSSL toolkit, the connector can be built without having to change any of the OpenSSL code and so is standalone and easy to maintain.

OpenSSL

The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, cross-platform, and open-source toolkit implementing the Secure Sockets Layer (SSL v2/v3) and its successor, Transport Layer Security (TLS v1). OpenSSL is based on the SSLeay library developed by Eric A. Young and Tim J. Hudson. The OpenSSL toolkit is licensed under an Apache-style license, which basically means that you are free to get and use it for commercial and noncommercial purposes, subject to some simple license conditions. However, one thing you should be aware of is that OpenSSL uses strong cryptography, and this falls under certain export/import restrictions in some parts of the world. You are strongly advised to pay close attention to any import/export laws or restrictions that apply to you.

The source-code distribution of OpenSSL provides many example programs that show you how to use it. There are some useful tutorials on the Web and books available on the subject (see References). The problem is that although the OpenSSL toolkit source code is cross platform, the example code tends to be written in a UNIX style, using blocking sockets and multiple threads or processes. When developing with sockets on the Win32 platform, it's common to want to use nonblocking, asynchronous sockets: The most scalable method of developing TCP server applications with Win32 is to use IO Completion Ports, which forces you to deal exclusively with asynchronous sockets; and using blocking sockets in a thread that serves a user interface is a recipe for disaster.

Although developed in C, the OpenSSL toolkit has an object-based design, with various components accessed by passing an object handle as the first parameter to the C API calls. The main object that we're interested in is the SSL object. This is where the actual SSL protocol is implemented. The developers of the OpenSSL toolkit designed the SSL object to use an abstract IO mechanism, the BIO, rather than hard coding it to use sockets. The BIO abstraction allows you to develop a BIO that suits your own needs and simply plug it in to the SSL object to run the protocol over whatever data stream you like. The toolkit comes with a BIO that works with a memory buffer, and one of the sample programs uses these memory BIOs to test the toolkit by running the protocol over memory buffers in a single process that is acting as both client and server side of the protocol. We can use this memory buffer BIO to provide an interface to the SSL object that's usable with asynchronous sockets.

The SSL object often needs to read and write to the data stream independently of your application. You may wish to perform a write, but if an SSL handshake is in progress, then the SSL object may need to perform several reads and writes before your application data can be sent. Likewise, data being received from the data stream needs to be processed by the SSL object to decrypt it before passing it up to the application but, due to the nature of the protocol, the arrival of data may require the SSL object to write to the data stream.

A Simple C++ Wrapper

Given the aforementioned information, we can begin to develop a C++ class that provides a clean interface to the OpenSSL memory BIOs that we will use to communicate with the SSL object. The object will act as a filter between the application and the socket. The interface to the class should provide read and write methods for pushing data into the SSL object, and be able to call back into the application when decrypted application data is available and when encrypted writes to the data stream are required. Since we're intending to operate in an asynchronous manner, and since the SSL object may not always be ready to process our application data immediately, we will need to be able to buffer data prior to passing it through our filter object. The buffer management will vary depending on how you want to use the filter, so rather than including it within the main filter class, we simply provide hooks so that derived classes can implement their own buffering scheme. Removing the uffer management code also allows us to see the actual code required to drive the SSL object more clearly.

COpenSSLConnectorBase provides the derived class with methods to push data into the SSL object. Unencrypted application data is pushed through the SSL object by calling DataToWrite() and will eventually emerge, ready to be written to the data stream, via the OnDataToWrite() virtual function. Encrypted data that arrives from the data stream is pushed into the SSL object by calling DataToRead(), and application data will emerge via the OnDataToRead() virtual function. Since an application write may require the SSL object to process responses read from the peer, the derived class should not call DataToWrite() and DataToRead() directly. Instead, when it has data that needs to be passed through the SSL object, it should buffer the data and then call RunSSL(). This will deal with any SSL-level data transmission and request read and write data from the derived class using the GetPendingOperations(), PerformRead(), and PerformWrite() virtual functions. GetPendingOperations() simply returns two boolean values that are set to True if the derived class has read or write data buffered and ready to pass to the SSL object. When the read or write data is required, PerformRead() or PerformWrite() will be called. These should call DataToRead() or DataToWrite() as appropriate to pass the buffered data to the SSL object. These calls take a pointer to a BYTE buffer and the length of that buffer; they return the number of bytes consumed. The caller should then adjust the buffered data to take into account the bytes that have been consumed and returned.

Since both read and write operations can result in the SSL object generating data that must be transmitted to the peer, the actual transmission is done as part of the RunSSL() method. If, after reading or writing, there is data to be transmitted, SendPendingData() is called and this extracts the data from the SSL's output BIO and calls OnDataToWrite() to pass it to the derived class for writing to the data stream.

CAsyncConnector is a class derived from COpenSSLConnectorBase that provides an example of simple buffering and integrates the SSL object with the MFC CAsyncSocket class.

The AsyncClient sample code puts all of this together in a simple adaptation of the Microsoft sample code detailed in Microsoft Knowledge Base, Article Q192570. The client presents a dialog interface that allows you to enter a server name to connect to and a URL to retrieve. The application connects to the specified server on port 443, the port commonly used for HTTPS servers, and sends an HTTP Get request for the specified URL. The results are displayed in an edit box. By stepping through the code and watching the debug traces, you will see the SSL connection established and the encrypted request sent. The results arrive asynchronously and are pushed through the SSL protocol to decrypt them before displaying them in the edit box.

Obviously, it would be easier to use the WinInet functions if you actually wanted to write a simple HTTPS client. The example does not attempt to validate the server's credentials or force the client to require a certificate or dictate which cipher suite to use; all of this is possible and you should consult the OpenSSL samples for details of how to use its more advanced features. The example simply demonstrates the ease at which OpenSSL can be integrated with CAsyncSocket. Using these techniques with the OpenSSL toolkit, you can easily protect any data stream with SSL.

A Potential For Optimization

The OpenSSL toolkit BIO interface includes a method of accessing the internal BIO data buffer directly, rather than passing in a new buffer for the data to be copied into or out of. This interface is exposed via the BIO_nread/nwrite methods. Unfortunately, only the BIO_pair BIO implements these methods. A BIO_pair could be used in place of our two memory BIOs and would allow us to extract data directly from the BIO in DataToWrite() and SendPendingData(); however, doing so complicates the example and so is left as an exercise for the reader.

Building the Sample Source Code

To build the sample source code you need to have OpenSSL installed; see the OpenSSL web site for details of how to obtain and install the toolkit. Once you have installed OpenSSL, you need to adjust the sample project so that it can find the toolkit headers and libraries. In the project settings dialog, on the C/C++ pane, in the Preprocessor category, change the additional include directory from "I:\openssl-0.9.6d\ inc32" to the include directory that is appropriate for your installation of OpenSSL. Likewise on the Link pane, in the input category, change the additional library path from "I:\openssl-0.9.6d\out32dll" to something more appropriate.

References

Download

The following source was built using Visual Studio 6.0 SP5. See above for other requirements.

SSLAsyncClient.zip - the sample code

Revision history

Note: If you need high performance SSL over TCP/IP then this is available as an optional part of The Server Framework; see here for details.

22 Comments

Hi, I am using your wrapper, and I am a little confused when you use SSL_Read. I did not understand how the SSL_Read can understand that it has to read data from the peer to which the CConnectSoc is connected?
Thanks,
David

When the socket has data available to read the OnReceive() method is called by the CAsyncSocket base class. That reads the data and passes it to the connector via ReadCompleted(). ReadCompleted() adds the data to a list and runs the SSL engine which eventually ends up in a call to DataToRead(). At this point we push the data that arrived over the socket into the BIO using BIO_write() and we then pull it through the OpenSSL engine with SSL_Read() this may or may not return some unencrypted data and may or may not generate data that needs to be sent to the other side...

Hi, I wonder where your wrapper handshakes the server? I am a little confused!

thanks,
Burns

It happens automatically as data flows to the server. Watch the output from the calls to OutputDebugString() in COpenSSLConnectorBase::RunSSL().

Thanks for your rapid reply!

But I don't want automatically to shakehand the server,do you have any suggestion?

Burns

Perhaps if you were to describe what you want to do it might help. Or describe how you would do it using synchronous calls...

I always learn more from your code!

I want to integrate openssl with I/O completion port! Some connections connect the server via ssl and some not! So automating ssl-handshaking may be not suitable for this!

thanks

Burns

I can provide you with a version of my framework that provides integration of this kind if you would like. Drop me an email if you'd like a quote for the work. You can take a look at the compiled version of a sample SSL enabled server here http://www.lenholgate.com/archives/000274.html.

If you want to do the work yourself then the trick is to just not bother passing the data through the SSL connector if the connection isn't a secure one...

Thanks!

burns7975@gmail.com

why do you use mutexes for locking and not critical sections?

The usual reason to use a mutex over a critical section is if you want to have multiple processes use the 'shared area'.

Yes, i know. But does OpenSSL share data between multiple processes? I don't think so...

Good point, and if you had been a bit more direct in your original comment I'd have realised what you were getting at.

As far as I know, that code came from some of the OpenSSL samples. It has been fixed in later releases of the code.

Hi,
Do you have some code that in Pure C++ (Not MFC) that does the SSL communication with certificates? Any help will be great.

I dont seem to find the code in openssl.org or Google :).

Thanks,
Saleem

Hi,
Also could you please share with me the code of the exes' in the attached zip file.

Thanks,
Saleem

The code on http://www.lenholgate.com/archives/000274.html is available if you license my server framework; it's not available for free, sorry. Drop me an email if you're interested in doing that.

Hello. I know that you use a buffer pool with IO operations. Did you addressed the massive use of malloc and free done by OpenSSL? My tip (if you not did it already) is the CRYPTO_set_mem_functions API to redirect malloc/free to another heap or even a buffer pool. Another tip: CRYPTO_set_add_lock_callback to force OpenSSL use InterlockedIncrement and InterlockedDecrement instead use a lock to atomically increment and decrement variables.

Congratulations for you great library.

Leandro,

I wasn't aware of those optimisations. Thanks. So far I haven't had any complaints from users of this or the licensed version of this code; but it's always useful to hear about potential performance improvements.

I am using your OpenSSL class to connect to a Pop3 server. Typically the server sends a +OK ready response on the initial connection. However, the application does not receive this response until after sending the user command. Is it possible to read from the SSL socket after connecting to the server?

Yes, though it sounds like the code here doesn't let you. The trick is to always force a zero byte write to kick off the SSL handshake. I'll try and have a look at the code later and add a comment which shows how this can be done (if it's not obvious).

I've looked at the code whilst connecting to pop.gmail.com and you have to "kick start" the SSL handshake from the client side by sending 0 bytes.

A quick hack with the current code would be to add a call to m_pConnector->Write(0, 0); to the end of CConnectSoc::OnConnect().

This then requires the following change to PerformWrite() (compare to the code you have).

void CAsyncConnector::PerformWrite()
{
OutputDebugString("PerformWrite\n");

Buffer *pBuffer = GetNextBuffer(m_pendingWriteList);

if (pBuffer)
{
size_t bufferUsed = DataToWrite(pBuffer->data, pBuffer->used);

// may not be able to push all data into the BIO in one go

if (!pBuffer->used)
{
OutputDebugString("Kicked handshake\n");

delete pBuffer;
}
else
{
UseData(m_pendingWriteList, pBuffer, bufferUsed);
}
}
}

And now it 'just works'. :)

Thanks Len.

Leave a comment

About this Entry

Sinking connection points in C++ objects. was the previous entry in this blog.

Welcome is the next entry in this blog.

I usually write about C++ development on Windows platforms, but I often ramble on about other less technical stuff...

Find recent content on the main index or look in the archives to find all content.

I have other blogs...

Subscribe to feed The Server Framework - high performance server development
Subscribe to feed Lock Explorer - deadlock detection and multi-threaded performance tools
Subscribe to feed l'Hexapod - embedded electronics and robotics
Subscribe to feed MegèveSki - skiing