A Simple Asynchronous Messaging System

A recent project of mine required some network communication code. Now, in the age of modern .NET Frameworks and Java APIs this is no longer the headache it used to be. However, just because I can get two computers talking to each other doesn't mean that it was done in the best possible way, so I decided to spend a bit of time to create something a bit relevant to the job at hand.

The main requirements I had included that:

1. It had to be fairly abstract and self-contained, since it was to be used by other developers.
2. It had to be type-safe, and be able to pass well defined data only.
   
3. It had to be immediate, and be able to both send and receive at any arbitrary time.
4. It should be resource-friendly, efficient and non-blocking.
5. It should work on a single channel of communication.

"But what about Remoting?", I hear you cry. Well that was a consideration, but it fails on a few of the above. Remote object could have been wrapped up and made self-contained, and they would have certainly been type-safe. However, the technology is pretty inefficient and not really suitable for performance applications or those communicating over the Internet. Further, its event mechanism requires a return channel to operate on - something we needed to avoid due to firewall and NAT issues. And although it wasn't likely we may have required platforms other than .NET to be able to listen or send. So no, Remoting wouldn't do here.

From the requirements, a design began to fall into place:

1. A maximum of two worker classes would be required: a client to initiate a connection and a server to listen for and spawn another (identical) client with which to interact with.
   
2. Data would be passed in well defined messages, and in order.
3. A single overloaded and non-blocking send method would be provided on a client to send messages. Clients will also have events, that fire on receipt of any messages, for consumers to subscribe to.
4. Any sends should be completed on a separate thread. Any polling (for messages and initiations) should also be done on their own dedicated threads. Any events should be fired off on separate threads.

With this design in hand it wasn't too difficult to implement a working solution. I decided to start with a messaging system suggested by Eric Gunnerson[1]. This, by design, was lightweight and efficient, and dealt with type-safety by defining an extendable message class. It also used a simple send method to pass messages and, as a bonus, worked fine over a single channel.

I thought that there was potential to do even better though. There was scope for more efficient threading, and a lot of the internals of the messaging mechanism were exposed to the application in Gunnerson's version. There was also a limited use of events to communicate back messages in a more generic fashion to a variety of consumers; it didn't seem completely asynchronous. Most of the adaptations were straightforward but I'll discuss some of the more complex issues below.

Encoding and Decoding Messages
Message objects need to be serialised into bytes in order to be stuffed down a TCP connection. This is made pretty straightforward by using a BinaryWriter which provides us with an overloaded Write method that converts primitive types (which is all our messages are a collection of) into the bytes required. Deserialisation is just as easy, by using the various BinaryWriter.ReadXXX methods to return us to us the original primitives back again. These can then be used to reconstruct a message object.

The problem however is figuring out what message is coming through in the first place, since on receipt of a bytestream a TCP socket wouldn't be able to tell (does this int belong to a Count message or a ClientID message?), and so wouldn't know in which order to decode or indeed when to stop altogether.

We can use either a prefix or suffix to delimit here and I decided to use the former by sending the type of message before any actual data, since if we know at the start what we're dealing with we'll both know how to decode and when to stop (at which point any further data should belong to another message). It all sounds a bit complicated, but will sense when we talk about receiving messages, below.

Sending Messages
As discussed, sending messages had to be non-blocking. Since sending on TCP can be slow, we need to spawn a thread to do this for us. And since we may be sending messages concurrently, we need to ensure that these threads are synchronised. Assuming we're in a message-sending thread already, the mechanism will look something like this:

Encode bytes to send, prefixing them with the message type
Obtain handle to networkstream
Wait for signal that networkstream is free
Indicate to other threads that networkstream is busy (A)
Send bytes
Indicate to other threads that networkstream is free

The synchronisation primitive I use for this is the AutoResetEvent, which means that I don't have to deal with step (A) above. This should be enough to keep any concurrent sending threads synchronised (and it's worth noting that sending and receiving on a single socket is already supposedly thread safe).

Polling and Receiving
The main strategy used for polling (both for receiving messages and listening for clients) is to block in a continuous loop:

Spawn thread
Begin loop
    Wait on activity (TCP receive or TCP Client Accept)
    Spawn thread to handle activity
End loop

Where "handle activity" usually means to fire an event. Server applications would receive an event notifying them of a new client; the eventargs associated with such an event would hold a reference to the server-side client created, to which you can subscribe for notification of the receipt of various message types. Receiving the messages themselves is a bit more complex, so we'll discuss that it its own section below.

Receiving Messages
From the above, we've already managed to receive something from the socket. Assuming this was a brand spanking new connection, the first thing we should have received is the type of the message to follow. Once we know that, we can then build the relevant message object by decoding in the manner described above. That should leave the stream empty or cued up for the start of any proceeding messages. We then, simply, carry on with the strategy described above and fire the relevant event:

Spawn Thread
Begin loop
    Wait for message type
    Based on above message type, decode rest of stream and    build the message object
    Spawn thread to deal with firing a type-safe message    received event
End loop

And that's all there is to it. This implementation is far from complete and there is more work to be done, including exception handling and being able to gracefully disconnect (possibly by sending a special message type), but the framework is there to build upon. The messaging system we end with is quick, robust and straightforward while managing to be flexible enough to cater for a variety of applications.

Source available on request.

[1] Building a Lightweight Message-Passing System, Eric Gunnerson