If you ever decide to issue more than one read or write operation on a single socket before the previous read or write operation has completed then you should probably be thinking about sequencing your TCP stream. Sequencing the stream enables you to ensure that your reads and writes are processed in the correct order. Unless you are working with a protocol where each write transmits a stand-alone data packet which can be processed completely independently of all others then sequencing is likely to be an issue for you.
Write sequencing
Due to the way the
system is designed most of the read and write requests that an application issues pass through an
I/O completion port before being executed. Whenever there is more than a single thread servicing an I/O completion port the completion packets that emerge from the port could be processed in any order (although they emerge from the completion port in a FIFO order the vaguaries of time slicing mean that a completion packet that emerges 'second' could be processed 'first' if the thread that is processing the 'second' packet is scheduled to run more than the thread that is processing the 'first'. This means that if you have some code that issues serveral write requests in sequence, without stream sequencing you would not know which order the actual writes would be performed.
could result in 123 being written to the wire, or 231 or 213 or 321, etc...
Thankfully it's easy to fix this issue. If you use an instance of
IAllocateSequencedStreamSockets to allocate your sockets then all writes will be sequenced automatically for you and the code above is guarenteed to always write 123 to the wire.
Key points for write sequencing
Read sequencing
If you ever have more than a single read outstanding on a single socket then read sequencing is likely to be an issue for you. Although the reads will emerge from the
completion port in the correct FIFO order they are subject to the same problem as detailed above; thread scheduling may cause a later read completion to be handled before an earlier one. Again the solution is to use a socket allocator that derives from
IAllocateSequencedStreamSockets. However, all this does is ensure that all of the data buffers that are passed to
OnReadCompleted() have correct and sequential sequence number in them, it doesn't, in itself, ensure that those buffers are actually processed in the correct order.
The simplest way to ensure that your read completions are processed in the correct order is to push an instance of
CReadSequencingStreamSocketConnectionFilter onto your connection manager's
filter chain. This simple filter will filter each read completion and pass them on to your code in the correct order. Only one completion will occur at a time and the buffers that arrive will be in the correct sequence.
Using the
CReadSequencingStreamSocketConnectionFilter is as easy as this:
CSocketServer::CSocketServer(
const IFullAddress &address,
const ListenBacklog listenBacklog,
IIOPool &ioPool,
IAllocateStreamSockets &socketAllocator,
IAllocateBuffers &bufferAllocator,
ILimitConnections &connectionLimiter)
: CFilteringStreamSocketServer(address, listenBacklog, *this, ioPool, socketAllocator, bufferAllocator, NoZeroByteRead, connectionLimiter),
m_readSequenceFilter(socketAllocator)
{
AddConnectionFilter(m_readSequenceFilter);
}
You can probably get a fraction more performance out of your read completions if you roll your own sequencing code rather than using the filter. See the filter code for details of what you'd need to put in your own
OnReadCompleted() and place your processing logic directly where the filter calls back into the filter manager to pass the completion on.
Key points for read sequencing
-
Always use a socket allocator derived from
IAllocateSequencedStreamSockets if you could ever have more than one read outstanding on a socket at one time.
-
You are still responsible for processing the read completions in the correct order, but using the
CReadSequencingStreamSocketConnectionFilter makes this easy.
-
If you profile your code and decide that you're losing too much performance by using the sequencing filter then you can use it as an example of how to write the required sequencing code directly into your completion handler.
More information
