Out of band data, TCP Urgent mode and overlapped I/O

General, New features
Page content

Some stream protocols have the concept of ‘out of band’ (OOB) data. This is a separate logical communication channel between the peers which enables data that is unrelated to the current data in the stream to be sent alongside the normal data stream. This is often a way for some data to jump ahead of the normal stream and arrive faster than if it were delivered via the the normal data stream.

Winsock supports out of band data in a protocol independent way, see here, but accessing it from networking code that uses overlapped I/O rather than the old-fashioned BSD API is somewhat under documented. By default, out of band data does not appear in the normal data stream, you have to read it explicitly by setting MSG_OOB in the flags of a call to WSARecv(). For non overlapped I/O designs you can use WSAAsyncSelect() or select() to explicitly check for the presence of out of band data. With overlapped I/O your options appear limited, it seems that you should be able to use an overlapped WSAIoctl() call with SIOCATMARK but this will return immediately when OOB is either present or not present, it doesn’t wait for OOB to become available.

The solution is to post a separate, out of band, overlapped WSARecv() passing the MSG_OOB flag. This will only return on socket closure or when out of band data arrives. By using a distinct indicator in your ‘per operation data’ you can distinguish this read from normal reads and deal with it accordingly. Once you have processed the special out of band data read you can then post another to read subsequent out of band data.

Inline Out Of Band Data

Winsock also allows you to include any OOB in the normal data stream, you can do this by setting the SO_OOBINLINE socket option on the connection. When SO_OOBINLINE is set the OOB data is included in the results of normal WSARecv() calls without needing to set MSG_OOB in the flags, however, with overlapped I/O it’s then impossible to determine if OOB exists and if it’s present in the data buffer returned by any particular WASRecv() call. When SO_OOBINLINE is set calls to WSAIoctl() with SIOCATMARK always return true to indicate that no OOB is waiting. For most Winsock Providers this is probably fine. You most probably want to read your OOB separately for X.25 or whatever and the last thing you want is for it to be mixed in with the normal data from the data stream.

For TCP things are somewhat more complex. TCP doesn’t actually have the concept of OOB, but the BSD API maps the TCP Urgent data concept onto OOB. Due to differences in the TCP RFCs and historical issues there’s complexity in using TCP Urgent data reliably if you don’t know which RFC the implementation adheres to. This complexity makes it almost impossible to use OOB with overlapped TCP in a reliable manner, especially if you have no control over the platform upon which your clients run.

TCP Urgent Data

In TCP out of band data is implemented in terms of ‘urgent data’ using the URG bit and the Urgent Pointer, see here, however, there are two conflicting descriptions of how this works, RFC 793 which details TCP says that the Urgent Pointer indicates the byte that follows the urgent data but RFC 1122 corrects this and states that the Urgent Pointer indicates the final byte of urgent data. This leads to interoperability issues if one peer uses the RFC 793 definition and the other uses the RFC 1122 definition. The Windows documentation for the standard TCP Winsock Provider claims that it operates as BSD does however this can be changed using the TCP_EXPEDITED_1122 socket option if the Winsock provider supports it. There’s more compatibility complexity in that Windows only supports a single byte of out of band data whereas RFC 1122 specifies that TCP MUST support sequences of urgent data bytes of any length. Windows also doesn’t specify how or if it will buffer subsequent out of band data, so if you are slow in reading a byte of urgent data and another byte of urgent data arrives then one of the bytes may be lost; though our tests have shown that Windows does buffer urgent data. This all makes the use of out of band signalling using urgent data somewhat unreliable on Windows with TCP.

The unreliability shows itself if you’re using separate OOB reads using MSG_OOB. In this instance you may get the ‘wrong’ byte of OOB data if you are dealing with a peer that implements the altenative RFC. This wouldn’t be a problem if the byte hadn’t been removed from the normal data stream and so you end up with an incorrect byte in the normal data stream (the intended OOB byte) and a missing byte, which has been treated as OOB data. This doesn’t cause problems for the two traditional users of TCP Urgent Data, Telnet and Rlogin as they simply use the Urgent data notification to alert the server that urgent data is present in the data stream and then read and discard all of the normal data in the stream until they can read the urgent data. See here for how this facility is used in the Telnet protocol’s synch function. It’s practically impossible to implement this functionality with overlapped I/O as you don’t want the urgent data removed from the data stream, so you need to operate in SO_OOBINLINE mode and yet when in that mode you will never be notified that urgent data exists. To implement the telnet model with overlapped I/O you pretty much need to always read all inbound data and buffer it in your server rather than allowing it to buffer in the TCP stack and allowing TCP flow control to prevent the client from sending more until you’re ready.

So, if you want to use out of band data with TCP with overlapped I/O you need to remember five things:

  • Out of band data in TCP on Windows when interoperating with other, non-Windows, operating systems is likely to be unreliable due to differences between RFC 793 and RFC 1122.
  • Expecting to send more than a single byte of out of bound data is likely not to work.
  • Your out of band data may get “lost” if you send out of band data faster than the receiver is processing it.
  • To read out of band data using overlapped I/O you need to post a special WSARecv() with MSG_OOB set in the flags.
  • Real out of band communication using TCP is better achieved with a separate ‘control’ connection rather than using TCP’s Urgent data function via MSG_OOB.

Despite all of this, The Server Framework will support out of band data in the next major release. By default OOB data will be disabled and we’ll abort connections that use it. You can also decide to enable OOB inline which effectively disables OOB data from the receiver’s point of view and simply keeps the out of band data in the main data stream. Or, you can enable async OOB which will read OOB data using an optional special purpose overlapped read with MSG_OOB set.