Skip to main content

RFC 6347 - 4.2. The DTLS Handshake Protocol

4.2. The DTLS Handshake Protocol

DTLS uses all of the same handshake messages and flows as TLS, with three principal changes:

  1. A stateless cookie exchange has been added to prevent denial-of-service attacks.

  2. Modifications to the handshake header to handle message loss, reordering, and DTLS message fragmentation (in order to avoid IP fragmentation).

  3. Retransmission timers to handle message loss.

With these exceptions, the DTLS message formats, flows, and logic are the same as those of TLS 1.2.

4.2.1. Denial-of-Service Countermeasures

Datagram security protocols are extremely susceptible to a variety of DoS attacks. Two attacks are of particular concern:

  1. An attacker can consume excessive resources on the server by transmitting a series of handshake initiation requests, causing the server to allocate state and potentially to perform expensive cryptographic operations.

  2. An attacker can use the server as an amplifier by sending connection initiation messages with a forged source of the victim. The server then sends its next message (in DTLS, a Certificate message, which can be quite large) to the victim machine, thus flooding it.

In order to counter both of these attacks, DTLS borrows the stateless cookie technique used by Photuris [PHOTURIS] and IKE [IKEv2]. When the client sends its ClientHello message to the server, the server MAY respond with a HelloVerifyRequest message. This message contains a stateless cookie generated using the technique of [PHOTURIS]. The client MUST retransmit the ClientHello with the cookie added. The server then verifies the cookie and proceeds with the handshake only if it is valid. This mechanism forces the attacker/client to be able to receive the cookie, which makes DoS attacks with spoofed IP addresses difficult. This mechanism does not provide any defense against DoS attacks mounted from valid IP addresses.

The exchange is shown below:

     Client                                   Server
------ ------
ClientHello ------>

<----- HelloVerifyRequest
(contains cookie)

ClientHello ------>
(with cookie)

[Rest of handshake]

DTLS therefore modifies the ClientHello message to add the cookie value.

struct {
ProtocolVersion client_version;
Random random;
SessionID session_id;
opaque cookie<0..2^8-1>; // New field
CipherSuite cipher_suites<2..2^16-1>;
CompressionMethod compression_methods<1..2^8-1>;
} ClientHello;

When sending the first ClientHello, the client does not have a cookie yet; in this case, the Cookie field is left empty (zero length).

The definition of HelloVerifyRequest is as follows:

struct {
ProtocolVersion server_version;
opaque cookie<0..2^8-1>;
} HelloVerifyRequest;

The HelloVerifyRequest message type is hello_verify_request(3).

The server_version field has the same syntax as in TLS. However, in order to avoid the requirement to do version negotiation in the initial handshake, DTLS 1.2 server implementations SHOULD use DTLS version 1.0 regardless of the version of TLS that is expected to be negotiated. DTLS 1.2 and 1.0 clients MUST use the version solely to indicate packet formatting (which is the same in both DTLS 1.2 and 1.0) and not as part of version negotiation. In particular, DTLS 1.2 clients MUST NOT assume that because the server uses version 1.0 in the HelloVerifyRequest that the server is not DTLS 1.2 or that it will eventually negotiate DTLS 1.0 rather than DTLS 1.2.

When responding to a HelloVerifyRequest, the client MUST use the same parameter values (version, random, session_id, cipher_suites, compression_method) as it did in the original ClientHello. The server SHOULD use those values to generate its cookie and verify that they are correct upon cookie receipt. The server MUST use the same version number in the HelloVerifyRequest that it would use when sending a ServerHello. Upon receipt of the ServerHello, the client MUST verify that the server version values match. In order to avoid sequence number duplication in case of multiple HelloVerifyRequests, the server MUST use the record sequence number in the ClientHello as the record sequence number in the HelloVerifyRequest.

Note: This specification increases the cookie size limit to 255 bytes for greater future flexibility. The limit remains 32 for previous versions of DTLS.

The DTLS server SHOULD generate cookies in such a way that they can be verified without retaining any per-client state on the server. One technique is to have a randomly generated secret and generate cookies as:

Cookie = HMAC(Secret, Client-IP, Client-Parameters)

When the second ClientHello is received, the server can verify that the Cookie is valid and that the client can receive packets at the given IP address. In order to avoid sequence number duplication in case of multiple cookie exchanges, the server MUST use the record sequence number in the ClientHello as the record sequence number in its initial ServerHello. Subsequent ServerHellos will only be sent after the server has created state and MUST increment normally.

One potential attack on this scheme is for the attacker to collect a number of cookies from different addresses and then reuse them to attack the server. The server can defend against this attack by changing the Secret value frequently, thus invalidating those cookies. If the server wishes that legitimate clients be able to handshake through the transition (e.g., they received a cookie with Secret 1 and then sent the second ClientHello after the server has changed to Secret 2), the server can have a limited window during which it accepts both secrets. [IKEv2] suggests adding a version number to cookies to detect this case. An alternative approach is simply to try verifying with both secrets.

DTLS servers SHOULD perform a cookie exchange whenever a new handshake is being performed. If the server is being operated in an environment where amplification is not a problem, the server MAY be configured not to perform a cookie exchange. The default SHOULD be that the exchange is performed, however. In addition, the server MAY choose not to do a cookie exchange when a session is resumed. Clients MUST be prepared to do a cookie exchange with every handshake.

If HelloVerifyRequest is used, the initial ClientHello and HelloVerifyRequest are not included in the calculation of the handshake_messages (for the CertificateVerify message) and verify_data (for the Finished message).

If a server receives a ClientHello with an invalid cookie, it SHOULD treat it the same as a ClientHello with no cookie. This avoids race/deadlock conditions if the client somehow gets a bad cookie (e.g., because the server changes its cookie signing key).

Note to implementors: This may result in clients receiving multiple HelloVerifyRequest messages with different cookies. Clients SHOULD handle this by sending a new ClientHello with a cookie in response to the new HelloVerifyRequest.

4.2.2. Handshake Message Format

In order to support message loss, reordering, and message fragmentation, DTLS modifies the TLS 1.2 handshake header:

struct {
HandshakeType msg_type;
uint24 length;
uint16 message_seq; // New field
uint24 fragment_offset; // New field
uint24 fragment_length; // New field
select (HandshakeType) {
case hello_request: HelloRequest;
case client_hello: ClientHello;
case hello_verify_request: HelloVerifyRequest; // New type
case server_hello: ServerHello;
case certificate:Certificate;
case server_key_exchange: ServerKeyExchange;
case certificate_request: CertificateRequest;
case server_hello_done:ServerHelloDone;
case certificate_verify: CertificateVerify;
case client_key_exchange: ClientKeyExchange;
case finished: Finished;
} body;
} Handshake;

The first message each side transmits in each handshake always has message_seq = 0. Whenever each new message is generated, the message_seq value is incremented by one. Note that in the case of a rehandshake, this implies that the HelloRequest will have message_seq = 0 and the ServerHello will have message_seq = 1. When a message is retransmitted, the same message_seq value is used. For example:

      Client                             Server
------ ------
ClientHello (seq=0) ------>

X<-- HelloVerifyRequest (seq=0)
(lost)

[Timer Expires]

ClientHello (seq=0) ------>
(retransmit)

<------ HelloVerifyRequest (seq=0)

ClientHello (seq=1) ------>
(with cookie)

<------ ServerHello (seq=1)
<------ Certificate (seq=2)
<------ ServerHelloDone (seq=3)

[Rest of handshake]

Note, however, that from the perspective of the DTLS record layer, the retransmission is a new record. This record will have a new DTLSPlaintext.sequence_number value.

DTLS implementations maintain (at least notionally) a next_receive_seq counter. This counter is initially set to zero. When a message is received, if its sequence number matches next_receive_seq, next_receive_seq is incremented and the message is processed. If the sequence number is less than next_receive_seq, the message MUST be discarded. If the sequence number is greater than next_receive_seq, the implementation SHOULD queue the message but MAY discard it. (This is a simple space/bandwidth tradeoff).

4.2.3. Handshake Message Fragmentation and Reassembly

As noted in Section 4.1.1, each DTLS message MUST fit within a single transport layer datagram. However, handshake messages are potentially bigger than the maximum record size. Therefore, DTLS provides a mechanism for fragmenting a handshake message over a number of records, each of which can be transmitted separately, thus avoiding IP fragmentation.

When transmitting the handshake message, the sender divides the message into a series of N contiguous data ranges. These ranges MUST NOT be larger than the maximum handshake fragment size and MUST jointly contain the entire handshake message. The ranges SHOULD NOT overlap. The sender then creates N handshake messages, all with the same message_seq value as the original handshake message. Each new message is labeled with the fragment_offset (the number of bytes contained in previous fragments) and the fragment_length (the length of this fragment). The length field in all messages is the same as the length field of the original message. An unfragmented message is a degenerate case with fragment_offset=0 and fragment_length=length.

When a DTLS implementation receives a handshake message fragment, it MUST buffer it until it has the entire handshake message. DTLS implementations MUST be able to handle overlapping fragment ranges. This allows senders to retransmit handshake messages with smaller fragment sizes if the PMTU estimate changes.

Note that as with TLS, multiple handshake messages may be placed in the same DTLS record, provided that there is room and that they are part of the same flight. Thus, there are two acceptable ways to pack two DTLS messages into the same datagram: in the same record or in separate records.

4.2.4. Timeout and Retransmission

DTLS messages are grouped into a series of message flights, according to the diagrams below. Although each flight of messages may consist of a number of messages, they should be viewed as monolithic for the purpose of timeout and retransmission.

Client                                          Server
------ ------

ClientHello --------> Flight 1

<------- HelloVerifyRequest Flight 2

ClientHello --------> Flight 3

ServerHello \
Certificate* \
ServerKeyExchange* Flight 4
CertificateRequest* /
<-------- ServerHelloDone /

Certificate* \
ClientKeyExchange \
CertificateVerify* Flight 5
[ChangeCipherSpec] /
Finished --------> /

[ChangeCipherSpec] \ Flight 6
<-------- Finished /

Figure 1. Message Flights for Full Handshake
Client                                           Server
------ ------

ClientHello --------> Flight 1

ServerHello \
[ChangeCipherSpec] Flight 2
<-------- Finished /

[ChangeCipherSpec] \Flight 3
Finished --------> /

Figure 2. Message Flights for Session-Resuming Handshake
(No Cookie Exchange)

DTLS uses a simple timeout and retransmission scheme with the following state machine. Because DTLS clients send the first message (ClientHello), they start in the PREPARING state. DTLS servers start in the WAITING state, but with empty buffers and no retransmit timer.

                     +-----------+
| PREPARING |
+---> | | <--------------------+
| | | |
| +-----------+ |
| | |
| | Buffer next flight |
| | |
| \|/ |
| +-----------+ |
| | | |
| | SENDING |<------------------+ |
| | | | | Send
Receive | | | | HelloRequest
next | | Send flight | |
flight | +--------+ | | or
| | | Set retransmit timer | | Receive
| | \|/ | | HelloRequest
| | +-----------+ | | Send
| | | | | | ClientHello
+--)--| WAITING |-------------------+ |
| | | | Timer expires | |
| | +-----------+ | |
| | | | |
| | | | |
| | +------------------------+ |
| | Read retransmit |
Receive | | |
last | | |
flight | | |
| | |
\|/\|/ |
|
+-----------+ |
| | |
| FINISHED | -------------------------------+
| |
+-----------+
| /|\
| |
| |
+---+

Read retransmit
Retransmit last flight

Figure 3. DTLS Timeout and Retransmission State Machine

The state machine has three basic states.

In the PREPARING state, the implementation does whatever computations are necessary to prepare the next flight of messages. It then buffers them up for transmission (emptying the buffer first) and enters the SENDING state.

In the SENDING state, the implementation transmits the buffered flight of messages. Once the messages have been sent, the implementation then enters the FINISHED state if this is the last flight in the handshake. Or, if the implementation expects to receive more messages, it sets a retransmit timer and then enters the WAITING state.

There are three ways to exit the WAITING state:

  1. The retransmit timer expires: the implementation transitions to the SENDING state, where it retransmits the flight, resets the retransmit timer, and returns to the WAITING state.

  2. The implementation reads a retransmitted flight from the peer: the implementation transitions to the SENDING state, where it retransmits the flight, resets the retransmit timer, and returns to the WAITING state. The rationale here is that the receipt of a duplicate message is the likely result of timer expiry on the peer and therefore suggests that part of one's previous flight was lost.

  3. The implementation receives the next flight of messages: if this is the final flight of messages, the implementation transitions to FINISHED. If the implementation needs to send a new flight, it transitions to the PREPARING state. Partial reads (whether partial messages or only some of the messages in the flight) do not cause state transitions or timer resets.

Because DTLS clients send the first message (ClientHello), they start in the PREPARING state. DTLS servers start in the WAITING state, but with empty buffers and no retransmit timer.

When the server desires a rehandshake, it transitions from the FINISHED state to the PREPARING state to transmit the HelloRequest. When the client receives a HelloRequest, it transitions from FINISHED to PREPARING to transmit the ClientHello.

In addition, for at least twice the default MSL defined for [TCP], when in the FINISHED state, the node that transmits the last flight (the server in an ordinary handshake or the client in a resumed handshake) MUST respond to a retransmit of the peer's last flight with a retransmit of the last flight. This avoids deadlock conditions if the last flight gets lost. This requirement applies to DTLS 1.0 as well, and though not explicit in [DTLS1], it was always required for the state machine to function correctly. To see why this is necessary, consider what happens in an ordinary handshake if the server's Finished message is lost: the server believes the handshake is complete but it actually is not. As the client is waiting for the Finished message, the client's retransmit timer will fire and it will retransmit the client's Finished message. This will cause the server to respond with its own Finished message, completing the handshake. The same logic applies on the server side for the resumed handshake.

Note that because of packet loss, it is possible for one side to be sending application data even though the other side has not received the first side's Finished message. Implementations MUST either discard or buffer all application data packets for the new epoch until they have received the Finished message for that epoch. Implementations MAY treat receipt of application data with a new epoch prior to receipt of the corresponding Finished message as evidence of reordering or packet loss and retransmit their final flight immediately, shortcutting the retransmission timer.

4.2.4.1. Timer Values

Though timer values are the choice of the implementation, mishandling of the timer can lead to serious congestion problems; for example, if many instances of a DTLS time out early and retransmit too quickly on a congested link. Implementations SHOULD use an initial timer value of 1 second (the minimum defined in RFC 6298 [RFC6298]) and double the value at each retransmission, up to no less than the RFC 6298 maximum of 60 seconds. Note that we recommend a 1-second timer rather than the 3-second RFC 6298 default in order to improve latency for time-sensitive applications. Because DTLS only uses retransmission for handshake and not dataflow, the effect on congestion should be minimal.

Implementations SHOULD retain the current timer value until a transmission without loss occurs, at which time the value may be reset to the initial value. After a long period of idleness, no less than 10 times the current timer value, implementations may reset the timer to the initial value. One situation where this might occur is when a rehandshake is used after substantial data transfer.

4.2.5. ChangeCipherSpec

As with TLS, the ChangeCipherSpec message is not technically a handshake message but MUST be treated as part of the same flight as the associated Finished message for the purposes of timeout and retransmission. This creates a potential ambiguity because the order of the ChangeCipherSpec cannot be established unambiguously with respect to the handshake messages in case of message loss.

This is not a problem with any current TLS mode because the expected set of handshake messages logically preceeding the ChangeCipherSpec is predictable from the rest of the handshake state. However, future modes MUST take care to avoid creating ambiguity.

4.2.6. CertificateVerify and Finished Messages

CertificateVerify and Finished messages have the same format as in TLS. Hash calculations include entire handshake messages, including DTLS-specific fields: message_seq, fragment_offset, and fragment_length. However, in order to remove sensitivity to handshake message fragmentation, the Finished MAC MUST be computed as if each handshake message had been sent as a single fragment. Note that in cases where the cookie exchange is used, the initial ClientHello and HelloVerifyRequest MUST NOT be included in the CertificateVerify or Finished MAC computations.

4.2.7. Alert Messages

Note that Alert messages are not retransmitted at all, even when they occur in the context of a handshake. However, a DTLS implementation which would ordinarily issue an alert SHOULD generate a new alert message if the offending record is received again (e.g., as a retransmitted handshake message). Implementations SHOULD detect when a peer is persistently sending bad messages and terminate the local connection state after such misbehavior is detected.

4.2.8. Establishing New Associations with Existing Parameters

If a DTLS client-server pair is configured in such a way that repeated connections happen on the same host/port quartet, then it is possible that a client will silently abandon one connection and then initiate another with the same parameters (e.g., after a reboot). This will appear to the server as a new handshake with epoch=0. In cases where a server believes it has an existing association on a given host/port quartet and it receives an epoch=0 ClientHello, it SHOULD proceed with a new handshake but MUST NOT destroy the existing association until the client has demonstrated reachability either by completing a cookie exchange or by completing a complete handshake including delivering a verifiable Finished message. After a correct Finished message is received, the server MUST abandon the previous association to avoid confusion between two valid associations with overlapping epochs. The reachability requirement prevents off-path/blind attackers from destroying associations merely by sending forged ClientHellos.