Skip to main content

13.3 Example of Robust Packet Scheduling

13.3. Example of Robust Packet Scheduling​

An example of robust packet scheduling follows. The communication system used in the example consists of the following components in the order that the video is processed from source to sink:

  • camera and capturing
  • pre-encoding buffer
  • encoder
  • encoded picture buffer
  • transmitter
  • transmission channel
  • receiver
  • receiver buffer
  • decoder
  • decoded picture buffer
  • display

The video communication system used in this example operates as follows. Note that processing of the video stream happens gradually and at the same time in all components of the system. The source video sequence is shot and captured to a pre-encoding buffer. The pre-encoding buffer can be used to order pictures from sampling order to encoding order or to analyze multiple uncompressed frames for bitrate control purposes, for example. In some cases, the pre-encoding buffer may not exist; instead, the sampled pictures are encoded right away. The encoder encodes pictures from the pre-encoding buffer and stores the output (i.e., coded pictures) to the encoded picture buffer. The transmitter encapsulates the coded pictures from the encoded picture buffer to transmission packets and sends them to a receiver through a transmission channel. The receiver stores the received packets to the receiver buffer. The receiver buffering process typically includes buffering for transmission delay jitter. The receiver buffer can also be used to recover correct decoding order of coded data. The decoder reads coded data from the receiver buffer and produces decoded pictures as output into the decoded picture buffer. The decoded picture buffer is used to recover the output (or display) order of pictures. Finally, pictures are displayed.

In the following example figures, I denotes an IDR picture, R denotes a reference picture, N denotes a non-reference picture, and the number after I, R, or N indicates the sampling time relative to the previous IDR picture in decoding order. Values below the sequence of pictures indicate scaled system clock timestamps. The system clock is initialized arbitrarily in this example, and time runs from left to right. Each I, R, and N picture is mapped into the same timeline compared to the previous processing step, if any, assuming that encoding, transmission, and decoding take no time. Thus, events happening at the same time are located in the same column throughout all example figures.

A subset of a sequence of coded pictures is depicted below in sampling order.

    ...  N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ...
    ... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ...
    ...  58  59  60  61  62  63  64  65  66  ... 128 129 130 131 ...

Figure 16. Sequence of pictures in sampling order

The sampled pictures are buffered in the pre-encoding buffer to arrange them in encoding order. In this example, we assume that the non-reference pictures are predicted from both the previous and the next reference picture in output order, except for the non-reference pictures immediately preceding an IDR picture, which are predicted only from the previous reference picture in output order. Thus, the pre-encoding buffer has to contain at least two pictures, and the buffering causes a delay of two picture intervals. The output of the pre-encoding buffering process and the encoding (and decoding) order of the pictures are as follows:

   ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...
   ... -|---|---|---|---|---|---|---|---|- ...
   ... 60  61  62  63  64  65  66  67  68  ...

Figure 17. Reordered pictures in the pre-encoding buffer

The encoder or the transmitter can set the value of DON for each picture to a value of DON for the previous picture in decoding order plus one.

For the sake of simplicity, let us assume that:

  • the frame rate of the sequence is constant,
  • each picture consists of only one slice,
  • each slice is encapsulated in a single NAL unit packet,
  • there is no transmission delay, and
  • pictures are transmitted at constant intervals (that is, 1 / (frame rate)).

When pictures are transmitted in decoding order, they are received as follows:

   ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...
   ... -|---|---|---|---|---|---|---|---|- ...
   ... 60  61  62  63  64  65  66  67  68  ...

Figure 18. Received pictures in decoding order

The OPTIONAL sprop-interleaving-depth media type parameter is set to 0, as the transmission (or reception) order is identical to the decoding order.

Initially, the decoder has to buffer for one picture interval in its decoded picture buffer to organize pictures from decoding order to output order, as depicted below:

   ... N58 N59 I00 N01 N02 R03 N04 N05 R06 ...
   ... -|---|---|---|---|---|---|---|---|- ...
   ... 61  62  63  64  65  66  67  68  69  ...

Figure 19. Output order

The amount of required initial buffering in the decoded picture buffer can be signaled in the buffering period SEI message or with the num_reorder_frames syntax element of H.264 video usability information. num_reorder_frames indicates the maximum number of frames, complementary field pairs, or non-paired fields that precede any frame, complementary field pair, or non-paired field in the sequence in decoding order and that follow it in output order. For the sake of simplicity, we assume that num_reorder_frames is used to indicate the initial buffer in the decoded picture buffer. In this example, num_reorder_frames is equal to 1.

It can be observed that if the IDR picture I00 is lost during transmission and a retransmission request is issued when the value of the system clock is 62, there is one picture interval of time (until the system clock reaches timestamp 63) to receive the retransmitted IDR picture I00.

Let us then assume that IDR pictures are transmitted two frame intervals earlier than their decoding position; that is, the pictures are transmitted as follows:

   ...  I00 N58 N59 R03 N01 N02 R06 N04 N05 ...
   ... --|---|---|---|---|---|---|---|---|- ...
   ...  62  63  64  65  66  67  68  69  70  ...

Figure 20. Interleaving: Early IDR pictures in sending order

The OPTIONAL sprop-interleaving-depth media type parameter is set equal to 1 according to its definition. (The value of sprop-interleaving-depth in this example can be derived as follows: picture I00 is the only picture preceding picture N58 or N59 in transmission order and following it in decoding order. Except for pictures I00, N58, and N59, the transmission order is the same as the decoding order of pictures. As a coded picture is encapsulated into exactly one NAL unit, the value of sprop-interleaving-depth is equal to the maximum number of pictures preceding any picture in transmission order and following the picture in decoding order).

The receiver buffering process contains two pictures at a time according to the value of the sprop-interleaving-depth parameter and orders pictures from the reception order to the correct decoding order based on the value of DON associated with each picture. The output of the receiver buffering process is as follows:

   ... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...
   ... -|---|---|---|---|---|---|---|---|- ...
   ... 63  64  65  66  67  68  69  70  71  ...

Figure 21. Interleaving: Receiver buffer

Again, an initial buffering delay of one picture interval is needed to organize pictures from decoding order to output order, as depicted below:

    ... N58 N59 I00 N01 N02 R03 N04 N05 ...
    ... -|---|---|---|---|---|---|---|- ...
    ... 64  65  66  67  68  69  70  71  ...

Figure 22. Interleaving: Receiver buffer after reordering

Note that the maximum delay that IDR pictures can undergo during transmission, including possible application, transport, or link layer retransmission, is equal to three picture intervals. Thus, the loss resiliency of IDR pictures is improved in systems supporting retransmission compared to the case in which pictures are transmitted in their decoding order.

Last updated on