5. Payload Format for Haptics
Diese Seite fasst den entsprechenden Abschnitt von RFC 9993 zusammen und bewahrt RTP payload, MIHS, SDP und IANA-Details.
The RTP header usage, payload header, payload structures, Table 1, and Figures 2-9 are preserved below.
5. Payload Format for Haptics
5.1. RTP Header Usage
The RTP header is defined in [RFC3550] and represented in Figure 2.
Unless contextualized below, the meaning of the fields depicted in
Figure 2 is the same as in Section 5.1 of [RFC3550].
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp (TS) |
+---------------------------------------------------------------+
| Synchronization Source (SSRC) Identifier |
+---------------------------------------------------------------+
| Contributing Source (CSRC) Identifiers |
| .... |
+---------------------------------------------------------------+
Figure 2: RTP Header for Haptics
Marker bit (M): 1 bit. The marker bit SHOULD be set to one in the
first non-silent RTP packet after a period of haptic silence.
This enables jitter buffer adaptation and haptics device washout
(i.e., reset to a neutral position) prior to the beginning of the
burst with minimal impact on the quality of experience for the end
user. The marker bit in all other packets MUST be set to zero.
Timestamp (TS): 32 bits. A timestamp representing the sampling time
of the first sample of the MIHS unit in the RTP payload. The
clock frequency MUST be set to the sample rate of the encoded
haptic data and is conveyed out of band (e.g., as an SDP
parameter).
5.2. Payload Header
The RTP payload header follows the RTP header. Figure 3 describes
the RTP payload header for haptics.
+-+-+-+-+-+-+-+-+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|D| UT | L |
+-+-----+-------+
Figure 3: RTP Payload Header for Haptics
D (Dependency): 1 bit. This field indicates whether the MIHS unit
included in the RTP payload is dependent (when its value is one)
or independent (when its value is zero).
UT (Unit Type): 3 bits. This field indicates the type of the MIHS
unit included in the RTP payload. UT field values are listed in
Table 1.
L (MIHS Layer): 4 bits. This field is an integer value that
indicates the priority order of the MIHS unit included in the RTP
payload, as determined by the haptic sender (e.g., by the haptic
codec), based on application-specific needs. For example, the
sender may use the MIHS layer to prioritize perceptions with the
largest impact on the end-user experience. Zero corresponds to
the highest priority. The semantics of individual MIHS layers are
not specified and are left for the application to assign. In
cases where the sender does not use the L field to indicate the
priority order of the MIHS unit, the L value is '0'.
5.3. Payload Structures
Three different types of RTP packet payload structures are specified.
A single unit packet contains a single MIHS unit in the payload. A
fragmentation unit contains a subset of a MIHS unit. An aggregation
packet contains multiple MIHS units in the payload. The unit type
(UT) field of the RTP payload header, as shown in Table 1, identifies
both the payload structure and, in the case of a single-unit
structure, the type of MIHS unit present in the payload.
+===========+===================+==========================+
| Unit Type | Payload Structure | Packet Type Name |
+===========+===================+==========================+
| 0 | N/A | Unassigned |
+-----------+-------------------+--------------------------+
| 1 | Single | Initialization MIHS Unit |
+-----------+-------------------+--------------------------+
| 2 | Single | Temporal MIHS Unit |
+-----------+-------------------+--------------------------+
| 3 | Single | Spatial MIHS Unit |
+-----------+-------------------+--------------------------+
| 4 | Single | Silent MIHS Unit |
+-----------+-------------------+--------------------------+
| 5 | Aggr | Single-Time Aggregation |
| | | Packet (STAP) |
+-----------+-------------------+--------------------------+
| 6 | Aggr | Multi-Time Aggregation |
| | | Packet (MTAP) |
+-----------+-------------------+--------------------------+
| 7 | Frag | Fragmentation Unit |
+-----------+-------------------+--------------------------+
Table 1: Payload Structure Type for Haptics
The payload structures are represented in Figure 4. The single unit
payload structure is specified in Section 5.3.1. The fragmented unit
payload structure is specified in Section 5.3.2. The aggregation
packet payload structure is specified in Section 5.3.3. The padding
in the figures of these sections refers to the RTP padding defined in
[RFC3550].
+-------------------+
| RTP Header |
+-------------------+
| RTP Payload Header|
+-------------------+ | (UT = Aggr) |
| RTP Header | +-------------------+
+-------------------+ +-------------------+ | MIHS Unit 1 Size |
| RTP Header | | RTP Payload Header| +-------------------+
+-------------------+ | (UT = Frag) | | MIHS Unit 1 |
| RTP Payload Header| +-------------------+ +-------------------+
+-------------------+ | FU Header | | MIHS Unit 2 Size |
| RTP Payload | +-------------------+ +-------------------+
| (Single MIHS unit)| | RTP Payload | | ... |
+-------------------+ +-------------------+ +-------------------+
(a) single unit (b)fragmentation unit (c) aggregation packet
Figure 4: RTP Transmission Modes
5.3.1. Single Unit Payload Structure
In a single unit payload structure, as described in Figure 5, the RTP
packet contains the RTP header, followed by the payload header and
one single MIHS unit. The payload header follows the structure
described in Section 5.2. The payload contains a MIHS unit as
defined in [ISO.IEC.23090-31].
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Payload Header | |
+---------------+ |
| MIHS Unit Data |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |...OPTIONAL RTP Padding |
+-------------------------------+-------------------------------+
Figure 5: Single Unit Payload Structure
5.3.2. Fragmented Unit Payload Structure
In a fragmented unit payload structure, as described in Figure 6, the
RTP packet contains the RTP header, followed by the payload header, a
Fragmented Unit (FU) header, and a MIHS unit fragment. The payload
header follows the structure described in Section 5.2. The value of
the UT field of the payload header is 7. The FU header follows the
structure described in Figure 7. In the case of fragmentation, all
RTP payload header fields MUST remain unchanged across all fragments.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Payload Header | FU Header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| MIHS Unit Fragment |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |...OPTIONAL RTP Padding |
+-------------------------------+-------------------------------+
Figure 6: Fragmentation Unit Payload Structure
FU headers are used to enable fragmenting a single MIHS unit into
multiple RTP packets. Fragments of the same MIHS unit MUST be sent
in consecutive order with ascending RTP sequence numbers (with no
other RTP packets within the same RTP stream being sent between the
first and last fragment). FUs MUST NOT be nested, i.e., an FU MUST
NOT contain a subset of another FU.
Figure 7 describes an FU header, including the following fields:
+---+---+---+---+---+---+---+---+
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+---+---+---+---+---+---+---+---+
|FUS|FUE| RSV | UT |
+---+---+-----------+-----------+
Figure 7: Fragmentation Unit Header
FUS (Fragmented Unit Start): 1 bit. This field MUST be set to 1 for
the first fragment and 0 for the other fragments.
FUE (Fragmented Unit End): 1 bit. This field MUST be set to 1 for
the last fragment and 0 for the other fragments.
The combination FUS=1 and FUE=1 MUST NOT occur; such packets are
invalid.
RSV (Reserved): 3 bits. These bits MUST be set to 0 by the sender
and ignored by the receiver.
UT (Unit Type): 3 bits. This field indicates the type of the MIHS
unit this fragment belongs to, using values defined in Table 1.
The use of MIHS unit fragmentation in RTP means that a media receiver
can receive some fragments, but not other fragments. The missing
fragments will typically not be retransmitted by RTP. This results
in partially received MIHS units, which can be either dropped or used
by the decoding application, based on implementation. In cases where
consecutive fragments with FUE and FUS are lost, the receiver may be
able to detect that surrounding fragments belong to a different
partially received MIHS unit (e.g., if the UT field holds a different
value).
5.3.3. Aggregation Packet Payload Structure
In an aggregation packet, as described in Figure 8, the RTP packet
contains an RTP header, followed by a payload header, and (for each
aggregated MIHS unit) a MIHS unit size followed by the MIHS unit.
The payload header follows the structure described in Section 5.2.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|RTP Payl. Head.| MIHS Unit 1 Size | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| MIHS Unit 1 |
: :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MIHS Unit 2 Size | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| MIHS Unit 2 |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |...OPTIONAL RTP Padding |
+-------------------------------+-------------------------------+
Figure 8: Single-Time Aggregation Packet
Figure 8 shows a Single-Time Aggregation Packet (STAP), which can be
used to transmit multiple MIHS units that correspond to the same
timestamp. For example, if two frequencies are used for the same
content, they can be transmitted at once in a STAP. Multiple spatial
units can also be sent together in a STAP, since this type of haptics
data is time independent. The MIHS unit length field (16 bits) holds
the length of the MIHS unit following it, in bytes. The value of the
UT field of the payload header is 5.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|RTP Payl. Head.| MIHS Unit 1 Size | TS Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS Offset | |
+-+-+-+-+-+-+-+-+ |
| MIHS Unit 1 |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MIHS Unit 2 Size | TS Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| MIHS Unit 2 |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |...OPTIONAL RTP Padding |
+-------------------------------+-------------------------------+
Figure 9: Multiple-Time Aggregation Packet
Figure 9 shows a Multi-Time Aggregation Packet (MTAP). It is used to
transmit multiple MIHS units with different timestamps, in one RTP
packet. Multi-time aggregation can help reduce the number of packets
in environments where some delay is acceptable. The value of the UT
field of the payload header is 6. The MIHS unit length field (16
bits) holds the length of the MIHS unit following it, in bytes. The
timestamp offset field (TS offset, 16 bits) is present in the MTAP
case and MUST be set to the value of (time of the MIHS unit - RTP
timestamp of the packet). The timestamp offset of the earliest
aggregation unit MUST always be zero. Therefore, the RTP timestamp
of the MTAP is identical to the earliest MIHS unit time.
5.4. MIHS Units Transmission and Reception Considerations
The following considerations apply for the streaming of MIHS units
over RTP.
The MIHS format enables variable duration units and uses
initialization MIHS units to declare the duration of subsequent non-
zero duration MIHS units, as well as the maximum variation of this
duration. A sender SHOULD set constant or low-variability (e.g.,
lower than the playout buffer) durations in initialization MIHS
units, for RTP streaming. This enables the receiver to determine
early (e.g., using a timer) when a unit has been lost and to make the
decoder more robust to RTP packet loss. If a sender sends MIHS units
with high duration variations, the receiver MAY need to wait for a
long period of time (e.g., the upper bound of the duration variation)
to determine if a MIHS unit was lost in transmission. Whether this
behavior is acceptable or not is application dependent, and the
application can configure the encoder to generate MIHS units with
lengths that have the appropriate variation.
The MIHS format uses silent MIHS units to signal haptic silence. A
sender MAY decide not to send silent units, to save network
resources. Since, from a receiver standpoint, a missed MIHS unit may
originate from a not-sent silent unit or a lost packet, a sender MAY
send one, or a few, MIHS silent units at the beginning of a haptic
silence. If a media receiver receives a MIHS silent unit, the
receiver SHOULD assume that silence is intended until the reception
of a non-silent MIHS unit. This can reduce the number of false
detections of lost RTP packets by the decoder.
In some multimedia conference scenarios using an RTP video mixer
(e.g., when adding or selecting a new source), it is recommended to
use Full Intra Request (FIR) feedback messages [RFC5104] with
haptics. The purpose of the FIR message is to cause an encoder to
send a decoder refresh point at the earliest opportunity. In the
context of haptics, an appropriate decoder refresh point is an
initialization MIHS unit. The initialization MIHS unit point enables
a decoder to be reset to a known state and to decode all MIHS units
following it.