4.2. VP9 Payload Descriptor

In flexible mode (with the F bit below set to one), the first octets after the RTP header are the VP9 payload descriptor, with the following structure.¶

      0 1 2 3 4 5 6 7
     +-+-+-+-+-+-+-+-+
     |I|P|L|F|B|E|V|Z| (REQUIRED)
     +-+-+-+-+-+-+-+-+
I:   |M| PICTURE ID  | (REQUIRED)
     +-+-+-+-+-+-+-+-+
M:   | EXTENDED PID  | (RECOMMENDED)
     +-+-+-+-+-+-+-+-+
L:   | TID |U| SID |D| (Conditionally RECOMMENDED)
     +-+-+-+-+-+-+-+-+                             -\
P,F: | P_DIFF      |N| (Conditionally REQUIRED)    - up to 3 times
     +-+-+-+-+-+-+-+-+                             -/
V:   | SS            |
     | ..            |
     +-+-+-+-+-+-+-+-+

In non-flexible mode (with the F bit below set to zero), the first octets after the RTP header are the VP9 payload descriptor, with the following structure.¶

      0 1 2 3 4 5 6 7
     +-+-+-+-+-+-+-+-+
     |I|P|L|F|B|E|V|Z| (REQUIRED)
     +-+-+-+-+-+-+-+-+
I:   |M| PICTURE ID  | (RECOMMENDED)
     +-+-+-+-+-+-+-+-+
M:   | EXTENDED PID  | (RECOMMENDED)
     +-+-+-+-+-+-+-+-+
L:   | TID |U| SID |D| (Conditionally RECOMMENDED)
     +-+-+-+-+-+-+-+-+
     |   TL0PICIDX   | (Conditionally REQUIRED)
     +-+-+-+-+-+-+-+-+
V:   | SS            |
     | ..            |
     +-+-+-+-+-+-+-+-+

Except as noted, the following field descriptions apply to the payload descriptor formats in both Figures 2 and 3.¶

I:

Picture ID (PID) present. When set to one, the OPTIONAL PID MUST be present after the mandatory first octet and specified as below. Otherwise, PID MUST NOT be present. If the V bit was set in the stream's most recent start of a keyframe (i.e., the SS field was present) and the F bit is set to zero (i.e., non-flexible scalability mode is in use), then this bit MUST be set on every packet.¶

P:

Inter-picture predicted frame. When set to zero, the frame does not utilize inter-picture prediction. In this case, up-switching to a current spatial layer's frame is possible from a directly lower spatial-layer frame. P SHOULD also be set to zero when encoding a layer synchronization frame in response to a Layer Refresh Request (LRR) [RFC9627] message (see Section 5.3). When P is set to zero, the Temporal-layer ID (TID) field (described below) MUST also be set to zero (if present). Note that the P bit does not forbid intra-picture, inter-layer prediction from earlier frames of the same picture, if any.¶

L:

Layer indices present. When set to one, the one or two octets following the mandatory first octet and the PID (if present) is as described by "Layer indices" below. If the F bit (described below) is set to one (indicating flexible mode), then only one octet is present for the layer indices. Otherwise, if the F bit is set to zero (indicating non-flexible mode), then two octets are present for the layer indices.¶

F:

Flexible mode. When set to one, this indicates flexible mode; if the P bit is also set to one, then the octets following the mandatory first octet, the PID, and layer indices (if present) are as described by "reference indices" below. This bit MUST only be set to one if the I bit is also set to one; if the I bit is set to zero, then this bit MUST also be set to zero and ignored by receivers. (Flexible mode's reference indices are defined as offsets from the Picture ID field, so they would have no meaning if I were not set.) The value of the F bit MUST only change on the first packet of a key picture. A "key picture" is a picture whose base spatial-layer frame is a keyframe, and thus one which completely resets the encoder state. This packet will have its P bit equal to zero, SID or L bit (described below) equal to zero, and B bit (described below) equal to one.¶

B:

Start of Frame. This bit MUST be set to one if the first payload octet of the RTP packet is the beginning of a new VP9 frame; otherwise, it MUST NOT be one. Note that this frame might not be the first frame of a picture.¶

E:

End of Frame. This bit MUST be set to one for the final RTP packet of a VP9 frame; otherwise, it is zero. This enables a decoder to finish decoding the frame, where it otherwise may need to wait for the next packet to explicitly know that the frame is complete. Note that, if spatial scalability is in use, more frames from the same picture may follow; see the description of the B bit above.¶

V:

Scalability Structure (SS) data present. When set to one, the OPTIONAL SS data MUST be present in the payload descriptor. Otherwise, the SS data MUST NOT be present.¶

Z:

Not a reference frame for upper spatial layers. If set to one, indicates that frames with higher spatial layers SID+1 and greater of the current and following pictures do not depend on the current spatial-layer SID frame. This enables a decoder that is targeting a higher spatial layer to know that it can safely discard this packet's frame without processing it, without having to wait for the D bit in the higher-layer frame (see below).¶

The mandatory first octet is followed by the extension data fields that are enabled:¶

M:

The most significant bit of the first octet is an extension flag. The field MUST be present if the I bit is equal to one. If M is set, the PID field MUST contain 15 bits; otherwise, it MUST contain 7 bits. See PID below.¶

Picture ID (PID):

Picture ID represented in 7 or 15 bits, depending on the M bit. This is a running index of the pictures, where the sender increments the value by one for each picture it sends. (Note, however, that because a middlebox can discard pictures where permitted by the SS, Picture IDs as received by a receiver might not be contiguous.) This field MUST be present if the I bit is equal to one. If M is set to zero, 7 bits carry the PID; else, if M is set to one, 15 bits carry the PID in network byte order. The sender may choose between a 7- or 15-bit index. The PID SHOULD start on a random number and MUST wrap after reaching the maximum ID (0x7f or 0x7fff depending on the index size chosen). The receiver MUST NOT assume that the number of bits in the PID stays the same through the session. If this field transitions from 7 bits to 15 bits, the value is zero-extended (i.e., the value after 0x6e is 0x006f); if the field transitions from 15 bits to 7 bits, it is truncated (i.e., the value after 0x1bbe is 0x3f).¶

In the non-flexible mode (when the F bit is set to zero), this PID is used as an index to the PG specified in the SS data below. In this mode, the PID of the keyframe corresponds to the first specified frame in the PG. Then subsequent PIDs are mapped to subsequently specified frames in the PG (modulo N_G, specified in the SS data below), respectively.¶

All frames of the same picture MUST have the same PID value.¶

Frames (and their corresponding pictures) with the VP9 show_frame field equal to zero MUST have distinct PID values from subsequent pictures with show_frame equal to one. Thus, a picture (as defined in this specification) is different than a VP9 superframe.¶

All frames of the same picture MUST have the same value for show_frame.¶

Layer indices:

This field is optional but RECOMMENDED whenever encoding with layers. For both flexible and non-flexible modes, one octet is used to specify a layer frame's Temporal-layer ID (TID) and Spatial-layer ID (SID) as shown both in Figures 2 and 3. Additionally, a bit (U) is used to indicate that the current frame is a "switching up point" frame. Another bit (D) is used to indicate whether inter-layer prediction is used for the current frame.¶

In the non-flexible mode (when the F bit is set to zero), another octet is used to represent the Temporal Layer 0 Picture Index (8 bits) (TL0PICIDX), as depicted in Figure 3. The TL0PICIDX is present so that all minimally required frames (the base temporal-layer frames) can be tracked.¶

The TID and SID fields indicate the temporal and spatial layers and can help middleboxes and endpoints quickly identify which layer a packet belongs to.¶

TID:

The temporal-layer ID of the current frame. In the case of non-flexible mode, if a PID is mapped to a picture in a specified PG, then the value of the TID MUST match the corresponding TID value of the mapped picture in the PG.¶

U:

Switching up point. When this bit is set to one, if the current picture has a temporal-layer ID equal to value T, then subsequent pictures with temporal-layer ID values higher than T will not depend on any picture before the current picture (in decode order) with a temporal-layer ID value greater than T.¶

SID:

The spatial-layer ID of the current frame. Note that frames with spatial-layer SID > 0 may be dependent on decoded spatial-layer SID-1 frame within the same picture. Different frames of the same picture MUST have distinct spatial-layer IDs, and frames' spatial layers MUST appear in increasing order within the frame.¶

D:

Inter-layer dependency is used. D MUST be set to one if and only if the current spatial-layer SID frame depends on spatial-layer SID-1 frame of the same picture; otherwise, it MUST be set to zero. For the base-layer frame (with SID equal to zero), the D bit MUST be set to zero.¶

TL0PICIDX:

Temporal Layer 0 Picture Index (8 bits). TL0PICIDX is only present in the non-flexible mode (F = 0). This is a running index for the temporal base-layer pictures, i.e., the pictures with a TID set to zero. If the TID is larger than zero, TL0PICIDX indicates which temporal base-layer picture the current picture depends on. TL0PICIDX MUST be incremented by one when the TID is equal to zero. The index SHOULD start on a random number and MUST restart at zero after reaching the maximum number 255.¶

Reference indices:

When P and F are both set to one, indicating a non-keyframe in flexible mode, then at least one reference index MUST be specified as below. Additional reference indices (a total of up to three reference indices are allowed) may be specified using the N bit below. When either P or F is set to zero, then no reference index is specified.¶

P_DIFF:

The reference index (in 7 bits) specified as the relative PID from the current picture. For example, when P_DIFF=3 on a packet containing the picture with PID 112 means that the picture refers back to the picture with PID 109. This calculation is done modulo the size of the PID field, i.e., either 7 or 15 bits. A P_DIFF value of zero is invalid.¶

N:

1 if there is additional P_DIFF following the current P_DIFF.¶

4.3. Frame Fragmentation

VP9 frames are fragmented into packets in RTP sequence number order: beginning with a packet with the B bit set and ending with a packet with the E bit set. There is no mechanism for finer-grained access to parts of a VP9 frame.¶

4.4. Scalable Encoding Considerations

In addition to the use of reference frames, VP9 has several additional forms of inter-frame dependencies, largely involving probability tables for the entropy and tree encoders. In VP9 syntax, the syntax element "error_resilient_mode" resets this additional inter-frame data, allowing a frame's syntax to be decoded independently.¶

Due to the requirements of scalable streams, a VP9 encoder producing a scalable stream needs to ensure that a frame does not depend on a previous frame (of the same or a previous picture) that can legitimately be removed from the stream. Thus, a frame that follows a frame that might be removed (in full decode order) MUST be encoded with "error_resilient_mode" set to true.¶

For spatially scalable streams, this means that "error_resilient_mode" needs to be turned on for the base spatial layer; however, it can be turned off for higher spatial layers, assuming they are sent with inter-layer dependency (i.e., with the D bit set). For streams that are only temporally scalable without spatial scalability, "error_resilient_mode" can additionally be turned off for any picture that immediately follows a temporal-layer 0 frame.¶

4.5. Example of a VP9 RTP Stream

5.1. Reference Picture Selection Indication (RPSI)

The RPSI is a payload-specific feedback message defined within the RTCP-based feedback format. The RPSI message is generated by a receiver and can be used in two ways: either it can signal a preferred reference picture when a loss has been detected by the decoder (preferably a reference that the decoder knows is perfect) or it can be used as positive feedback information to acknowledge correct decoding of certain reference pictures. The positive feedback method is useful for VP9 used for point-to-point (unicast) communication. The use of RPSI for VP9 is preferably combined with a special update pattern of the codec's two special reference frames -- the golden frame and the altref frame -- in which they are updated in an alternating leapfrog fashion. When a receiver has received and correctly decoded a golden or altref frame, and that frame had a Picture ID in the payload descriptor, the receiver can acknowledge this simply by sending an RPSI message back to the sender. The message body (i.e., the "native RPSI bit string" in [RFC4585]) is simply the (7- or 15-bit) Picture ID of the received frame.¶

5.2. Full Intra Request (FIR)

The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a receiver to request a full state refresh of an encoded stream.¶

Upon receipt of a FIR request, a VP9 sender MUST send a picture with a keyframe for its spatial-layer 0 layer frame and then send frames without inter-picture prediction (P=0) for any higher-layer frames.¶

5.3. Layer Refresh Request (LRR)

The Layer Refresh Request (LRR) [RFC9627] allows a receiver to request a single layer of a spatially or temporally encoded stream to be refreshed without necessarily affecting the stream's other layers.¶

            +---------------+---------------+
            |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
            +---------------+---------+-----+
            |   RES   | TID | RES     | SID |
            +---------------+---------+-----+

Figure 5 shows the format of an LRR's layer index fields for VP9 streams. The two "RES" fields MUST be set to zero on transmission and ignored on reception. See Section 4.2 for details on the TID and SID fields.¶

Identification of a layer refresh frame can be derived from the reference IDs of each frame by backtracking the dependency chain until reaching a point where only decodable frames are being referenced. Therefore, it's recommended for both the flexible and the non-flexible mode that, when switching up points are being encoded in response to an LRR, those packets contain layer indices and the reference field or fields so that the decoder or selective forwarding middleboxes [RFC7667] can make this derivation.¶

Example:¶

LRR {1,0}, {2,1} is sent by a Multipoint Control Unit (MCU) when it is currently relaying {1,0} to a receiver that wants to upgrade to {2,1}. In response, the encoder should encode the next frames in layers {1,1} and {2,1} by only referring to frames in {1,0} or {0,0}.¶

In the non-flexible mode, periodic upgrade frames can be defined by the layer structure of the SS; thus, periodic upgrade frames can be automatically identified by the Picture ID.¶

6.1. SDP Parameters

m=video 49170 RTP/AVPF 98
a=rtpmap:98 VP9/90000
a=fmtp:98 max-fr=30;max-fs=3600;profile-id=0

11.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.

[RFC3264]

Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, DOI 10.17487/RFC3264, June 2002, <https://www.rfc-editor.org/info/rfc3264>.

[RFC3550]

Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, July 2003, <https://www.rfc-editor.org/info/rfc3550>.

[RFC4585]

Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 10.17487/RFC4585, July 2006, <https://www.rfc-editor.org/info/rfc4585>.

[RFC4855]

Casner, S., "Media Type Registration of RTP Payload Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007, <https://www.rfc-editor.org/info/rfc4855>.

[RFC5104]

Wenger, S., Chandra, U., Westerlund, M., and B. Burman, "Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, February 2008, <https://www.rfc-editor.org/info/rfc5104>.

[RFC6838]

Freed, N., Klensin, J., and T. Hansen, "Media Type Specifications and Registration Procedures", BCP 13, RFC 6838, DOI 10.17487/RFC6838, January 2013, <https://www.rfc-editor.org/info/rfc6838>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8866]

Begen, A., Kyzivat, P., Perkins, C., and M. Handley, "SDP: Session Description Protocol", RFC 8866, DOI 10.17487/RFC8866, January 2021, <https://www.rfc-editor.org/info/rfc8866>.

[RFC9627]

Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. Flodman, "The Layer Refresh Request (LRR) RTCP Feedback Message", RFC 9627, DOI 10.17487/RFC9627, March 2025, <https://www.rfc-editor.org/info/rfc9627>.

[VP9-BITSTREAM]

Grange, A., de Rivaz, P., and J. Hunt, "VP9 Bitstream & Decoding Process Specification", Version 0.6, 31 March 2016, <https://storage.googleapis.com/downloads.webmproject.org/docs/vp9/vp9-bitstream-specification-v0.6-20160331-draft.pdf>.

11.2. Informative References

[RFC3551]

Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, DOI 10.17487/RFC3551, July 2003, <https://www.rfc-editor.org/info/rfc3551>.

[RFC3711]

Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, DOI 10.17487/RFC3711, March 2004, <https://www.rfc-editor.org/info/rfc3711>.

[RFC5124]

Ott, J. and E. Carrara, "Extended Secure RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 2008, <https://www.rfc-editor.org/info/rfc5124>.

[RFC6386]

Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding Guide", RFC 6386, DOI 10.17487/RFC6386, November 2011, <https://www.rfc-editor.org/info/rfc6386>.

[RFC7201]

Westerlund, M. and C. Perkins, "Options for Securing RTP Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014, <https://www.rfc-editor.org/info/rfc7201>.

[RFC7202]

Perkins, C. and M. Westerlund, "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 2014, <https://www.rfc-editor.org/info/rfc7202>.

[RFC7667]

Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667, DOI 10.17487/RFC7667, November 2015, <https://www.rfc-editor.org/info/rfc7667>.