WebRTC研究：FEC之RED封装

首先看下webrtcglossary中的定义。

RED stands for REDundant coding and it is a RTP payload format defined in RFC 2198 for encoding redundant audio or video data.

The primary motivation of sending redundant data is to be able to recover packets lost under lossy network conditions. If a packet is lost then the missing information may be reconstructed at the receiver from the redundant data that arrives in the following packet(s).

The use of RED is negotiated in the SDP as an additional payload type and when used the audio/video RTP packets are packaged using RED format with a 6 bytes header, before the primary and secondary payloads conveying the actual audio/video packet and the redundant information.

RED封装简介

RED指冗余编码，并且依据RFC 2198定义了一种RTP有效载荷格式，用于携带冗余编码的音视频数据。RED封装的数据作为payload跟在RTP报头后面。RED封装的数据有自己的Header。在一个RTP包中，会包含多个RED Header，指定了后面携带的编码数据信息。这些RED Header依次跟在RTP报头后面，在这些RED Header后就是各个RED Header对应的冗余编码数据。

RED Header结构如下：

0                   1                    2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3  4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|   block PT  |  timestamp offset         |   block length    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

F： 1 bit长度，说明后面是否还有其他RED Header跟着。为1说明还有其他RED Header跟着，为0说明这是最后一个RED Header
blcok PT：7 bits长度，表示该RED Header对应的冗余编码类型
timeoffset： 14 bits长度，表示无符号长度时间戳偏移，该偏移是相对于RTP Header的时间戳。用无符号长度做偏移意味着冗余编码数据必须在发送完原始数据后才能发送
block length: 10 bits长度，表示该RED Header对应编码数据块长度，该长度包含RED Header字段

对于RTP包中最后一个RED Header，可以忽略block length以及timestamp。因为它们可以从RTP Fixed Header以及整个RTP包长度计算得到。最后一个RED Header结构如下：

0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|0|   Block PT  |
+-+-+-+-+-+-+-+-+

0 1 2 3 4 5 6 7

+-+-+-+-+-+-+-+-+

|0| Block PT |

+-+-+-+-+-+-+-+-+

其中F-bit位始终为0。

如下是一个完整RTP包示例，包含两个编码数据块：DVI4编码块，以及一个LPC冗余编码块。可以看到有两个RED Header跟在RTP Fixed Header后，第一个RED header F-bit为1，第二个RED Header由于是最后一个，所以F-bit为0，不包含时间戳偏移以及数据块长度信息。在RED Headrs后就是对应的编码数据块。

0                   1                    2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3  4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC=0  |M|      PT     |   sequence number of primary  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              timestamp  of primary encoding                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           synchronization source (SSRC) identifier            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| block PT=7  |  timestamp offset         |   block length    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| block PT=5  |                                               |
+-+-+-+-+-+-+-+-+                                               +
|                                                               |
+                LPC encoded redundant data (PT=7)              +
|                (14 bytes)                                     |
+                                               +---------------+
|                                               |               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                DVI4 encoded primary data (PT=5)               |
+                (84 bytes, not to scale)                       +
/                                                               /
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                               +---------------+
|                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|V=2|P|X| CC=0 |M| PT | sequence number of primary |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| timestamp of primary encoding |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| synchronization source (SSRC) identifier |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|0| block PT=5 | |

+-+-+-+-+-+-+-+-+ +

| |

+ LPC encoded redundant data (PT=7) +

| (14 bytes) |

+ +---------------+

| | |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +

| |

+ +

| |

+ +

| |

+ +

| DVI4 encoded primary data (PT=5) |

+ (84 bytes, not to scale) +

/ /

+ +

| |

+ +

| |

+ +---------------+

| |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

视频RED封装代码

这里我们以视频FEC为例，在WebRTC中音频的带外FEC也用到了RED编码。

UlpfecGenerator::GetUlpfecPacketsAsRed

WebRTC视频相关RED封装代码位于UlpfecGenerator类中。RED封装主要配合UlpFEC使用。这里我们看下UlpFEC编码数据封装为RED包的过程:

std::vector<std::unique_ptr<RedPacket>> UlpfecGenerator::GetUlpfecPacketsAsRed(
int red_payload_type,
int ulpfec_payload_type,
uint16_t first_seq_num) {
std::vector<std::unique_ptr<RedPacket>> red_packets;
red_packets.reserve(generated_fec_packets_.size());
RTC_DCHECK(!media_packets_.empty());
ForwardErrorCorrection::Packet* last_media_packet =
media_packets_.back().get();
uint16_t seq_num = first_seq_num;
for (const auto* fec_packet : generated_fec_packets_) {
// Wrap FEC packet (including FEC headers) in a RED packet. Since the
// FEC packets in |generated_fec_packets_| don't have RTP headers, we
// reuse the header from the last media packet.
RTC_DCHECK_GT(last_media_packet_rtp_header_length_, 0);
std::unique_ptr<RedPacket> red_packet(
new RedPacket(last_media_packet_rtp_header_length_ +
kRedForFecHeaderLength + fec_packet->data.size()));
// 创建报头
red_packet->CreateHeader(last_media_packet->data.data(),
last_media_packet_rtp_header_length_,
red_payload_type, ulpfec_payload_type);
red_packet->SetSeqNum(seq_num++);
red_packet->ClearMarkerBit();
// 设置UlpFEC编码数据
red_packet->AssignPayload(fec_packet->data.data(), fec_packet->data.size());
red_packets.push_back(std::move(red_packet));
}
ResetState();
return red_packets;
}

std::vector<std::unique_ptr<RedPacket>> UlpfecGenerator::GetUlpfecPacketsAsRed(

int red_payload_type,

int ulpfec_payload_type,

uint16_t first_seq_num) {

std::vector<std::unique_ptr<RedPacket>> red_packets;

red_packets.reserve(generated_fec_packets_.size());

RTC_DCHECK(!media_packets_.empty());

ForwardErrorCorrection::Packet* last_media_packet =

media_packets_.back().get();

uint16_t seq_num = first_seq_num;

for (const auto* fec_packet : generated_fec_packets_) {

// Wrap FEC packet (including FEC headers) in a RED packet. Since the

// FEC packets in |generated_fec_packets_| don't have RTP headers, we

// reuse the header from the last media packet.

RTC_DCHECK_GT(last_media_packet_rtp_header_length_, 0);

std::unique_ptr<RedPacket> red_packet(

new RedPacket(last_media_packet_rtp_header_length_ +

kRedForFecHeaderLength + fec_packet->data.size()));

// 创建报头

red_packet->CreateHeader(last_media_packet->data.data(),

last_media_packet_rtp_header_length_,

red_payload_type, ulpfec_payload_type);

red_packet->SetSeqNum(seq_num++);

red_packet->ClearMarkerBit();

// 设置UlpFEC编码数据

red_packet->AssignPayload(fec_packet->data.data(), fec_packet->data.size());

red_packets.push_back(std::move(red_packet));

}

ResetState();

return red_packets;

}

RedPacket::CreateHeader

对于视频，WebRTC中RED HeaderF-bit总是为0，不包含时间戳偏移等信息，也就是说一个RTP包只包含一个RED header：

void RedPacket::CreateHeader(const uint8_t* rtp_header,
size_t header_length,
int red_payload_type,
int payload_type) {
RTC_DCHECK_LE(header_length + kRedForFecHeaderLength, length_);
memcpy(data_.get(), rtp_header, header_length);
// Replace payload type.
data_[1] &= 0x80;
data_[1] += red_payload_type;
// Add RED header
// f-bit always 0
data_[header_length] = static_cast<uint8_t>(payload_type);
header_length_ = header_length + kRedForFecHeaderLength;
}

void RedPacket::CreateHeader(const uint8_t* rtp_header,

size_t header_length,

int red_payload_type,

int payload_type) {

RTC_DCHECK_LE(header_length + kRedForFecHeaderLength, length_);

memcpy(data_.get(), rtp_header, header_length);

// Replace payload type.

data_[1] &= 0x80;

data_[1] += red_payload_type;

// Add RED header

// f-bit always 0

data_[header_length] = static_cast<uint8_t>(payload_type);

header_length_ = header_length + kRedForFecHeaderLength;

}

视频RED解析代码

相关处理位于UlpfecReceiverImpl类中，我们看下主要的代码：

bool UlpfecReceiverImpl::AddReceivedRedPacket(const RtpPacket& rtp_packet,
uint8_t ulpfec_payload_type) {
// Remove RED header of incoming packet and store as a virtual RTP packet.
auto received_packet =
std::make_unique<ForwardErrorCorrection::ReceivedPacket>();
received_packet->pkt = new ForwardErrorCorrection::Packet();
// Get payload type from RED header and sequence number from RTP header.
uint8_t payload_type = rtp_packet.payload()[0] & 0x7f;
received_packet->is_fec = payload_type == ulpfec_payload_type;
received_packet->ssrc = rtp_packet.Ssrc();
received_packet->seq_num = rtp_packet.SequenceNumber();
// 如前面所说，WebRTC中一个视频RTP包只包含一个RED Header
if (rtp_packet.payload()[0] & 0x80) {
// f bit set in RED header, i.e. there are more than one RED header blocks.
// WebRTC never generates multiple blocks in a RED packet for FEC.
RTC_LOG(LS_WARNING) << "More than 1 block in RED packet is not supported.";
return false;
}
++packet_counter_.num_packets;
packet_counter_.num_bytes += rtp_packet.size();
if (packet_counter_.first_packet_time_ms == -1) {
packet_counter_.first_packet_time_ms = rtc::TimeMillis();
}
// 获取RED Header后携带的FEC冗余编码数据
if (received_packet->is_fec) {
++packet_counter_.num_fec_packets;
// everything behind the RED header
received_packet->pkt->data =
rtp_packet.Buffer().Slice(rtp_packet.headers_size() + kRedHeaderLength,
rtp_packet.payload_size() - kRedHeaderLength);
} 
return true;
}

bool UlpfecReceiverImpl::AddReceivedRedPacket(const RtpPacket& rtp_packet,

uint8_t ulpfec_payload_type) {

// Remove RED header of incoming packet and store as a virtual RTP packet.

auto received_packet =

std::make_unique<ForwardErrorCorrection::ReceivedPacket>();

received_packet->pkt = new ForwardErrorCorrection::Packet();

// Get payload type from RED header and sequence number from RTP header.

uint8_t payload_type = rtp_packet.payload()[0] & 0x7f;

received_packet->is_fec = payload_type == ulpfec_payload_type;

received_packet->ssrc = rtp_packet.Ssrc();

received_packet->seq_num = rtp_packet.SequenceNumber();

// 如前面所说，WebRTC中一个视频RTP包只包含一个RED Header

if (rtp_packet.payload()[0] & 0x80) {

// f bit set in RED header, i.e. there are more than one RED header blocks.

// WebRTC never generates multiple blocks in a RED packet for FEC.

RTC_LOG(LS_WARNING) << "More than 1 block in RED packet is not supported.";

return false;

}

++packet_counter_.num_packets;

packet_counter_.num_bytes += rtp_packet.size();

if (packet_counter_.first_packet_time_ms == -1) {

packet_counter_.first_packet_time_ms = rtc::TimeMillis();

}

// 获取RED Header后携带的FEC冗余编码数据

if (received_packet->is_fec) {

++packet_counter_.num_fec_packets;

// everything behind the RED header

received_packet->pkt->data =

rtp_packet.Buffer().Slice(rtp_packet.headers_size() + kRedHeaderLength,

rtp_packet.payload_size() - kRedHeaderLength);

}

return true;

}

参考

[1] RED (REDundant coding).https://webrtcglossary.com/red/.
[2] RTP Payload for Redundant Audio Data.https://tools.ietf.org/html/rfc2198.

本文已收录到大话WebRTC专栏，更多精彩请访问《大话WebRTC》。

本作品采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可

文章评论

悠哉嗑瓜子

"RED封装的数据作为payload跟在RTP固定报头（FixedHeader）后面。"
请教博主，如果rtp包存在扩展头，那么扩展头是跟在red数据之后？

2021年3月17日

Jeff

@悠哉嗑瓜子 “作为payload”，有扩展时，你觉得payload怎么放？我顺便完善下文章。

2021年3月17日

回复

应该是fix header + header extention + payload，主要是看到了“跟在RTP固定报头（FixedHeader）后面”这个描述，所以有次一问

2021年3月18日

MarkCao

@悠哉嗑瓜子谢谢你的提问和解答，对我有帮助！

2021年6月7日

回复

5555

webrt中开启ulp fec 交互的offer和answer 具体示例能贴下么，最近在研究fec，打算在媒体服务上加上这个，大佬赐教下

2021年11月30日

dever

大佬，使用ulpfec时，那原始rtp的seq经过fec编码后，seq变成什么样了？

2021年12月1日

Jeff

@dever FEC包以及媒体包RTP序列号是连续的，共享同一个序列号空间，ULPFEC编码后的冗余包附在原来视频帧RTP包后，序列号在原始媒体包序列号后连续递增

2022年1月26日

回复
- huowa222
  
  @Jeff 接收端怎么处理包序号的跳跃？FEC ：正常媒体包， pk1 pk2 pkg3 fec4 pk5 pk6 pk7 fec8， fec 的包序号这样。接收端解码的时候，pk number 重新排序吗？ pk1 pk2 pkg3 pk4 pk5 pk6
  
  2022年11月15日
  
  回复

chris

如何知道RTP header后面接着的是RED Head 而不是其他payload数据呢？

2022年1月12日

huowa222

@chris rtp header 有 pt 字段

2022年11月15日

回复