WebRTC研究：FEC之RED封装

首先看下webrtcglossary中的定义。

RED stands for REDundant coding and it is a RTP payload format defined in RFC 2198 for encoding redundant audio or video data.

The primary motivation of sending redundant data is to be able to recover packets lost under lossy network conditions. If a packet is lost then the missing information may be reconstructed at the receiver from the redundant data that arrives in the following packet(s).

The use of RED is negotiated in the SDP as an additional payload type and when used the audio/video RTP packets are packaged using RED format with a 6 bytes header, before the primary and secondary payloads conveying the actual audio/video packet and the redundant information.

RED封装简介

RED指冗余编码，并且依据RFC 2198定义了一种RTP有效载荷格式，用于携带冗余编码的音视频数据。RED封装的数据作为payload跟在RTP报头后面。RED封装的数据有自己的Header。在一个RTP包中，会包含多个RED Header，指定了后面携带的编码数据信息。这些RED Header依次跟在RTP报头后面，在这些RED Header后就是各个RED Header对应的冗余编码数据。

RED Header结构如下：

0                   1                    2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3  4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|   block PT  |  timestamp offset         |   block length    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

F： 1 bit长度，说明后面是否还有其他RED Header跟着。为1说明还有其他RED Header跟着，为0说明这是最后一个RED Header
blcok PT：7 bits长度，表示该RED Header对应的冗余编码类型
timeoffset： 14 bits长度，表示无符号长度时间戳偏移，该偏移是相对于RTP Header的时间戳。用无符号长度做偏移意味着冗余编码数据必须在发送完原始数据后才能发送
block length: 10 bits长度，表示该RED Header对应编码数据块长度，该长度包含RED Header字段

对于RTP包中最后一个RED Header，可以忽略block length以及timestamp。因为它们可以从RTP Fixed Header以及整个RTP包长度计算得到。最后一个RED Header结构如下：

0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|0|   Block PT  |
+-+-+-+-+-+-+-+-+

0 1 2 3 4 5 6 7

+-+-+-+-+-+-+-+-+

|0| Block PT |

+-+-+-+-+-+-+-+-+

其中F-bit位始终为0。

如下是一个完整RTP包示例，包含两个编码数据块：DVI4编码块，以及一个LPC冗余编码块。可以看到有两个RED Header跟在RTP Fixed Header后，第一个RED header F-bit为1，第二个RED Header由于是最后一个，所以F-bit为0，不包含时间戳偏移以及数据块长度信息。在RED Headrs后就是对应的编码数据块。

0                   1                    2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3  4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC=0  |M|      PT     |   sequence number of primary  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              timestamp  of primary encoding                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           synchronization source (SSRC) identifier            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| block PT=7  |  timestamp offset         |   block length    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| block PT=5  |                                               |
+-+-+-+-+-+-+-+-+                                               +
|                                                               |
+                LPC encoded redundant data (PT=7)              +
|                (14 bytes)                                     |
+                                               +---------------+
|                                               |               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                DVI4 encoded primary data (PT=5)               |
+                (84 bytes, not to scale)                       +
/                                                               /
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                               +---------------+
|                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|V=2|P|X| CC=0 |M| PT | sequence number of primary |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| timestamp of primary encoding |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| synchronization source (SSRC) identifier |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|0| block PT=5 | |

+-+-+-+-+-+-+-+-+ +

| |

+ LPC encoded redundant data (PT=7) +

| (14 bytes) |

+ +---------------+

| | |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +

| |

+ +

| |

+ +

| |

+ +

| DVI4 encoded primary data (PT=5) |

+ (84 bytes, not to scale) +

/ /

+ +

| |

+ +

| |

+ +---------------+

| |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

视频RED封装代码

这里我们以视频FEC为例，在WebRTC中音频的带外FEC也用到了RED编码。

UlpfecGenerator::GetUlpfecPacketsAsRed

WebRTC视频相关RED封装代码位于UlpfecGenerator类中。RED封装主要配合UlpFEC使用。这里我们看下UlpFEC编码数据封装为RED包的过程:

std::vector<std::unique_ptr<RedPacket>> UlpfecGenerator::GetUlpfecPacketsAsRed(
int red_payload_type,
int ulpfec_payload_type,
uint16_t first_seq_num) {
std::vector<std::unique_ptr<RedPacket>> red_packets;
red_packets.reserve(generated_fec_packets_.size());
RTC_DCHECK(!media_packets_.empty());
ForwardErrorCorrection::Packet* last_media_packet =
media_packets_.back().get();
uint16_t seq_num = first_seq_num;
for (const auto* fec_packet : generated_fec_packets_) {
// Wrap FEC packet (including FEC headers) in a RED packet. Since the
// FEC packets in |generated_fec_packets_| don't have RTP headers, we
// reuse the header from the last media packet.
RTC_DCHECK_GT(last_media_packet_rtp_header_length_, 0);
std::unique_ptr<RedPacket> red_packet(
new RedPacket(last_media_packet_rtp_header_length_ +
kRedForFecHeaderLength + fec_packet->data.size()));
// 创建报头
red_packet->CreateHeader(last_media_packet->data.data(),
last_media_packet_rtp_header_length_,
red_payload_type, ulpfec_payload_type);
red_packet->SetSeqNum(seq_num++);
red_packet->ClearMarkerBit();
// 设置UlpFEC编码数据
red_packet->AssignPayload(fec_packet->data.data(), fec_packet->data.size());
red_packets.push_back(std::move(red_packet));
}
ResetState();
return red_packets;
}

std::vector<std::unique_ptr<RedPacket>> UlpfecGenerator::GetUlpfecPacketsAsRed(

int red_payload_type,

int ulpfec_payload_type,

uint16_t first_seq_num) {

std::vector<std::unique_ptr<RedPacket>> red_packets;

red_packets.reserve(generated_fec_packets_.size());

RTC_DCHECK(!media_packets_.empty());

ForwardErrorCorrection::Packet* last_media_packet =

media_packets_.back().get();

uint16_t seq_num = first_seq_num;

for (const auto* fec_packet : generated_fec_packets_) {

// Wrap FEC packet (including FEC headers) in a RED packet. Since the

// FEC packets in |generated_fec_packets_| don't have RTP headers, we

// reuse the header from the last media packet.

RTC_DCHECK_GT(last_media_packet_rtp_header_length_, 0);

std::unique_ptr<RedPacket> red_packet(

new RedPacket(last_media_packet_rtp_header_length_ +

kRedForFecHeaderLength + fec_packet->data.size()));

// 创建报头

red_packet->CreateHeader(last_media_packet->data.data(),

last_media_packet_rtp_header_length_,

red_payload_type, ulpfec_payload_type);

red_packet->SetSeqNum(seq_num++);

red_packet->ClearMarkerBit();

// 设置UlpFEC编码数据

red_packet->AssignPayload(fec_packet->data.data(), fec_packet->data.size());

red_packets.push_back(std::move(red_packet));

}

ResetState();

return red_packets;

}

RedPacket::CreateHeader

对于视频，WebRTC中RED HeaderF-bit总是为0，不包含时间戳偏移等信息，也就是说一个RTP包只包含一个RED header：

void RedPacket::CreateHeader(const uint8_t* rtp_header,
size_t header_length,
int red_payload_type,
int payload_type) {
RTC_DCHECK_LE(header_length + kRedForFecHeaderLength, length_);
memcpy(data_.get(), rtp_header, header_length);
// Replace payload type.
data_[1] &= 0x80;
data_[1] += red_payload_type;
// Add RED header
// f-bit always 0
data_[header_length] = static_cast<uint8_t>(payload_type);
header_length_ = header_length + kRedForFecHeaderLength;
}

void RedPacket::CreateHeader(const uint8_t* rtp_header,

size_t header_length,

int red_payload_type,

int payload_type) {

RTC_DCHECK_LE(header_length + kRedForFecHeaderLength, length_);

memcpy(data_.get(), rtp_header, header_length);

// Replace payload type.

data_[1] &= 0x80;

data_[1] += red_payload_type;

// Add RED header

// f-bit always 0

data_[header_length] = static_cast<uint8_t>(payload_type);

header_length_ = header_length + kRedForFecHeaderLength;

}

视频RED解析代码

相关处理位于UlpfecReceiverImpl类中，我们看下主要的代码：

bool UlpfecReceiverImpl::AddReceivedRedPacket(const RtpPacket& rtp_packet,
uint8_t ulpfec_payload_type) {
// Remove RED header of incoming packet and store as a virtual RTP packet.
auto received_packet =
std::make_unique<ForwardErrorCorrection::ReceivedPacket>();
received_packet->pkt = new ForwardErrorCorrection::Packet();
// Get payload type from RED header and sequence number from RTP header.
uint8_t payload_type = rtp_packet.payload()[0] & 0x7f;
received_packet->is_fec = payload_type == ulpfec_payload_type;
received_packet->ssrc = rtp_packet.Ssrc();
received_packet->seq_num = rtp_packet.SequenceNumber();
// 如前面所说，WebRTC中一个视频RTP包只包含一个RED Header
if (rtp_packet.payload()[0] & 0x80) {
// f bit set in RED header, i.e. there are more than one RED header blocks.
// WebRTC never generates multiple blocks in a RED packet for FEC.
RTC_LOG(LS_WARNING) << "More than 1 block in RED packet is not supported.";
return false;
}
++packet_counter_.num_packets;
packet_counter_.num_bytes += rtp_packet.size();
if (packet_counter_.first_packet_time_ms == -1) {
packet_counter_.first_packet_time_ms = rtc::TimeMillis();
}
// 获取RED Header后携带的FEC冗余编码数据
if (received_packet->is_fec) {
++packet_counter_.num_fec_packets;
// everything behind the RED header
received_packet->pkt->data =
rtp_packet.Buffer().Slice(rtp_packet.headers_size() + kRedHeaderLength,
rtp_packet.payload_size() - kRedHeaderLength);
} 
return true;
}