WebRTC研究：关键帧请求

WebRTC采用UDP传输流媒体数据，不可避免存在丢包情况。WebRTC主要采用FEC（Forward Error Correction，前向纠错）以及NACK（negative-acknowledge character，否定应答）对抗网络丢包。对于NACK，遇到丢包了才通知发送端重传对应数据包，但不是所有情况下某个包丢了就一定重传该包，有些场景下，重传该包会带来其它问题，例如增大延时，缓存过大，同时也可能发送端没有该数据包缓存，导致无法重传，此时会放弃重传该包。由于关键帧可以单独解码出图像，不参考前后视频帧，所以会采取请求关键帧这种更便捷的方式替代重传该数据包，使解码端能立刻刷新出新图像，避免丢包过多，长时间等待重传数据包导致的画面停顿问题，以及获取不到重传包导致后续数据解码花屏问题。除了丢包，在WebRTC还存在其他请求关键帧的场景。

关键帧请求场景

下面结合代码列举几种常见场景。

H264解码无sps,pps信息

解码H264时无法获取sps，pps，导致无法解码，此时就需要请求获取关键帧，在H264SpsPpsTracker中，相关处理代码如下：

H264SpsPpsTracker::PacketAction H264SpsPpsTracker::CopyAndFixBitstream(
VCMPacket* packet) {
bool append_sps_pps = false;
auto sps = sps_data_.end();
auto pps = pps_data_.end();
for (size_t i = 0; i < h264_header.nalus_length; ++i) {
const NaluInfo& nalu = h264_header.nalus[i];
switch (nalu.type) {
case H264::NaluType::kSps: {
sps_data_[nalu.sps_id].width = packet->width();
sps_data_[nalu.sps_id].height = packet->height();
break;
}
case H264::NaluType::kPps: {
pps_data_[nalu.pps_id].sps_id = nalu.sps_id;
break;
}
case H264::NaluType::kIdr: {
// If this is the first packet of an IDR, make sure we have the required
// SPS/PPS and also calculate how much extra space we need in the buffer
// to prepend the SPS/PPS to the bitstream with start codes.
if (video_header.is_first_packet_in_frame) {
if (nalu.pps_id == -1) {
RTC_LOG(LS_WARNING) << "No PPS id in IDR nalu.";
return kRequestKeyframe;
}
pps = pps_data_.find(nalu.pps_id);
if (pps == pps_data_.end()) {
RTC_LOG(LS_WARNING)
<< "No PPS with id << " << nalu.pps_id << " received";
return kRequestKeyframe;
}
sps = sps_data_.find(pps->second.sps_id);
if (sps == sps_data_.end()) {
RTC_LOG(LS_WARNING)
<< "No SPS with id << " << pps->second.sps_id << " received";
return kRequestKeyframe;
}
.....
}

H264SpsPpsTracker::PacketAction H264SpsPpsTracker::CopyAndFixBitstream(

VCMPacket* packet) {

bool append_sps_pps = false;

auto sps = sps_data_.end();

auto pps = pps_data_.end();

for (size_t i = 0; i < h264_header.nalus_length; ++i) {

const NaluInfo& nalu = h264_header.nalus[i];

switch (nalu.type) {

case H264::NaluType::kSps: {

sps_data_[nalu.sps_id].width = packet->width();

sps_data_[nalu.sps_id].height = packet->height();

break;

}

case H264::NaluType::kPps: {

pps_data_[nalu.pps_id].sps_id = nalu.sps_id;

break;

}

case H264::NaluType::kIdr: {

// If this is the first packet of an IDR, make sure we have the required

// SPS/PPS and also calculate how much extra space we need in the buffer

// to prepend the SPS/PPS to the bitstream with start codes.

if (video_header.is_first_packet_in_frame) {

if (nalu.pps_id == -1) {

RTC_LOG(LS_WARNING) << "No PPS id in IDR nalu.";

return kRequestKeyframe;

}

pps = pps_data_.find(nalu.pps_id);

if (pps == pps_data_.end()) {

RTC_LOG(LS_WARNING)

<< "No PPS with id << " << nalu.pps_id << " received";

return kRequestKeyframe;

}

sps = sps_data_.find(pps->second.sps_id);

if (sps == sps_data_.end()) {

RTC_LOG(LS_WARNING)

<< "No SPS with id << " << pps->second.sps_id << " received";

return kRequestKeyframe;

}

.....

}

丢失包过多

在非常高的丢包率情况下，丢失的包太多，若都一一重传，将造成延时增大（等帧数据完整了才会去解码渲染），此时新来的数据也只能一直缓存，所以jitterbuffer大小也会不断增大，此时不如直接请求发送一个关键帧来得实际，以前丢的那些包都不管了，由于关键帧可以单独解码，所以不会造成解码端花屏马赛克现象。但是由于前面那些视频帧都丢弃了，此时生成的关键帧会与之前播放的视频存在不连贯性，所以画面变化大时会有轻微卡顿现象，相当于跳帧了。NackModule中相关处理代码如下：

void NackModule::AddPacketsToNack(uint16_t seq_num_start,
uint16_t seq_num_end) {
// Remove old packets.
auto it = nack_list_.lower_bound(seq_num_end - kMaxPacketAge);
nack_list_.erase(nack_list_.begin(), it);
// If the nack list is too large, remove packets from the nack list until
// the latest first packet of a keyframe. If the list is still too large,
// clear it and request a keyframe.
uint16_t num_new_nacks = ForwardDiff(seq_num_start, seq_num_end);
if (nack_list_.size() + num_new_nacks > kMaxNackPackets) {
while (RemovePacketsUntilKeyFrame() &&
nack_list_.size() + num_new_nacks > kMaxNackPackets) {
}
if (nack_list_.size() + num_new_nacks > kMaxNackPackets) {
nack_list_.clear();
RTC_LOG(LS_WARNING) << "NACK list full, clearing NACK"
" list and requesting keyframe.";
keyframe_request_sender_->RequestKeyFrame();
return;
}
}
}

void NackModule::AddPacketsToNack(uint16_t seq_num_start,

uint16_t seq_num_end) {

// Remove old packets.

auto it = nack_list_.lower_bound(seq_num_end - kMaxPacketAge);

nack_list_.erase(nack_list_.begin(), it);

// If the nack list is too large, remove packets from the nack list until

// the latest first packet of a keyframe. If the list is still too large,

// clear it and request a keyframe.

uint16_t num_new_nacks = ForwardDiff(seq_num_start, seq_num_end);

if (nack_list_.size() + num_new_nacks > kMaxNackPackets) {

while (RemovePacketsUntilKeyFrame() &&

nack_list_.size() + num_new_nacks > kMaxNackPackets) {

}

if (nack_list_.size() + num_new_nacks > kMaxNackPackets) {

nack_list_.clear();

RTC_LOG(LS_WARNING) << "NACK list full, clearing NACK"

" list and requesting keyframe.";

keyframe_request_sender_->RequestKeyFrame();

return;

}

上面代码中，要重传的包数量nack_list_.size()在进行RemovePacketsUntilKeyFrame()操作后若还超过规定大小，就开始清空要重传的数据包列表：nack_list_.clear()，然后请求关键帧。

丢失包过旧

发送端默认缓存600个RTP包，如果丢失的包太旧，超出缓存范围，此时发送端就没有该数据包的缓存，就无法重传该包。在VCMJitterBuffer中，相关处理代码如下：

bool VCMJitterBuffer::UpdateNackList(uint16_t sequence_number) {
if (nack_mode_ == kNoNack) {
return true;
}
// Make sure we don't add packets which are already too old to be decoded.
if (!last_decoded_state_.in_initial_state()) {
latest_received_sequence_number_ = LatestSequenceNumber(
latest_received_sequence_number_, last_decoded_state_.sequence_num());
}
if (IsNewerSequenceNumber(sequence_number,
latest_received_sequence_number_)) {
// Push any missing sequence numbers to the NACK list.
for (uint16_t i = latest_received_sequence_number_ + 1;
IsNewerSequenceNumber(sequence_number, i); ++i) {
missing_sequence_numbers_.insert(missing_sequence_numbers_.end(), i);
}
if (TooLargeNackList() && !HandleTooLargeNackList()) {
RTC_LOG(LS_WARNING) << "Requesting key frame due to too large NACK list.";
return false;
}
if (MissingTooOldPacket(sequence_number) &&
!HandleTooOldPackets(sequence_number)) {
RTC_LOG(LS_WARNING)
<< "Requesting key frame due to missing too old packets";
return false;
}
} else {
missing_sequence_numbers_.erase(sequence_number);
}
return true;
}

bool VCMJitterBuffer::UpdateNackList(uint16_t sequence_number) {

if (nack_mode_ == kNoNack) {

return true;

}

// Make sure we don't add packets which are already too old to be decoded.

if (!last_decoded_state_.in_initial_state()) {

latest_received_sequence_number_ = LatestSequenceNumber(

latest_received_sequence_number_, last_decoded_state_.sequence_num());

}

if (IsNewerSequenceNumber(sequence_number,

latest_received_sequence_number_)) {

// Push any missing sequence numbers to the NACK list.

for (uint16_t i = latest_received_sequence_number_ + 1;

IsNewerSequenceNumber(sequence_number, i); ++i) {

missing_sequence_numbers_.insert(missing_sequence_numbers_.end(), i);

}

if (TooLargeNackList() && !HandleTooLargeNackList()) {

RTC_LOG(LS_WARNING) << "Requesting key frame due to too large NACK list.";

return false;

}

if (MissingTooOldPacket(sequence_number) &&

!HandleTooOldPackets(sequence_number)) {

RTC_LOG(LS_WARNING)

<< "Requesting key frame due to missing too old packets";

return false;

}

} else {

missing_sequence_numbers_.erase(sequence_number);

}

return true;

}

获取帧数据超时

要解码的帧数据存放在FrameBuffer中，解码器解码时如果超过规定时间一直无法从FrameBuffer中获取解码数据，此时就会请求关键帧。相关处理代码如下：

void VideoReceiveStream::HandleFrameBufferTimeout() {
int64_t now_ms = clock_->TimeInMilliseconds();
absl::optional<int64_t> last_packet_ms =
rtp_video_stream_receiver_.LastReceivedPacketMs();
// To avoid spamming keyframe requests for a stream that is not active we
// check if we have received a packet within the last 5 seconds.
bool stream_is_active = last_packet_ms && now_ms - *last_packet_ms < 5000;
if (!stream_is_active)
stats_proxy_.OnStreamInactive();
if (stream_is_active && !IsReceivingKeyFrame(now_ms) &&
(!config_.crypto_options.sframe.require_frame_encryption ||
rtp_video_stream_receiver_.IsDecryptable())) {
RTC_LOG(LS_WARNING) << "No decodable frame in " << GetWaitMs()
<< " ms, requesting keyframe.";
RequestKeyFrame(now_ms);
}
}

void VideoReceiveStream::HandleFrameBufferTimeout() {

int64_t now_ms = clock_->TimeInMilliseconds();

absl::optional<int64_t> last_packet_ms =

rtp_video_stream_receiver_.LastReceivedPacketMs();

// To avoid spamming keyframe requests for a stream that is not active we

// check if we have received a packet within the last 5 seconds.

bool stream_is_active = last_packet_ms && now_ms - *last_packet_ms < 5000;

if (!stream_is_active)

stats_proxy_.OnStreamInactive();

if (stream_is_active && !IsReceivingKeyFrame(now_ms) &&

(!config_.crypto_options.sframe.require_frame_encryption ||

rtp_video_stream_receiver_.IsDecryptable())) {

RTC_LOG(LS_WARNING) << "No decodable frame in " << GetWaitMs()

<< " ms, requesting keyframe.";

RequestKeyFrame(now_ms);

}

解码出错

当解码器返回解码失败，或者解码器返回请求关键帧结果时，需要请求关键帧。相关处理代码如下：

void VideoReceiveStream::HandleEncodedFrame(
std::unique_ptr<EncodedFrame> frame) {
int64_t now_ms = clock_->TimeInMilliseconds();
int decode_result = video_receiver_.Decode(frame.get());
if (decode_result == WEBRTC_VIDEO_CODEC_OK ||
decode_result == WEBRTC_VIDEO_CODEC_OK_REQUEST_KEYFRAME) {
keyframe_required_ = false;
frame_decoded_ = true;
rtp_video_stream_receiver_.FrameDecoded(frame->id.picture_id);
if (decode_result == WEBRTC_VIDEO_CODEC_OK_REQUEST_KEYFRAME)
RequestKeyFrame(now_ms);
} else if (!frame_decoded_ || !keyframe_required_ ||
(last_keyframe_request_ms_ + max_wait_for_keyframe_ms_ < now_ms)) {
keyframe_required_ = true;
// TODO(philipel): Remove this keyframe request when downstream project
//                 has been fixed.
RequestKeyFrame(now_ms);
}
}

void VideoReceiveStream::HandleEncodedFrame(

std::unique_ptr<EncodedFrame> frame) {

int64_t now_ms = clock_->TimeInMilliseconds();

int decode_result = video_receiver_.Decode(frame.get());

if (decode_result == WEBRTC_VIDEO_CODEC_OK ||

decode_result == WEBRTC_VIDEO_CODEC_OK_REQUEST_KEYFRAME) {

keyframe_required_ = false;

frame_decoded_ = true;

rtp_video_stream_receiver_.FrameDecoded(frame->id.picture_id);

if (decode_result == WEBRTC_VIDEO_CODEC_OK_REQUEST_KEYFRAME)

RequestKeyFrame(now_ms);

} else if (!frame_decoded_ || !keyframe_required_ ||

(last_keyframe_request_ms_ + max_wait_for_keyframe_ms_ < now_ms)) {

keyframe_required_ = true;

// TODO(philipel): Remove this keyframe request when downstream project

// has been fixed.

RequestKeyFrame(now_ms);

}

在上面我们列举了几种需要关键帧请求的情况，我们只需要规定好RTCP报文格式，就能通知编码发送端发送关键帧。关键帧请求RTCP报文格式比较简单，在RFC4585（RTP/AVPF）以及RFC5104（AVPF）规定了两种不同的关键帧请求报文格式：Picture Loss Indication (PLI)、Full Intra Request (FIR)。WebRTC中关键帧请求也只用到了这两种消息，相关代码如下：

int32_t ModuleRtpRtcpImpl::RequestKeyFrame() {
switch (key_frame_req_method_) {
case kKeyFrameReqPliRtcp:
return SendRTCP(kRtcpPli);
case kKeyFrameReqFirRtcp:
return SendRTCP(kRtcpFir);
}
return -1;
}

int32_t ModuleRtpRtcpImpl::RequestKeyFrame() {

switch (key_frame_req_method_) {

case kKeyFrameReqPliRtcp:

return SendRTCP(kRtcpPli);

case kKeyFrameReqFirRtcp:

return SendRTCP(kRtcpFir);

}

return -1;

}

Picture Loss Indication (PLI)

在RFC4585中定义，属于RTCP反馈消息中的一种。RTCP反馈消息数据包格式按如下规定：

// RFC 4585: Feedback format.
Common packet format:
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|   FMT   |       PT      |          length               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  SSRC of packet sender                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  SSRC of media source                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:            Feedback Control Information (FCI)                 :
:                                                               :

// RFC 4585: Feedback format.

Common packet format:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|V=2|P| FMT | PT | length |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| SSRC of packet sender |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| SSRC of media source |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

: Feedback Control Information (FCI) :

: :

其中PT字段按如下规定：

Name   | Value | Brief Description
----------+-------+------------------------------------
RTPFB  |  205  | Transport layer FB message
PSFB   |  206  | Payload-specific FB message

Name | Value | Brief Description

----------+-------+------------------------------------

RTPFB | 205 | Transport layer FB message

PSFB | 206 | Payload-specific FB message

对于PLI，由于只需要通知发送关键帧，无需携带其他消息，所以FCI部分为空。对于FMT规定为1，PT规定为PSFB。

在WebRTC源码中，PLI相关解析封装代码位于webrtc::Pli中。相关代码如下：

// Picture loss indication (PLI) (RFC 4585).
// FCI: no feedback control information.
bool Pli::Parse(const CommonHeader& packet) {
RTC_DCHECK_EQ(packet.type(), kPacketType);
RTC_DCHECK_EQ(packet.fmt(), kFeedbackMessageType);
if (packet.payload_size_bytes() < kCommonFeedbackLength) {
RTC_LOG(LS_WARNING) << "Packet is too small to be a valid PLI packet";
return false;
}
ParseCommonFeedback(packet.payload());
return true;
}
size_t Pli::BlockLength() const {
return kHeaderLength + kCommonFeedbackLength;
}
bool Pli::Create(uint8_t* packet,
size_t* index,
size_t max_length,
PacketReadyCallback callback) const {
while (*index + BlockLength() > max_length) {
if (!OnBufferFull(packet, index, callback))
return false;
}
CreateHeader(kFeedbackMessageType, kPacketType, HeaderLength(), packet,
index);
CreateCommonFeedback(packet + *index);
*index += kCommonFeedbackLength;
return true;
}

// Picture loss indication (PLI) (RFC 4585).

// FCI: no feedback control information.

bool Pli::Parse(const CommonHeader& packet) {

RTC_DCHECK_EQ(packet.type(), kPacketType);

RTC_DCHECK_EQ(packet.fmt(), kFeedbackMessageType);

if (packet.payload_size_bytes() < kCommonFeedbackLength) {

RTC_LOG(LS_WARNING) << "Packet is too small to be a valid PLI packet";

return false;

}

ParseCommonFeedback(packet.payload());

return true;

}

size_t Pli::BlockLength() const {

return kHeaderLength + kCommonFeedbackLength;

}

bool Pli::Create(uint8_t* packet,

size_t* index,

size_t max_length,

PacketReadyCallback callback) const {

while (*index + BlockLength() > max_length) {

if (!OnBufferFull(packet, index, callback))

return false;

}

CreateHeader(kFeedbackMessageType, kPacketType, HeaderLength(), packet,

index);

CreateCommonFeedback(packet + *index);

*index += kCommonFeedbackLength;

return true;

}

PLI消息用于解码端通知编码端我要解码的图像的编码数据丢失了。对于基于帧间预测的视频编码类型，编码端收到PLI消息就要知道视频数据丢失了，由于帧间预测需要基于前后完整的视频帧才能解码（例如H264中，存在B帧，需要参考前后帧才能解码），前面的数据丢失了，后面的视频帧不能正常解码出图像，此时编码端可以直接生成一个关键帧，然后发送给解码端。

Full Intra Request (FIR)

在RFC5104中定义。参照上一小节RTCP反馈消息数据包格式，对于FMT规定为4，PT规定为PSFB。由于FIR可用于通知多个编码发送端（例如多点视频会议情况），所以用到了FCI部分，填充多个发送端的ssrc信息。具体包格式如下：

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|   FMT   |       PT      |          length               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  SSRC of packet sender                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             SSRC of media source (unused) = 0                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:            Feedback Control Information (FCI)                 :
:                                                               :
// Full intra request (FIR) (RFC 5104).
// The Feedback Control Information (FCI) for the Full Intra Request
// consists of one or more FCI entries.
// FCI:
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              SSRC                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seq nr.       |    Reserved = 0                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|V=2|P| FMT | PT | length |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| SSRC of packet sender |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| SSRC of media source (unused) = 0 |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

: Feedback Control Information (FCI) :

: :

// Full intra request (FIR) (RFC 5104).

// The Feedback Control Information (FCI) for the Full Intra Request

// consists of one or more FCI entries.

// FCI:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| SSRC |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Seq nr. | Reserved = 0 |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

在WebRTC源码中，FIR相关解析封装代码位于webrtc::Fir中。相关处理代码就不贴出来了，类似PLI处理，除了FCI部分要填充一些信息。

当解码端需要刷新时，可以发送FIR消息给编码端，编码端此时发送关键帧，刷新解码端。这有点类似PLI消息，但是PLI消息是用于丢包情况下的通知，而FIR却不是，在有些非丢包情况下，FIR就要用到。举两个例子：
1）解码端需要切换到另一路不同视频时，由于需要新的解码参数，所以可通过发送FIR消息，通知编码端生成关键帧，获取新的解码参数，刷新视频解码器；
2）在视频会议中，新用户随机时刻加入，各个编码端发送的视频不一定都是关键帧，所以新用户不一定能正常解码。此时该新加入用户发送FIR消息，通知各个编码端给它发关键帧，获取关键帧后即可正常解码。

总结

本文主要介绍了几种关键帧请求场景，讲了AVPF中定义的两种关键帧请求消息，虽然这两种消息获取的结果一样，但是表达的意义却不一样，用于不同场景，使用时需要区分下。

This article is licensed with Creative Commons Attribution-NonCommercial-No Derivatives 4.0 International License

Comments

纽约贤贤

补充一种关键帧请求的情况：网络丢包，jitter buffer无法在timeout（200ms/3000ms）之前找到下一连续帧。https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/video_receive_stream2.cc;l=744-764;drc=dfe9cd6c29a4a1871928ba06c4459305769df2a4;bpv=1;bpt=1

2021年1月28日

Jeff

@纽约贤贤已更新

2021年2月28日

- sshsu
  
  @Jeff 额...，发现webrtc源码里面的对于pli和fir处理好像是一样的
  https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/source/rtcp_receiver.cc;l=1036?q=packet_information.packet_type_flags%20%26%20kRtcpPli&ss=chromium%2Fchromium%2Fsrc:third_party%2Fwebrtc%2F
  
  2021年6月4日
  
- sshsu
  
  @Jeff 引用错了
  
  2021年6月4日

sshsu

博主能不能详细的说一下发送端对pli的处理，请问发送发接收到pli之后处理是否和fir一样，把接下来的帧编码成IDR，让接收方解码器刷新参考队列，之前接收到的旧包可以丢弃了，如果发送端的处理方式是一样的-生成一个IDR，刷新GOP，虽然使用场景不一样，但是发送端处理方式完全一样的花，为什么还要分fir和pli两种请求？(发送端应该在处理上有区别才对)

还是说，假设接收端现在接收了 [I1, P1, P2, P4,P5] 这个时候发现P3丢了，接收端发PLI给发送端，然后发送端收到了PLI之后得知P3丢了生成一个关键帧(非IDR，只是I frame) P3_new, 接收端得到这个P3_new后，p4, p5就能成功解码？对编解码器研究的不深，但是好像没看到编码器有提供这样的接口--重新生成某个旧帧，而且也没必要将P3_new生成为I帧，她是P帧能过去， p4和p5就能解码了呀

2021年6月4日

博主你好，我想问一个问题，我看了以上客户端会发关键帧请求的一些场景，都是因为某个单独客户端的问题，比如说刚加入会议，或者网络差丢帧等原因，如果某个客户端网络一直很差不断的发关键帧请求，整个会议室不久爆炸？一般rtc服务器是如何处理这个问题的？

2021年6月11日

Jeff

@sshsu 假设发布端source，订阅端是多个sink：source->sinks，某个或多个sink异常或者同时入会导致大量关键帧请求，最简单方法就是在source那里操作，控制关键帧请求频率即可。对于网络差情况，做得好的话，会有网络自适应，例如码率自适应，SVC丢帧策略等，避免恶化。

2021年6月15日