ffmpeg推流本地MP4 h264视频文件到rtsp服务器，rtsp客户端NVDECODE无法解码

谁践踏了优雅 2022-10-11 12:29 1476阅读 0赞

问题描述：  
最近在音视频遇到一个问题，因为需要，必须采用英伟达的NVDEC解码接受到的rtsp流，刚开始从摄像头拉流，解码非常正常，后来摄像头没了，就采用ffmpeg+ZLMediaKit进行rtsp推流，然后拉流解码，但是但是，结果突然就无法解码了。

解决：  
首先想到的第一个是推的流有问题，结果用VLC又能打开。  
第二个想到的是bsf的h264\_mp4toannexb转换有误，  
参考：https://github.com/gongluck/AnalysisAVP\#%E6%B5%81%E5%AA%92%E4%BD%93%E5%8D%8F%E8%AE%AE  
将bsf转换后的流打印出来，发现也没有问题，有startcodec ，但是惊奇的发现关键帧前面没有sps，pps。果然问题在这里，那么我们在将其送入到解码器之前，在关键帧之前加入sps和pps就可以了。

关于这个问题网上已经有人给了解决方案：  
从mp4,flv文件中解析出h264和aac,送解码器解码失败  
https://www.cnblogs.com/lihaiping/p/5285166.html

然后将摄像头找回来做同样的事情，用分析工具一览：  
这个工具非常好，可以从网上找到如下：  
可以发现摄像头的h264流是自带sps,pps,sei的。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM2MDcwNw_size_16_color_FFFFFF_t_70]  
特别注意67 sps 68 pps 106 sei 61

但是如何将这些信息补充上去呢，其实前面说了ffmpeg提供了bit stram filter函数：

av_bsf_get_by_name("h264_mp4toannexb")
    av_bsf_alloc（）
    av_bsf_init（）
    av_bsf_send_packet（）
    av_bsf_receive_packet（）

这组函数通过一个过滤器h264\_mp4toannexb\_filter，将AVCodecParameters中extradata中的数据转换为sps,pps,sei后补充上去。  
补充上去之后，英伟达解码器就可以正常解码了。  
参考：https://www.cnblogs.com/nsnow/p/3862709.html

那么我这里也是正常加载了过滤器h264\_mp4toannexb，为什么还是没有sps，pps，sei呢？继续查找原因：  
原因是推流过来的rtsp流已经进行过h264\_mp4toannexb转换了，所以bsf过滤器失效了，但是流中又没有在IDR帧中插入sps pps 等信息，所以手动插入：

auto nbsfRet = av_bsf_receive_packet(m_bsfc, &m_pktFiltered);
    //在其后边加入
    appendPPSandSPS(m_pktFiltered.data, m_pktFiltered.size,
                        m_bsfc->par_in->extradata, m_bsfc->par_in->extradata_size);
    //m_bsfc->par_in->extradata已经就是pps sps了
    //0 00 00 01 67 64 00 32 AC 2C 6A 80 A0 02 D6 9B 80 80 80 A0 00 00 E1 00 00 2B F2 00 80 00 00 00 01 68 EE 3C B0
                        
    bool appendPPSandSPS(uint8_t *src, unsigned int len1,
                                       uint8_t *inf, unsigned int len2) { 
      bool ret = false;
      uint8_t *hdr = nullptr;
      if ((len1 > 5 && src[0] == 0 && src[1] == 0 && src[2] == 0 && src[3] == 1) &&
          (len2 > 5 && inf[0] == 0 && inf[1] == 0 && inf[2] == 0 && inf[3] == 67)) { 
        src = (uint8_t *)realloc(src, len1 + len2);
        hdr = src;
        memmove(src + len2, src, len1);
        memcpy(hdr, inf, len2);
        ret = true;
      }
      return ret;
    }

这里说一下，m\_bsfc->par\_in这个是AVCodecParameters，这个是输入流的解码参数，在创建avformat\_open\_input（）中获取到的codecpar,它里面的extradata存放的就是sps,pps，其实它还有一个参数siede\_data可以参考：https://blog.csdn.net/bolitongyue/article/details/109053503

使用RTP传输H264的时候,需要用到sdp协议描述,其中有两项:Sequence Parameter Sets (SPS) 和Picture Parameter Set (PPS)需要用到,那么这两项从哪里获取呢?答案是从H264码流中获取.在H264码流中,都是以"0x00 0x00 0x01"或者"0x00 0x00 0x00 0x01"为开始码的,找到开始码之后,使用开始码之后的第一个字节的低5位判断是否为7(sps)或者8(pps), 及data\[4\] & 0x1f == 7 || data\[4\] & 0x1f == 8.然后对获取的nal去掉开始码之后进行**base64编码**,得到的信息就可以用于sdp.sps和pps需要用逗号分隔开来.

SDP中的H.264的SPS和PPS串，包含了初始化H.264解码器所需要的信息参数，包括编码所用的profile，level，图像的宽和高，deblock滤波器等。  
sdp中正常是包含sps和pps内容的，但是有时候sdp中没有sps和pps这些内容，这时候就要从流中获取。  
详细看这里：https://blog.csdn.net/Jody1989/article/details/46127561

H264码流中SPS PPS详解  
https://zhuanlan.zhihu.com/p/27896239

在这里看一个sdp会话

v=0
    o=- 0 0 IN IP4 192.168.1.112
    s=Stream-0
    i=N/A
    c=IN IP4 192.168.1.104
    t=0 0
    a=recvonly
    m=video 5006 RTP/AVP 96
    a=rtpmap:96 H264/90000
    a=fmtp:96 packetization-mode=1;profile-level-id=42c016;sprop-parameter-sets=Z0LAFqtAUB7QgAAAAwCAAAAPR4sXUA==,aM48gA==;
    a=control:trackID=1
    ————————————————
    版权声明：本文为CSDN博主「Devil_Lee」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
    原文链接：https://blog.csdn.net/devil__lee/article/details/9717471

其中主要的3个参数即为 profile-level-id sprop-parameter-sets

其中sprop-parameter-sets 包含sps和pps的信息，以逗号隔开。

那么这3个数值从哪里获取？有什么含义？

可以参考：http://www.cnblogs.com/skyseraph/archive/2012/04/01/2429384.html

通过这段时间的研究发现，sprop-parameter-sets 后面的数值为码流中sps和pps获取的值经过base64编码以后的数值，

profile-level-id为sps数值67以后的3个字节对应的16进制字符串

那么获取了sps和pps的数值以后就可以获取sdp文件的关键信息了

如果采用ffmpeg拉流，ffmpeg会解析sps和pps,然后存放在AVFormatContext中streams\[0\]中的codecpar中的extradata中。

有兴趣继续看一下过滤函数吧。  
在 av\_bsf\_init(bsfcCtx)的时候，有一个非常重要的参数：  
AVBSFContext \*bsfcCtx

typedef struct AVBSFContext { 
        /** * A class for logging and AVOptions */
        const AVClass *av_class;
    
        /** * The bitstream filter this context is an instance of. */
        const struct AVBitStreamFilter *filter;
    
        /** * Opaque libavcodec internal data. Must not be touched by the caller in any * way. */
        AVBSFInternal *internal;
    
        /** * Opaque filter-specific private data. If filter->priv_class is non-NULL, * this is an AVOptions-enabled struct. */
        void *priv_data;//H264BSFContext这个结构体
    
        /** * Parameters of the input stream. This field is allocated in * av_bsf_alloc(), it needs to be filled by the caller before * av_bsf_init(). */
        AVCodecParameters *par_in;//这个参数中存储的就是输入流的AVCodecParameters
    
        /** * Parameters of the output stream. This field is allocated in * av_bsf_alloc(), it is set by the filter in av_bsf_init(). */
        AVCodecParameters *par_out;
    
        /** * The timebase used for the timestamps of the input packets. Set by the * caller before av_bsf_init(). */
        AVRational time_base_in;
    
        /** * The timebase used for the timestamps of the output packets. Set by the * filter in av_bsf_init(). */
        AVRational time_base_out;
    } AVBSFContext;

我们看看它的创建：

const AVBitStreamFilter *bsf;
    bsf = av_bsf_get_by_name("h264_mp4toannexb");    if (!bsf) { 
            LOG_INFO("h264_mp4toannexb bitstream filter get failed");
            return false;
          }
    
    const AVBitStreamFilter *av_bsf_get_by_name(const char *name)
    { 
        const AVBitStreamFilter *f = NULL;
        void *i = 0;
    
        if (!name)
            return NULL;
    //遍历找到fileter,那么我们有多少过滤器呢
        while ((f = av_bsf_iterate(&i))) { //遍历返回一个bsf过滤器
            if (!strcmp(f->name, name))
                return f;
        }
    
        return NULL;
    }
    //
    //ffmpeg有这么多过滤器
    static const AVBitStreamFilter * const bitstream_filters[] = { 
        &ff_aac_adtstoasc_bsf,
        &ff_av1_frame_split_bsf,
        &ff_av1_metadata_bsf,
        &ff_chomp_bsf,
        &ff_dump_extradata_bsf,
        &ff_dca_core_bsf,
        &ff_eac3_core_bsf,
        &ff_extract_extradata_bsf,
        &ff_filter_units_bsf,
        &ff_h264_metadata_bsf,
        &ff_h264_mp4toannexb_bsf,
        &ff_h264_redundant_pps_bsf,
        &ff_hapqa_extract_bsf,
        &ff_hevc_metadata_bsf,
        &ff_hevc_mp4toannexb_bsf,
        &ff_imx_dump_header_bsf,
        &ff_mjpeg2jpeg_bsf,
        &ff_mjpega_dump_header_bsf,
        &ff_mp3_header_decompress_bsf,
        &ff_mpeg2_metadata_bsf,
        &ff_mpeg4_unpack_bframes_bsf,
        &ff_mov2textsub_bsf,
        &ff_noise_bsf,
        &ff_null_bsf,
        &ff_prores_metadata_bsf,
        &ff_remove_extradata_bsf,
        &ff_text2movsub_bsf,
        &ff_trace_headers_bsf,
        &ff_truehd_core_bsf,
        &ff_vp9_metadata_bsf,
        &ff_vp9_raw_reorder_bsf,
        &ff_vp9_superframe_bsf,
        &ff_vp9_superframe_split_bsf,
        NULL };
    
    /
    
    auto ret=  av_bsf_alloc(bsf, &m_bsfc);
    int av_bsf_alloc(const AVBitStreamFilter *filter, AVBSFContext **pctx)
    { 
        AVBSFContext *ctx;
        int ret;
    
        ctx = av_mallocz(sizeof(*ctx));
        if (!ctx)
            return AVERROR(ENOMEM);
    
        ctx->av_class = &bsf_class;
        //这里进行过滤器赋值
        ctx->filter   = filter;
    
        ctx->par_in  = avcodec_parameters_alloc();
        ctx->par_out = avcodec_parameters_alloc();
        if (!ctx->par_in || !ctx->par_out) { 
            ret = AVERROR(ENOMEM);
            goto fail;
        }
    
        ctx->internal = av_mallocz(sizeof(*ctx->internal));
        if (!ctx->internal) { 
            ret = AVERROR(ENOMEM);
            goto fail;
        }
    
        ctx->internal->buffer_pkt = av_packet_alloc();
        if (!ctx->internal->buffer_pkt) { 
            ret = AVERROR(ENOMEM);
            goto fail;
        }
    
        av_opt_set_defaults(ctx);
    
        /* allocate priv data and init private options */
        if (filter->priv_data_size) { 
        //这里存储的一般是option参数等
            ctx->priv_data = av_mallocz(filter->priv_data_size);
            if (!ctx->priv_data) { 
                ret = AVERROR(ENOMEM);
                goto fail;
            }
            //将filter中的priv_class赋值为ctx中的priv_data
            if (filter->priv_class) { 
                *(const AVClass **)ctx->priv_data = filter->priv_class;
                av_opt_set_defaults(ctx->priv_data);
            }
        }
    
        *pctx = ctx;
        return 0;
    fail:
        av_bsf_free(&ctx);
        return ret;
    }

接下来进行par\_in赋值

//这里主要进行的就是extradata的拷贝
    avcodec_parameters_copy(m_bsfc->par_in, m_param.codecParams);
    int avcodec_parameters_copy(AVCodecParameters *dst, const AVCodecParameters *src)
    { 
        codec_parameters_reset(dst);
        memcpy(dst, src, sizeof(*dst));
    
        dst->extradata      = NULL;
        dst->extradata_size = 0;
        if (src->extradata) { 
        //这里存储的就是sps以及pps
        //这些数据是ffmpeg从rtsp中的sdp中解析出来的，有时候sdp中没有sps和pps信息，那么说明流中包含了sps和pps
        //如果sdp中包含了sps和pps那么流中就不包含sps和pps
        //00 00 00 01 67 64 00 32 AC D9 40 28 00 B5 A6 C8 00 00 03 00 08 00 00 03 01 90 78 C1 8C B0 
        //00 00 00 01 68 EB E3 CB 22 C0 
            dst->extradata = av_mallocz(src->extradata_size + AV_INPUT_BUFFER_PADDING_SIZE);
            if (!dst->extradata)
                return AVERROR(ENOMEM);
            memcpy(dst->extradata, src->extradata, src->extradata_size);
            dst->extradata_size = src->extradata_size;
        }
    
        return 0;
    }

其中codecParams参数如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM2MDcwNw_size_16_color_FFFFFF_t_70 1]接下来进行av\_bsf\_init，这个时候bsfc的值如下，其中initernal是存放输入packet的。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM2MDcwNw_size_16_color_FFFFFF_t_70 2]

av_bsf_init(m_bsfc);
    //检查了一下filter中编解码id（AV_CODEC_ID_H264）是否支持，然后对par_out赋值
    int av_bsf_init(AVBSFContext *ctx)
    { 
        int ret, i;
    
        /* check that the codec is supported */
       /
       static void h264_mp4toannexb_flush(AVBSFContext *ctx)
    { 
        H264BSFContext *s = ctx->priv_data;
    
        s->idr_sps_seen = 0;
        s->idr_pps_seen = 0;
        s->new_idr      = s->extradata_parsed;
    }
    
    static const enum AVCodecID codec_ids[] = { 
        AV_CODEC_ID_H264, AV_CODEC_ID_NONE,
    };
    
    const AVBitStreamFilter ff_h264_mp4toannexb_bsf = { 
        .name           = "h264_mp4toannexb",
        .priv_data_size = sizeof(H264BSFContext),
        .init           = h264_mp4toannexb_init,
        .filter         = h264_mp4toannexb_filter,
        .flush          = h264_mp4toannexb_flush,
        .codec_ids      = codec_ids,
    };
      
        if (ctx->filter->codec_ids) { 
        //查看输入流的code_id是否被filter所支持
            for (i = 0; ctx->filter->codec_ids[i] != AV_CODEC_ID_NONE; i++)
                if (ctx->par_in->codec_id == ctx->filter->codec_ids[i])
                    break;
             //如果遍历到了最后，那就说明不支持
            if (ctx->filter->codec_ids[i] == AV_CODEC_ID_NONE) { 
                const AVCodecDescriptor *desc = avcodec_descriptor_get(ctx->par_in->codec_id);
                av_log(ctx, AV_LOG_ERROR, "Codec '%s' (%d) is not supported by the "
                       "bitstream filter '%s'. Supported codecs are: ",
                       desc ? desc->name : "unknown", ctx->par_in->codec_id, ctx->filter->name);
                for (i = 0; ctx->filter->codec_ids[i] != AV_CODEC_ID_NONE; i++) { 
                    desc = avcodec_descriptor_get(ctx->filter->codec_ids[i]);
                    av_log(ctx, AV_LOG_ERROR, "%s (%d) ",
                           desc ? desc->name : "unknown", ctx->filter->codec_ids[i]);
                }
                av_log(ctx, AV_LOG_ERROR, "\n");
                return AVERROR(EINVAL);
            }
        }
    
        /* initialize output parameters to be the same as input * init below might overwrite that */
        ret = avcodec_parameters_copy(ctx->par_out, ctx->par_in);
        if (ret < 0)
            return ret;
    
        ctx->time_base_out = ctx->time_base_in;
    
        if (ctx->filter->init) { 
        //h264_mp4toannexb_init
            ret = ctx->filter->init(ctx);
            if (ret < 0)
                return ret;
        }
    
        return 0;
    }
    
    static int h264_mp4toannexb_init(AVBSFContext *ctx)
    { 
        H264BSFContext *s = ctx->priv_data;
        //这里将sps pps等数据大小拷贝一下
        int extra_size = ctx->par_in->extradata_size;
        int ret;
    
        /* retrieve sps and pps NAL units from extradata */
        //在这里流程中The input looks like it is Annex B already，也就是
        //输入的extra_data已经是Annex B了
        /
    #ifndef AV_RB24
    # define AV_RB24(x) \ ((((const uint8_t*)(x))[0] << 16) | \ (((const uint8_t*)(x))[1] << 8) | \ ((const uint8_t*)(x))[2])
    #endif
    
    #ifndef AV_RB32
    # define AV_RB32(x) \ (((uint32_t)((const uint8_t*)(x))[0] << 24) | \ (((const uint8_t*)(x))[1] << 16) | \ (((const uint8_t*)(x))[2] << 8) | \ ((const uint8_t*)(x))[3])
    #endif
        
        //这里在做什么 AV_RB24
        //0x00 00 01
        //0x00 00 00 00 01
        //查看是否是上述包头
        if (!extra_size                                               ||
            (extra_size >= 3 && AV_RB24(ctx->par_in->extradata) == 1) ||
            (extra_size >= 4 && AV_RB32(ctx->par_in->extradata) == 1)) { 
            av_log(ctx, AV_LOG_VERBOSE,
                   "The input looks like it is Annex B already\n");
        } else if (extra_size >= 6) { 
            ret = h264_extradata_to_annexb(ctx, AV_INPUT_BUFFER_PADDING_SIZE);
            if (ret < 0)
                return ret;
    
            s->length_size      = ret;
            s->new_idr          = 1;
            s->idr_sps_seen     = 0;
            s->idr_pps_seen     = 0;
            s->extradata_parsed = 1;
        } else { 
            av_log(ctx, AV_LOG_ERROR, "Invalid extradata size: %d\n", extra_size);
            return AVERROR_INVALIDDATA;
        }
    
        return 0;
    }

下面看一下h264\_extradata\_to\_annexb  
总的来说H264的码流的打包方式有两种,一种为annex-b byte stream format的格式，这个是绝大部分编码器的默认输出格式，就是每个帧的开头的3~4个字节是H264的start\_code,0x00000001或者0x000001。  
另一种是原始的NAL打包格式，就是开始的若干字节（1，2，4字节）是NAL的长度，而不是start\_code,此时必须借助某个全局的数据来获得编码器的profile,level,PPS,SPS等信息才可以解码。

我一直疑问为什么有些视频解码时显示格式是:H264，大部分又是：AVC1  
我在搜索编程资料时在微软的msdn上发现的：  
原文:http://msdn.microsoft.com/en-us/library/dd757808(v=vs.85).aspx  
FOURCC:AVC1 描述:H.264 bitstream without start codes.  
FOURCC:H264 描述:H.264 bitstream with start codes.  
mp4 文件中的h264 avc1格式介绍  
https://blog.csdn.net/haima1998/article/details/50426944/

下面这篇文章很详细的介绍了h264\_extradata\_to\_annexb的作用  
ffmpeg 从mp4上提取H264的nalu  
https://blog.csdn.net/gavinr/article/details/7183499

H264—MP4格式及在MP4文件中提取H264的SPS、PPS及码流  
https://www.cnblogs.com/skyseraph/archive/2012/04/01/2429384.html

使用FFmpeg提取MP4中的H264和AAC  
https://jiangdg.blog.csdn.net/article/details/102665541

static int h264_extradata_to_annexb(AVBSFContext *ctx, const int padding)
    { 
        H264BSFContext *s = ctx->priv_data;
        //
        typedef struct H264BSFContext { 
        int32_t  sps_offset;
        int32_t  pps_offset;
        uint8_t  length_size;
        uint8_t  new_idr;
        uint8_t  idr_sps_seen;
        uint8_t  idr_pps_seen;
        int      extradata_parsed;
    } H264BSFContext;
    //
        uint16_t unit_size;
        uint64_t total_size                 = 0;
        uint8_t *out                        = NULL, unit_nb, sps_done = 0,
                 sps_seen                   = 0, pps_seen = 0;
        const uint8_t *extradata            = ctx->par_in->extradata + 4;
        static const uint8_t nalu_header[4] = {  0, 0, 0, 1 };
        //0000 0011，因为nal包后5位是包的长度，这里计算出长度+1
        int length_size = (*extradata++ & 0x3) + 1; // retrieve length coded size
    
        s->sps_offset = s->pps_offset = -1;
    
        /* retrieve sps and pps unit(s) */
        //0x1f 0001 1111 下面这句就是提取 nal中的后5位
        unit_nb = *extradata++ & 0x1f; /* number of sps unit(s) */
        if (!unit_nb) { 
            goto pps;
        } else { 
            s->sps_offset = 0;
            sps_seen = 1;
        }
    
        while (unit_nb--) { 
            int err;
    
            unit_size   = AV_RB16(extradata);
            total_size += unit_size + 4;
            if (total_size > INT_MAX - padding) { 
                av_log(ctx, AV_LOG_ERROR,
                       "Too big extradata size, corrupted stream or invalid MP4/AVCC bitstream\n");
                av_free(out);
                return AVERROR(EINVAL);
            }
            if (extradata + 2 + unit_size > ctx->par_in->extradata + ctx->par_in->extradata_size) { 
                av_log(ctx, AV_LOG_ERROR, "Packet header is not contained in global extradata, "
                       "corrupted stream or invalid MP4/AVCC bitstream\n");
                av_free(out);
                return AVERROR(EINVAL);
            }
            if ((err = av_reallocp(&out, total_size + padding)) < 0)
                return err;
            memcpy(out + total_size - unit_size - 4, nalu_header, 4);
            memcpy(out + total_size - unit_size, extradata + 2, unit_size);
            extradata += 2 + unit_size;
    pps:
            if (!unit_nb && !sps_done++) { 
                unit_nb = *extradata++; /* number of pps unit(s) */
                if (unit_nb) { 
                    s->pps_offset = total_size;
                    pps_seen = 1;
                }
            }
        }
    
        if (out)
            memset(out + total_size, 0, padding);
    
        if (!sps_seen)
            av_log(ctx, AV_LOG_WARNING,
                   "Warning: SPS NALU missing or invalid. "
                   "The resulting stream may not play.\n");
    
        if (!pps_seen)
            av_log(ctx, AV_LOG_WARNING,
                   "Warning: PPS NALU missing or invalid. "
                   "The resulting stream may not play.\n");
    
        av_freep(&ctx->par_out->extradata);
        ctx->par_out->extradata      = out;
        ctx->par_out->extradata_size = total_size;
    
        return length_size;
    }

接着就是循环处理了

auto ret = av_bsf_send_packet(m_bsfc, inPacket);
    //这个函数的主要作用就是把pkt转到ctx->internal->buffer_pkt中
    int av_bsf_send_packet(AVBSFContext *ctx, AVPacket *pkt)
    { 
        int ret;
    
        if (!pkt || (!pkt->data && !pkt->side_data_elems)) { 
            ctx->internal->eof = 1;
            return 0;
        }
    
        if (ctx->internal->eof) { 
            av_log(ctx, AV_LOG_ERROR, "A non-NULL packet sent after an EOF.\n");
            return AVERROR(EINVAL);
        }
    
        if (ctx->internal->buffer_pkt->data ||
            ctx->internal->buffer_pkt->side_data_elems)
            return AVERROR(EAGAIN);
    
        ret = av_packet_make_refcounted(pkt);
        if (ret < 0)
            return ret;
        av_packet_move_ref(ctx->internal->buffer_pkt, pkt);
    
        return 0;
    }

auto nbsfRet = av_bsf_receive_packet(m_bsfc, &m_pktFiltered);
    int av_bsf_receive_packet(AVBSFContext *ctx, AVPacket *pkt)
    { 
        return ctx->filter->filter(ctx, pkt);
    }
    
    static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
    { 
        H264BSFContext *s = ctx->priv_data;
    
        AVPacket *in;
        uint8_t unit_type;
        int32_t nal_size;
        uint32_t cumul_size    = 0;
        const uint8_t *buf;
        const uint8_t *buf_end;
        int            buf_size;
        int ret = 0, i;
    //获取到pkt
        ret = ff_bsf_get_packet(ctx, &in);
        if (ret < 0)
            return ret;
    /ff_bsf_get_packet
    int ff_bsf_get_packet(AVBSFContext *ctx, AVPacket **pkt)
    { 
        AVBSFInternal *in = ctx->internal;
        AVPacket *tmp_pkt;
    
        if (in->eof)
            return AVERROR_EOF;
    
        if (!ctx->internal->buffer_pkt->data &&
            !ctx->internal->buffer_pkt->side_data_elems)
            return AVERROR(EAGAIN);
    
        tmp_pkt = av_packet_alloc();
        if (!tmp_pkt)
            return AVERROR(ENOMEM);
    
        *pkt = ctx->internal->buffer_pkt;
        ctx->internal->buffer_pkt = tmp_pkt;
    
        return 0;
    }
    /
    
        /* nothing to filter */
        //记得前面already annex B 这里就直接返回了
        if (!s->extradata_parsed) { 
            av_packet_move_ref(out, in);
            av_packet_free(&in);
            return 0;
        }
    
        buf      = in->data;
        buf_size = in->size;
        buf_end  = in->data + in->size;
    
        do { 
            ret= AVERROR(EINVAL);
            if (buf + s->length_size > buf_end)
                goto fail;
    
            for (nal_size = 0, i = 0; i<s->length_size; i++)
                nal_size = (nal_size << 8) | buf[i];
    
            buf += s->length_size;
            unit_type = *buf & 0x1f;
    
            if (nal_size > buf_end - buf || nal_size < 0)
                goto fail;
    
            if (unit_type == H264_NAL_SPS)
                s->idr_sps_seen = s->new_idr = 1;
            else if (unit_type == H264_NAL_PPS) { 
                s->idr_pps_seen = s->new_idr = 1;
                /* if SPS has not been seen yet, prepend the AVCC one to PPS */
                if (!s->idr_sps_seen) { 
                    if (s->sps_offset == -1)
                        av_log(ctx, AV_LOG_WARNING, "SPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                    else { 
                        if ((ret = alloc_and_copy(out,
                                             ctx->par_out->extradata + s->sps_offset,
                                             s->pps_offset != -1 ? s->pps_offset : ctx->par_out->extradata_size - s->sps_offset,
                                             buf, nal_size, 1)) < 0)
                            goto fail;
                        s->idr_sps_seen = 1;
                        goto next_nal;
                    }
                }
            }
    
            /* if this is a new IDR picture following an IDR picture, reset the idr flag. * Just check first_mb_in_slice to be 0 as this is the simplest solution. * This could be checking idr_pic_id instead, but would complexify the parsing. */
            if (!s->new_idr && unit_type == H264_NAL_IDR_SLICE && (buf[1] & 0x80))
                s->new_idr = 1;
    
            /* prepend only to the first type 5 NAL unit of an IDR picture, if no sps/pps are already present */
            if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && !s->idr_sps_seen && !s->idr_pps_seen) { 
            //关键帧拷贝的时候同sps pps一起拷贝
                if ((ret=alloc_and_copy(out,
                                   ctx->par_out->extradata, ctx->par_out->extradata_size,
                                   buf, nal_size, 1)) < 0)
                    goto fail;
                s->new_idr = 0;
            /* if only SPS has been seen, also insert PPS */
            } else if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && s->idr_sps_seen && !s->idr_pps_seen) { 
                if (s->pps_offset == -1) { 
                    av_log(ctx, AV_LOG_WARNING, "PPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                    if ((ret = alloc_and_copy(out, NULL, 0, buf, nal_size, 0)) < 0)
                        goto fail;
                } else if ((ret = alloc_and_copy(out,
                                            ctx->par_out->extradata + s->pps_offset, ctx->par_out->extradata_size - s->pps_offset,
                                            buf, nal_size, 1)) < 0)
                    goto fail;
            } else { 
                if ((ret=alloc_and_copy(out, NULL, 0, buf, nal_size, unit_type == H264_NAL_SPS || unit_type == H264_NAL_PPS)) < 0)
                    goto fail;
                if (!s->new_idr && unit_type == H264_NAL_SLICE) { 
                    s->new_idr = 1;
                    s->idr_sps_seen = 0;
                    s->idr_pps_seen = 0;
                }
            }
    
    next_nal:
            buf        += nal_size;
            cumul_size += nal_size + s->length_size;
        } while (cumul_size < buf_size);
    
        ret = av_packet_copy_props(out, in);
        if (ret < 0)
            goto fail;
    
    fail:
        if (ret < 0)
            av_packet_unref(out);
        av_packet_free(&in);
    
        return ret;
    }

typedef struct AVCodecParameters { 
        /** * General type of the encoded data. */
        enum AVMediaType codec_type;
        /** * Specific type of the encoded data (the codec used). */
        enum AVCodecID   codec_id;
        /** * Additional information about the codec (corresponds to the AVI FOURCC). */
        uint32_t         codec_tag;
    
        /** * Extra binary data needed for initializing the decoder, codec-dependent. * * Must be allocated with av_malloc() and will be freed by * avcodec_parameters_free(). The allocated size of extradata must be at * least extradata_size + AV_INPUT_BUFFER_PADDING_SIZE, with the padding * bytes zeroed. * 0 00 00 01 67 64 00 32 AC D9 40 28 00 B5 A6 C8 00 00 03 00 08 00 00 03 01 90 78 C1 8C B0 00 00 00 01 68 EB E3 CB 22 C0 */
        uint8_t *extradata;//这个参数中存储就是pps sps
        /** * Size of the extradata content in bytes. */
        int      extradata_size;//
    
        /** * - video: the pixel format, the value corresponds to enum AVPixelFormat. * - audio: the sample format, the value corresponds to enum AVSampleFormat. */
        int format;
    
        /** * The average bitrate of the encoded data (in bits per second). */
        int64_t bit_rate;
    
        /** * The number of bits per sample in the codedwords. * * This is basically the bitrate per sample. It is mandatory for a bunch of * formats to actually decode them. It's the number of bits for one sample in * the actual coded bitstream. * * This could be for example 4 for ADPCM * For PCM formats this matches bits_per_raw_sample * Can be 0 */
        int bits_per_coded_sample;
    
        /** * This is the number of valid bits in each output sample. If the * sample format has more bits, the least significant bits are additional * padding bits, which are always 0. Use right shifts to reduce the sample * to its actual size. For example, audio formats with 24 bit samples will * have bits_per_raw_sample set to 24, and format set to AV_SAMPLE_FMT_S32. * To get the original sample use "(int32_t)sample >> 8"." * * For ADPCM this might be 12 or 16 or similar * Can be 0 */
        int bits_per_raw_sample;
    
        /** * Codec-specific bitstream restrictions that the stream conforms to. */
        int profile;
        int level;
    
        /** * Video only. The dimensions of the video frame in pixels. */
        int width;
        int height;
    
        /** * Video only. The aspect ratio (width / height) which a single pixel * should have when displayed. * * When the aspect ratio is unknown / undefined, the numerator should be * set to 0 (the denominator may have any value). */
        AVRational sample_aspect_ratio;
    
        /** * Video only. The order of the fields in interlaced video. */
        enum AVFieldOrder                  field_order;
    
        /** * Video only. Additional colorspace characteristics. */
        enum AVColorRange                  color_range;
        enum AVColorPrimaries              color_primaries;
        enum AVColorTransferCharacteristic color_trc;
        enum AVColorSpace                  color_space;
        enum AVChromaLocation              chroma_location;
    
        /** * Video only. Number of delayed frames. */
        int video_delay;
    
        /** * Audio only. The channel layout bitmask. May be 0 if the channel layout is * unknown or unspecified, otherwise the number of bits set must be equal to * the channels field. */
        uint64_t channel_layout;
        /** * Audio only. The number of audio channels. */
        int      channels;
        /** * Audio only. The number of audio samples per second. */
        int      sample_rate;
        /** * Audio only. The number of bytes per coded audio frame, required by some * formats. * * Corresponds to nBlockAlign in WAVEFORMATEX. */
        int      block_align;
        /** * Audio only. Audio frame size, if known. Required by some formats to be static. */
        int      frame_size;
    
        /** * Audio only. The amount of padding (in samples) inserted by the encoder at * the beginning of the audio. I.e. this number of leading decoded samples * must be discarded by the caller to get the original audio without leading * padding. */
        int initial_padding;
        /** * Audio only. The amount of padding (in samples) appended by the encoder to * the end of the audio. I.e. this number of decoded samples must be * discarded by the caller from the end of the stream to get the original * audio without any trailing padding. */
        int trailing_padding;
        /** * Audio only. Number of samples to skip after a discontinuity. */
        int seek_preroll;
    } AVCodecParameters;

我们接着来分析：

static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
    { 
        H264BSFContext *s = ctx->priv_data;
    
        AVPacket *in;
        uint8_t unit_type;
        int32_t nal_size;
        uint32_t cumul_size    = 0;
        const uint8_t *buf;
        const uint8_t *buf_end;
        int            buf_size;
        int ret = 0, i;
    
        ret = ff_bsf_get_packet(ctx, &in);
        if (ret < 0)
            return ret;
    
        /* nothing to filter */
        if (!s->extradata_parsed) { 
            av_packet_move_ref(out, in);
            av_packet_free(&in);
            return 0;
        }
    
        buf      = in->data;
        buf_size = in->size;
        buf_end  = in->data + in->size;
    
        do { 
            ret= AVERROR(EINVAL);
            if (buf + s->length_size > buf_end)
                goto fail;
    
            for (nal_size = 0, i = 0; i<s->length_size; i++)
                nal_size = (nal_size << 8) | buf[i];
    
            buf += s->length_size;
            unit_type = *buf & 0x1f;
    
            if (nal_size > buf_end - buf || nal_size < 0)
                goto fail;
    
            if (unit_type == H264_NAL_SPS)
                s->idr_sps_seen = s->new_idr = 1;
            else if (unit_type == H264_NAL_PPS) { 
                s->idr_pps_seen = s->new_idr = 1;
                /* if SPS has not been seen yet, prepend the AVCC one to PPS */
                if (!s->idr_sps_seen) { 
                    if (s->sps_offset == -1)
                        av_log(ctx, AV_LOG_WARNING, "SPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                    else { 
                        if ((ret = alloc_and_copy(out,
                                             ctx->par_out->extradata + s->sps_offset,
                                             s->pps_offset != -1 ? s->pps_offset : ctx->par_out->extradata_size - s->sps_offset,
                                             buf, nal_size, 1)) < 0)
                            goto fail;
                        s->idr_sps_seen = 1;
                        goto next_nal;
                    }
                }
            }
    
            /* if this is a new IDR picture following an IDR picture, reset the idr flag. * Just check first_mb_in_slice to be 0 as this is the simplest solution. * This could be checking idr_pic_id instead, but would complexify the parsing. */
            if (!s->new_idr && unit_type == H264_NAL_IDR_SLICE && (buf[1] & 0x80))
                s->new_idr = 1;
    
            /* prepend only to the first type 5 NAL unit of an IDR picture, if no sps/pps are already present */
            if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && !s->idr_sps_seen && !s->idr_pps_seen) { 
                if ((ret=alloc_and_copy(out,
                                   ctx->par_out->extradata, ctx->par_out->extradata_size,
                                   buf, nal_size, 1)) < 0)
                    goto fail;
                s->new_idr = 0;
            /* if only SPS has been seen, also insert PPS */
            } else if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && s->idr_sps_seen && !s->idr_pps_seen) { 
                if (s->pps_offset == -1) { 
                    av_log(ctx, AV_LOG_WARNING, "PPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                    if ((ret = alloc_and_copy(out, NULL, 0, buf, nal_size, 0)) < 0)
                        goto fail;
                } else if ((ret = alloc_and_copy(out,
                                            ctx->par_out->extradata + s->pps_offset, ctx->par_out->extradata_size - s->pps_offset,
                                            buf, nal_size, 1)) < 0)
                    goto fail;
            } else { 
                if ((ret=alloc_and_copy(out, NULL, 0, buf, nal_size, unit_type == H264_NAL_SPS || unit_type == H264_NAL_PPS)) < 0)
                    goto fail;
                if (!s->new_idr && unit_type == H264_NAL_SLICE) { 
                    s->new_idr = 1;
                    s->idr_sps_seen = 0;
                    s->idr_pps_seen = 0;
                }
            }
    
    next_nal:
            buf        += nal_size;
            cumul_size += nal_size + s->length_size;
        } while (cumul_size < buf_size);
    
        ret = av_packet_copy_props(out, in);
        if (ret < 0)
            goto fail;
    
    fail:
        if (ret < 0)
            av_packet_unref(out);
        av_packet_free(&in);
    
        return ret;
    }

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM2MDcwNw_size_16_color_FFFFFF_t_70]: /images/20221005/862905069f3044a491014c735f6b6ff5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM2MDcwNw_size_16_color_FFFFFF_t_70 1]: /images/20221005/c4ffabc55f7447da8c749f60f95f7602.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzM2MDcwNw_size_16_color_FFFFFF_t_70 2]: /images/20221005/4050ad49d07a4af39e3a83bfe3e01ef1.png