spring boot集成javacv + websocket实现实时视频推流回放（延时1-2秒）

喜欢ヅ旅行 2023-07-01 14:57 5阅读 0赞

最近项目需要实时直播和回放，集成海康威视摄像头：**（适合少量用户，或者内部系统使用）**

<dependency>
                <groupId>org.bytedeco</groupId>
                <artifactId>javacv-platform</artifactId>
                <version>1.5.1</version>
            </dependency>

这里是利用javacv抓取rtsp地址的视频流，通过转换成图片使用websocket实时推送

首先了解rtsp地址，这里举例就用海康威视的规则

通道号下面详细介绍
    #  单播
            rtsp://{用户名}:{密码}@{IP地址}:{端口号}/Streaming/Channels/{通道号}?transportmode=unicast
    #  多播
            rtsp://{用户名}:{密码}@{IP地址}:{端口号}/Streaming/Channels/{通道号}?transportmode=multicast
    #  分时获取
            rtsp://{用户名}:{密码}@{IP地址}:{端口号}/Streaming/tracks/{通道号}?starttime=20191008t063812z&endtime=20191008t064816z
    
    上面地址获取不到时可以试试下面的其他版本地址：
            rtsp://{用户名}:{密码}@{IP地址}:{端口号}/Streaming/Unicast/channels/{通道号}
            rtsp://{用户名}:{密码}@{IP地址}:{端口号}/h264/{通道号}/main/av_stream

这里通道号分类是IP通道号和模拟通道号，如果是12年以前的老机器，地址建议使用最后一个获取实时视频

可以使用官方提供的java demo 来查看，测试是否能获取视频

这里有打包好的jar包和环境可以直接运行jar包使用，**解压后将里面内容放置在System32下，cmd执行jar**

地址 [百度网盘 请输入提取码][Link 1]  密码 67v1

下图就是IP通道号 若只是Camera20 则为模拟通道号

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIzOTQ1Njg1_size_16_color_FFFFFF_t_70][]

**通道号可以参考这里**[最新海康摄像机、NVR、流媒体服务器、回放取流RTSP地址规则说明 - github.com/starRTC - 博客园][NVR_RTSP_ - github.com_starRTC -]

接下来就是java代码:

**首先yml配置动态RTSP地址**

myconfig:
      rtsp: rtsp://%s:%s@%s:%s/Streaming/Channels/%s01
      replay-rtsp: rtsp://%s:%s@%s:%s/Streaming/tracks/%s01?starttime=%s&endtime=%s

**媒体工具类：**

@Slf4j
    @Component
    public class MediaUtils {
        /**
         * 直播摄像机id集合，避免重复拉流
         */
        private static Set<Long> liveSet = new ConcurrentSet<>();
        /**
         * 用于构造回放rtsp地址
         */
        @Value("${myconfig.replay-rtsp}")
        private String rtspReplayPattern;
        /**
         * 视频帧率
         */
        public static int frameRate = 15;
        /**
         * 视频宽度
         */
        public static int frameWidth = 480;
        /**
         * 视频高度
         */
        public static int frameHeight = 270;
    
        /**
         * 摄像机直播
         * @param rtsp 摄像机直播地址
         * @param cameraName 摄像机名称
         * @param cameraId 摄像机id
         * @throws Exception e
         */
        @Async
        public void live(String rtsp, String cameraName, Long cameraId) throws Exception {
            if (liveSet.contains(cameraId)) {
                return;
            }
            liveSet.add(cameraId);
            FFmpegFrameGrabber grabber = createGrabber(rtsp);
            startCameraPush(grabber, cameraName, cameraId);
        }
    
    
        /**
         * 构造视频抓取器
         * @param rtsp 拉流地址
         * @return
         */
        public FFmpegFrameGrabber createGrabber(String rtsp) {
            // 获取视频源
            FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(rtsp);
            grabber.setOption("rtsp_transport","tcp");
            //设置帧率
            grabber.setFrameRate(frameRate);
            //设置获取的视频宽度
            grabber.setImageWidth(frameWidth);
            //设置获取的视频高度
            grabber.setImageHeight(frameHeight);
            //设置视频bit率
            grabber.setVideoBitrate(2000000);
            return grabber;
        }
    
        /**
         * 推送图片（摄像机直播）
         * @param grabber
         * @throws Exception
         */
        @Async
        public void startCameraPush(FFmpegFrameGrabber grabber, String cameraName, Long cameraId) throws Exception {
            Java2DFrameConverter java2DFrameConverter = new Java2DFrameConverter();
            try {
                grabber.start();
                int i = 1;
                while (liveSet.contains(cameraId)) {
                    Frame frame = grabber.grabImage();
                    if (null == frame) {
                        continue;
                    }
                    BufferedImage bufferedImage = java2DFrameConverter.getBufferedImage(frame);
    
                    byte[] bytes = imageToBytes(bufferedImage, "jpg");
    
                    //使用websocket发送图片数据
                    LiveWebsocket.sendImage(ByteBuffer.wrap(bytes), cameraId);
                }
            } finally {
                if (grabber != null) {
                    grabber.stop();
                }
            }
        }
    
        /**
         * 图片转字节数组
         * @param bImage 图片数据
         * @param format 格式
         * @return 图片字节码
         */
        private byte[] imageToBytes(BufferedImage bImage, String format) {
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            try {
                ImageIO.write(bImage, format, out);
            } catch (IOException e) {
                e.printStackTrace();
            }
            return out.toByteArray();
        }
    
        /**
         * 回放视频播放超期检查
         * @param userId 用户id
         * @return
         */
        private boolean replayOverTime(Integer userId) {
            if (replayMap.containsKey(userId)) {
                Long updateTime = replayMap.get(userId);
                if (updateTime != null) {
                    if (System.currentTimeMillis() - updateTime < 10000) {
                        return false;
                    }
                }
            }
            return true;
        }
    
        /**
         * 构造监控回放查询字段
         * @param date 时间
         * @param start
         * @return
         */
        private String formatPullTime(Date date, boolean start) {
            Calendar calendar = Calendar.getInstance();
            if (date != null) {
                calendar.setTime(date);
            }
            if (start) {
                calendar.add(Calendar.SECOND, -10);
            } else {
                calendar.add(Calendar.SECOND, 10);
            }
            //海康威视取回放的时间格式
            SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd#HHmmss$");
            String ret = sdf.format(calendar.getTime());
            ret = ret.replace("#", "t");
            ret = ret.replace("$", "z");
            return ret;
        }
    
     /**
         * 回放视频播放超期检查
         * @param userId 用户id
         * @return
         */
        private boolean replayOverTime(Integer userId) {
            if (replayMap.containsKey(userId)) {
                Long updateTime = replayMap.get(userId);
                if (updateTime != null) {
                    if (System.currentTimeMillis() - updateTime < 10000) {
                        return false;
                    }
                }
            }
            return true;
        }
     /**
         * 监控回放
         * @param userId 用户id  websocket 用户编号，作为心跳标识检测心跳
         * @param startDate 起始时间
         * @param endDate 结束时间
         * @param channel 通道号
         * @param username 用户名
         * @param password 密码
         * @param ip
         * @param port
         * @throws Exception e
         */
        @Async
        public void replayVideo(Integer userId, Date startDate, Date endDate,
                                Integer channel, String username, String password, String ip, Integer port) throws Exception {
            Java2DFrameConverter java2DFrameConverter = new Java2DFrameConverter();
                FFmpegFrameGrabber grabber = null;
                try {
                    if (grabber != null) {
                        grabber.stop();
                    }
                    if (channel != null) {
                        String st = formatPullTime(startDate, true);
                        String et = formatPullTime(endDate, false);
                        //构造rtsp回放流地址 username password ip port
                        String rtsp = String.format(
                                rtspReplayPattern,
                                username,
                                password,
                                ip,
                                port,
                                channel,
                                st,
                                et
                        );
                        if (grabber != null) {
                            grabber.stop();
                        }
                        grabber = createGrabber(rtsp);
                        grabber.setTimeout(10000);
                        grabber.start();
                        //心跳消失停止推流
                        while (!replayOverTime(userId)) {
                            Frame frame = grabber.grabImage();
                            if (null == frame) {
                                continue;
                            }
    
                            BufferedImage bufferedImage = java2DFrameConverter.getBufferedImage(frame);
    
                            byte[] bytes = imageToBytes(bufferedImage, "jpg");
                            ByteBuffer buffer = ByteBuffer.wrap(bytes);
    
                            //使用websocket发送图片数据
                            ReplayWebsocke.sendImage(buffer, userId);
                        }
                    }
                } catch (Exception e){
                    e.printStackTrace();
                } finally {
                    if (grabber != null) {
                        grabber.stop();
                    }
                }
        }
    }

**视频水印添加：**

@Component
    public class ImgMarker {
    
        /**
         * 视频水印图片
         */
        BufferedImage logoImg;
    
        private Font font;
        private Font font2;
        private FontDesignMetrics metrics;
        private FontDesignMetrics metrics2;
    
        @PostConstruct
        private void init() {
            // 加水印图片
            try {
                ImageIO.read(new File("图片地址"));
            } catch (IOException e) {
                e.printStackTrace();
            }
            font = new Font("黑体", Font.BOLD, 16);
            font2 = new Font("黑体", Font.BOLD, 24);
            metrics = FontDesignMetrics.getMetrics(font);
            metrics2 = FontDesignMetrics.getMetrics(font2);
        }
    
        /**
         * 加水印
         * @param bufImg 视频帧
         */
        public void mark(BufferedImage bufImg) {
            if (bufImg == null || logoImg == null) {
                return;
            }
            int width = bufImg.getWidth();
            int height = bufImg.getHeight();
            Graphics2D graphics = bufImg.createGraphics();
            graphics.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
            graphics.setComposite(AlphaComposite.getInstance(AlphaComposite.SRC_OVER));
            //设置图片背景
            graphics.drawImage(bufImg, 0, 0, width, height, null);
            //添加右上角水印
            graphics.drawImage(logoImg, width - 130, 8, 121, 64, null);
        }
    
        /**
         *
         * @param bufImg 视频帧
         */
        public void markTag(BufferedImage bufImg, String msg, int videoWidth) {
            int width = bufImg.getWidth();
            int height = bufImg.getHeight();
            Graphics2D graphics = bufImg.createGraphics();
            graphics.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
            graphics.setComposite(AlphaComposite.getInstance(AlphaComposite.SRC_OVER));
            //设置图片背景
            graphics.drawImage(bufImg, 0, 0, width, height, null);
            //设置由上方标签号
            graphics.setColor(Color.orange);
            if (videoWidth <= 400) {
                graphics.setFont(font2);
                graphics.drawString(msg,  width - metrics2.stringWidth(tagId) - 24, metrics2.getAscent());
            } else {
                graphics.setFont(font);
                graphics.drawString(msg,  width - metrics.stringWidth(msg) - 12, metrics.getAscent());
            }
            graphics.dispose();
        }
    
    }

至此可以调用了(还有其他方式是**直接调用SDK和推送视频流媒体服务器**，后续文章更新)

websocket 涉及到高并发阻塞情况 后续更新，建议使用**netty**

[Link 1]: https://pan.baidu.com/s/1QxELvKLmgqjeKu6NKUGP0A
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIzOTQ1Njg1_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/2020012310293085.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIzOTQ1Njg1,size_16,color_FFFFFF,t_70
[NVR_RTSP_ - github.com_starRTC -]: https://www.cnblogs.com/elesos/p/9881690.html