【AR】使用OpenCV中的aruco模块实现增强现实

你的名字 2022-12-29 06:54 351阅读 0赞

# 1 ArUco marker #

ArUco marker是由S.Garrido-Jurado等人在2014年提出的，全称是Augmented Reality University of Cordoba，详见他们的论文《Automatic generation and detection of highly reliable fiducial markers under occlusion》。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc_size_16_color_FFFFFF_t_70_pic_center]

它类似于二维码，通过预先标定的相机参数和捕获到的aruco标记图像，就可以求解出marker与相机的相对坐标关系。实现增强现实大致包括以下几个步骤：

*  相机标定，包括相机的内参、畸变系数等；
 *  利用相机参数和预先设定好的marker参数（包括边长、角点等）求解相机3D坐标系到marker所在3D坐标系（z=0）的空间变换关系T；
 *  将T作用于虚拟物体计算其在相机成像平面的投影。

# 2 Opencv中实现一个简单的AR应用 #

## 2.1 编译aruco模块 ##

aruco在opencv的extra\_moduls（opencv\_contrib）里，需要额外编译才能使用。

*  （不推荐）编译全部opencv\_contrib并引用，参考[链接][Link 1]；
 *  （推荐）只编译某些模块，把这些模块从“opencv\_contrib-x.x.x/modules”拷贝到“opencv/modules”，然后执行opencv编译即可。

注意一定要下载与opencv版本相对应的opencv\_contrib，可从国内镜像（[链接][Link 2]）快速下载。

## 2.2 生成aruco marker ##

aruco字典中定义了不同大小的marker（4x4、5x5、6x6、7x7），每个词典中包含了固定数量的标记（50、100、250、1000），详细可以参考opencv里面的定义：

/** * @brief Predefined markers dictionaries/sets * Each dictionary indicates the number of bits and the number of markers contained * - DICT_ARUCO_ORIGINAL: standard ArUco Library Markers. 1024 markers, 5x5 bits, 0 minimum distance */
    enum CV_EXPORTS_W_SIMPLE PREDEFINED_DICTIONARY_NAME { 
        DICT_4X4_50 = 0,
        DICT_4X4_100,
        DICT_4X4_250,
        DICT_4X4_1000,
        DICT_5X5_50,
        DICT_5X5_100,
        DICT_5X5_250,
        DICT_5X5_1000,
        DICT_6X6_50,
        DICT_6X6_100,
        DICT_6X6_250,
        DICT_6X6_1000,
        DICT_7X7_50,
        DICT_7X7_100,
        DICT_7X7_250,
        DICT_7X7_1000,
        DICT_ARUCO_ORIGINAL
    };

通过以下C++代码可以创建一个marker图像，生成的图像大小为固定的200x200像素：

// 获取大小为6x6、包含1000个marker的字典
    cv::Ptr<cv::aruco::Dictionary> dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_1000);
    // 绘制marker图像到markerImage，且在边缘添加1格黑色边界
    cv::Mat markerImage;
    cv::aruco::drawMarker(dictionary, 23, 200, markerImage, 1);

![在这里插入图片描述][20201221112926495.png_pic_center]

## 2.3 检测aruco marker ##

将打印出来的marker放置在环境中，我们就可以通过相机和以下代码对其进行实时检测：

// 检测到的marker的id
    std::vector<int> ids;
    // 检测到的marker的角点列表
    std::vector<std::vector<cv::Point2f>> corners;
    // 执行检测
    cv::aruco::detectMarkers(image, dictionary, corners, ids, params);

为了提高检测精度，我们可以通过detectMarkers的cv::aruco::DetectorParameters参数（即上一段代码中的params）来设置一些参数。首先让我们先看一下可以设置的参数列表：

/** * @brief Parameters for the detectMarker process: * - adaptiveThreshWinSizeMin: minimum window size for adaptive thresholding before finding * contours (default 3). * - adaptiveThreshWinSizeMax: maximum window size for adaptive thresholding before finding * contours (default 23). * - adaptiveThreshWinSizeStep: increments from adaptiveThreshWinSizeMin to adaptiveThreshWinSizeMax * during the thresholding (default 10). * - adaptiveThreshConstant: constant for adaptive thresholding before finding contours (default 7) * - minMarkerPerimeterRate: determine minimum perimeter for marker contour to be detected. This * is defined as a rate respect to the maximum dimension of the input image (default 0.03). * - maxMarkerPerimeterRate: determine maximum perimeter for marker contour to be detected. This * is defined as a rate respect to the maximum dimension of the input image (default 4.0). * - polygonalApproxAccuracyRate: minimum accuracy during the polygonal approximation process to * determine which contours are squares. * - minCornerDistanceRate: minimum distance between corners for detected markers relative to its * perimeter (default 0.05) * - minDistanceToBorder: minimum distance of any corner to the image border for detected markers * (in pixels) (default 3) * - minMarkerDistanceRate: minimum mean distance beetween two marker corners to be considered * similar, so that the smaller one is removed. The rate is relative to the smaller perimeter * of the two markers (default 0.05). * - cornerRefinementMethod: corner refinement method. (CORNER_REFINE_NONE, no refinement. * CORNER_REFINE_SUBPIX, do subpixel refinement. CORNER_REFINE_CONTOUR use contour-Points) * - cornerRefinementWinSize: window size for the corner refinement process (in pixels) (default 5). * - cornerRefinementMaxIterations: maximum number of iterations for stop criteria of the corner * refinement process (default 30). * - cornerRefinementMinAccuracy: minimum error for the stop cristeria of the corner refinement * process (default: 0.1) * - markerBorderBits: number of bits of the marker border, i.e. marker border width (default 1). * - perpectiveRemovePixelPerCell: number of bits (per dimension) for each cell of the marker * when removing the perspective (default 8). * - perspectiveRemoveIgnoredMarginPerCell: width of the margin of pixels on each cell not * considered for the determination of the cell bit. Represents the rate respect to the total * size of the cell, i.e. perpectiveRemovePixelPerCell (default 0.13) * - maxErroneousBitsInBorderRate: maximum number of accepted erroneous bits in the border (i.e. * number of allowed white bits in the border). Represented as a rate respect to the total * number of bits per marker (default 0.35). * - minOtsuStdDev: minimun standard deviation in pixels values during the decodification step to * apply Otsu thresholding (otherwise, all the bits are set to 0 or 1 depending on mean higher * than 128 or not) (default 5.0) * - errorCorrectionRate error correction rate respect to the maximun error correction capability * for each dictionary. (default 0.6). */
    struct CV_EXPORTS_W DetectorParameters { 
    
        DetectorParameters();
    
        CV_WRAP static Ptr<DetectorParameters> create();
    
        CV_PROP_RW int adaptiveThreshWinSizeMin;
        CV_PROP_RW int adaptiveThreshWinSizeMax;
        CV_PROP_RW int adaptiveThreshWinSizeStep;
        CV_PROP_RW double adaptiveThreshConstant;
        CV_PROP_RW double minMarkerPerimeterRate;
        CV_PROP_RW double maxMarkerPerimeterRate;
        CV_PROP_RW double polygonalApproxAccuracyRate;
        CV_PROP_RW double minCornerDistanceRate;
        CV_PROP_RW int minDistanceToBorder;
        CV_PROP_RW double minMarkerDistanceRate;
        CV_PROP_RW int cornerRefinementMethod;
        CV_PROP_RW int cornerRefinementWinSize;
        CV_PROP_RW int cornerRefinementMaxIterations;
        CV_PROP_RW double cornerRefinementMinAccuracy;
        CV_PROP_RW int markerBorderBits;
        CV_PROP_RW int perspectiveRemovePixelPerCell;
        CV_PROP_RW double perspectiveRemoveIgnoredMarginPerCell;
        CV_PROP_RW double maxErroneousBitsInBorderRate;
        CV_PROP_RW double minOtsuStdDev;
        CV_PROP_RW double errorCorrectionRate;
    };

这里设置挑几个参数进行设置，其他参数可根据应用场景自行添加：

cv::Ptr<cv::aruco::DetectorParameters> params = cv::aruco::DetectorParameters::create();
    params->adaptiveThreshWinSizeMin = 7;
    params->cornerRefinementWinSize = 5;
    params->cornerRefinementMaxIterations = 100;
    params->cornerRefinementMinAccuracy = 0.1;
    params->cornerRefinementMethod = cv::aruco::CORNER_REFINE_CONTOUR;

## 2.4 绘制检测结果及AR显示 ##

利用下面的代码可将检测到的marker边界及角点绘制在图像上：

// 绘制检测到的marker的框
    cv::aruco::drawDetectedMarkers(image, corners, ids);

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc_size_16_color_FFFFFF_t_70_pic_center 1]

利用检测到的角点（corners）以及下面代码中出现的相机参数（cameraMatrix、distCoeffs）、marker尺寸（markerLength）来计算marker与相机坐标系之间的三维空间变换关系，分别得到旋转分量rvecs和平移分量tvecs。其中rvecs可以通过罗德里格斯变换计算对应的旋转矩阵，tvecs的单位与markerLength的单位一致。

// 估计第一个marker的位姿矩阵
    double markerLength = 50;
    std::vector<cv::Vec3d> rvecs, tvecs;
    cv::aruco::estimatePoseSingleMarkers(corners, markerLength, cameraMatrix, distCoeffs, rvecs, tvecs);
    cv::aruco::drawAxis(image, cameraMatrix, distCoeffs, rvecs, tvecs, 0.1);

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc_size_16_color_FFFFFF_t_70_pic_center 2]

如果想绘制其他的虚拟物体，将前面得到的rvecs和tvecs应用到该虚拟物体的变换中即可。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/0ebfef6bcec2465e94a9431f34c3e33d.png
[Link 1]: https://blog.csdn.net/cfan927/article/details/106879454
[Link 2]: https://www.bzblog.online/opencv/opencv_contrib/
[20201221112926495.png_pic_center]: /images/20221120/2b77f4f3e2604fdcb2e5186c216633b9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/b680eab81d994c769d8f9c19c4432ef5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/2020122111504914.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NmYW45Mjc=,size_16,color_FFFFFF,t_70#pic_center