[paper reading] CenterNet (Object as Points)

野性酷女 2022-12-19 11:59 210阅读 0赞

# \[paper reading\] CenterNet (Object as Points) #

GitHub：[Notes of Classic Detection Papers][]

> 2020.11.09更新：更新了Use Yourself，即对于本文的理解和想法，详情参见[GitHub：Notes-of-Classic-Detection-Papers][Notes of Classic Detection Papers]

> 本来想放到GitHub的，结果GitHub不支持公式。  
> 没办法只能放到CSDN，但是格式也有些乱  
> **强烈建议去GitHub上下载源文件**，来阅读学习！！！这样阅读体验才是最好的  
> 当然，**如果有用，希望能给个star**！

<table> 
 <thead> 
 <tr> 
 <th align="center">topic</th> 
 <th align="center">motivation</th> 
 <th align="center">technique</th> 
 <th align="center">key element</th> 
 <th align="center">math</th> 
 <th align="center">use yourself</th> 
 <th align="center">relativity</th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td align="center">CenterNet (Object as Points)</td> 
 <td align="center">Problem to Solve Idea</td> 
 <td align="center">CenterNet Architecture</td> 
 <td align="center">Center Point &amp; Anchor Getting Ground-Truth Model Output Data Augmentation Inference TTA Compared with SOTA Additional Experiments</td> 
 <td align="center">Loss Function KeyPoint Loss L k \text{L}_k Lk Offset Loss L o f f \text{L}_{off} Loff Size Loss L s i z e \text{L}_{size} Lsize</td> 
 <td align="center">……</td> 
 <td align="center">Anchor-Based KeyPoint-Based</td> 
 </tr> 
 </tbody> 
</table>

### 文章目录 ###

*  \[paper reading\] CenterNet (Object as Points)
 *   *  Motivation
     *   *  Problem to Solve
         *  Idea
     *  Technique
     *   *  CenterNet Architecture
         *   *  Components
             *  Advantage
     *  Key Element
     *   *  Center Point & Anchor
         *   *  Connection
             *  Difference
         *  Getting Ground-Truth
         *   *  Keypoint Ground-Truth
             *   *  Ground-Truth：Input Image ==> Output Feature Map
                 *  Gaussian Penalty Reduction
             *  Size Ground-Truth
         *  Model Output
         *  Data Augmentation
         *  Inference
         *  TTA
         *  Compared with SOTA
         *  Additional Experiments
         *   *  Center Point Collision
             *  NMS
             *  Training & Testing Resolution
             *  Regression Loss
             *  Bounding Box Size Weight
             *  Training Schedule
     *  Math
     *   *  Symbol Definition
         *  Loss Function
         *  KeyPoint Loss  L k \\text\{L\}\_k Lk
         *  Offset Loss  L o f f \\text\{L\}\_\{off\} Loff
         *  Size Loss  L s i z e \\text\{L\}\_\{size\} Lsize
     *  Use Yourself
     *  Related work
     *   *  Anchor-Based Method
         *   *  Essence
             *  Two-Stage Method
             *  One-Stage Method
             *  Post-Processing（NMS）
         *  KeyPoint-Based Method
         *   *  Essence
             *  CornerNet
             *  ExtremeNet
             *  Drawback

## Motivation ##

### Problem to Solve ###

**anchor-based method**有以下的缺点：

*  **wasteful & inefficient**：
    
    需要对**object**进行**饱和式检测**（饱和式地列出object的潜在位置）
 *  **need post-processing**（e.g. **NMS**）

### Idea ###

*  从**本质**上讲：
    
    将Object Detection转化为Standard Keypoint Estimation
 *  从**思路**上讲：
    
    使用**bounding box的center point**表示一个**object**
 *  从**具体流程**上讲：
    
    使用**keypoint estimation**寻找**center point**，并**根据center point回归其他的属性**（因为**其他的属性**都和**center point**存在**确定的数学关系**）

## Technique ##

### CenterNet Architecture ###

#### Components ####

*  **Backbone**
    
     *  **Stacked Hourglass Network**
        
        > 详见 \[CornerNet\](./\[paper reading\] RetinaNet.md)
     *  **Upconvolutional Residual Netwotk**
     *  **Deep Layer Aggregation**（DLA）
 *  **Task-Specific Modality**
    
     *  **1 个 3×3 Convolution**
     *  **ReLU**
     *  **1 个 1×1 Convolution**

#### Advantage ####

*  **simpler & faster & accurate**
 *  **end-to-end differential**
    
    **所有的输出**都是**直接**从**keypoint estimation network**输出，**不需要NMS**（以及其他post-processing）
    
    Peak Keypoint Extraction由  3 × 3 Max Pooling 3×3 \\ \\text\{Max Pooling\} 3×3Max Pooling 实现，足够用来替换NMS
 *  **estimate additional object properties in one single forward pass**
    
    在**单次前向传播**中，可以估计出**多种object properties**

## Key Element ##

### Center Point & Anchor ###

#### Connection ####

**center point**可以看作是**shape-agnostic anchor**（**形状不可知**的anchor）

#### Difference ####

*  **center point**仅仅与**location**有关（**与box overlap无关**）
    
    即：**不需要手动设置foreground和background的threshold**
 *  **每个object**仅对应**1个center point**
    
    **直接**在**keypoint heatmap**上提取**local peak**，**不存在重复检测**的问题
 *  **CenterNet**有**更大的输出分辨率**
    
    **降采样步长为4**（常见为16）

### Getting Ground-Truth ###

> 详见 \[Symbol Definition\](\#Symbol Definition)

#### Keypoint Ground-Truth ####

##### Ground-Truth：Input Image ==> Output Feature Map #####

*   p ∈ R 2 p \\in \\mathcal\{R\}^2 p∈R2 ：**ground-truth keypoint**
 *   p ~ = ⌊ p R ⌋ \\widetilde\{p\} = \\lfloor\\frac pR \\rfloor p=⌊Rp⌋ ：**low-resolution equivalent**

将**image**的**ground-truth keypoint**  p p p 映射为**output feature map**上**ground-truth keypoint**  p ~ \\widetilde p p  
 p ~ = ⌊ p R ⌋ \\widetilde\{p\} = \\lfloor\\frac pR \\rfloor p=⌊Rp⌋

##### Gaussian Penalty Reduction #####

Y x y c = exp ⁡ ( − ( x − p ~ x ) 2 + ( y − p ~ y ) 2 2 σ p 2 ) Y\_\{x y c\}=\\exp \\left(-\\frac\{\\left(x-\\tilde\{p\}\_\{x\}\\right)^\{2\}+\\left(y-\\tilde\{p\}\_\{y\}\\right)^\{2\}\}\{2 \\sigma\_\{p\}^\{2\}\}\\right) Yxyc=exp(−2σp2(x−p~x)2\+(y−p~y)2)

*   σ p \\sigma\_\{p\} σp ：**object size-adaptive**的**标准差**

> 如果**同一个类别**的**2个Gaussian**发生**重叠**，则取**element-wise maximum**

**keypoint heatmap**：  
 Y ^ ∈ \[ 0 , 1 \] W R × H R × C \\hat\{Y\}\\in\[0,1\]^\{\\frac\{W\}\{R\}×\\frac HR×C\} Y^∈\[0,1\]RW×RH×C

*   Y ^ x , y , c = 1 \\hat Y \_\{x,y,c\} =1 Y^x,y,c=1 ==> **keypoint**
 *   Y ^ x , y , c = 0 \\hat Y \_\{x,y,c\} =0 Y^x,y,c=0 ==> **background**

> 注意：这里的**center**是**bounding box**的**几何中心**，即**center**到**左右边和上下边**的**距离是相等的**

#### Size Ground-Truth ####

bounding box 用4个点表示（第  k k k 个object，类别为  c k c\_k ck）：  
 ( x 1 ( k ) , y 1 ( k ) , x 2 ( k ) , y 2 ( k ) ) (x\_1^\{(k)\}, y\_1^\{(k)\}, x\_2^\{(k)\}, y\_2^\{(k)\}) (x1(k),y1(k),x2(k),y2(k))  
**Center** 表示为：  
 p k = ( x 1 ( k ) + x 2 ( k ) 2 , y 1 ( k ) + y 2 ( k ) 2 ) p\_k = \\big( \\frac\{x\_1^\{(k)\} + x\_2^\{(k)\} \}\{2\} , \\frac\{y\_1^\{(k)\} + y\_2^\{(k)\} \}\{2\} \\big) pk=(2x1(k)\+x2(k),2y1(k)\+y2(k))  
**Size Ground-Truth** 表示为：  
 s k = ( x 2 ( k ) − x 1 ( k ) , y 2 ( k ) − y 1 ( k ) ) s\_k = \\big(x\_2^\{(k)\} - x\_1^\{(k)\}, y\_2^\{(k)\}-y\_1^\{(k)\} \\big) sk=(x2(k)−x1(k),y2(k)−y1(k))

> 注意：**不对scale进行归一化**，而是**直接使用raw pixel coordinate**

### Model Output ###

> **Input & Output Resolution**：
> 
>  *  512×512
>  *  128×128

**所有的输出共享一个共用的全卷积网络**

*  **keypoint**  Y ^ \\hat Y Y^ ==>  C C C
 *  **offset**  O ^ \\hat O O^ ==> 2
 *  **size**  S ^ \\hat S S^ ==> 2

即：每个**location**都有**C+4个output**

对于each modality，在将feature经过：

*  **1 个 3×3 Convolution**
 *  **ReLU**
 *  **1 个 1×1 Convolution**

### Data Augmentation ###

*  **random flip**
 *  **random scaling**（0.6~1.3）
 *  **cropping**
 *  **color jittering**

### Inference ###

**CenterNet**的**Inference**是**single network forward pass**

1.  将**image**输入**backbone（e.g. FCN）**，得到**3个输出**：
    
     *  **keypoint**  Y ^ \\hat Y Y^ ==>  C C C
        
        **heatmap的peak**对应**object的center**（取**top-100**）
        
        **peak的判定**：值  ≥ \\ge ≥ 其8个邻居
     *  **offset**  O ^ \\hat O O^ ==> 2
     *  **size**  S ^ \\hat S S^ ==> 2
2.  根据**keypoint**  Y ^ \\hat Y Y^、 **offset**  O ^ \\hat O O^、**size**  S ^ \\hat S S^ 计算bounding box
    
    ![在这里插入图片描述][20201107161727385.png_pic_center]
    
     *   ( δ x ^ i , δ x ^ i ) = O ^ x ^ i , y ^ i (\\delta \\hat x\_i, \\delta \\hat x\_i) = \\hat O\_\{\\hat x\_i, \\hat y\_i\} (δx^i,δx^i)=O^x^i,y^i ：**offset prediction**
     *   ( w ^ i , h ^ i ) = S ^ x ^ i , y ^ i ( \\hat w\_i, \\hat h\_i) = \\hat S \_\{\\hat x\_i, \\hat y\_i\} (w^i,h^i)=S^x^i,y^i ：**size prediction**
3.  计算**keypoint**的**confidence**：keypoint对应位置的value  
     Y ^ x i , y i c \\hat Y\_\{x\_i,y\_ic\} Y^xi,yic

### TTA ###

有3种TTA方式：

1.  **no augmentation**
2.  **flip augmentation**
    
    **flip**：在**decoding**之前，进行**output average**
3.  **flip & multi-scale（0.5，0.75，1，1.25，1.5）**
    
    **multi-scale**：使用**NMS**对结果进行聚合

### Compared with SOTA ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNjgzMDY1_size_16_color_FFFFFF_t_70_pic_center]

### Additional Experiments ###

#### Center Point Collision ####

多个object经过下采样，其center keypoint有可能重叠

CenterNet可以减少Center Keypoint的冲突

#### NMS ####

CenterNet使用了NMS提升很小，说明CenterNet不需要NMS

#### Training & Testing Resolution ####

1.  低分辨率速度最快但是精度最差
2.  高分辨率精度提高，但速度降低
3.  原尺寸速度略高于高分辨率，但速度略慢

![image-20201105211137786][]

#### Regression Loss ####

smooth L1 Loss的效果略差于L1 Loss

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNjgzMDY1_size_16_color_FFFFFF_t_70_pic_center 1]

#### Bounding Box Size Weight ####

λ s i z e \\lambda\_\{size\} λsize 为0.1时最佳，增大时AP快速衰减，减小时鲁棒

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNjgzMDY1_size_16_color_FFFFFF_t_70_pic_center 2]

#### Training Schedule ####

训练时间更长，效果更好

![在这里插入图片描述][2020110716181560.png_pic_center]

## Math ##

> ### Symbol Definition ###
> 
>  *   I ∈ R W × H × 3 I \\in R^\{W×H×3\} I∈RW×H×3 ：**image**
>  *   R R R ：**output stride**，实验中为4
>  *   C C C ：**keypoint**的**类别数**

### Loss Function ###

L d e t = L k + λ s i z e L s i z e + λ o f f L o f f \\text\{L\}\_\{det\} = \\text\{L\}\_k + \\lambda\_\{size\} \\text\{L\}\_\{size\} + \\lambda\_\{off\} \\text\{L\}\_\{off\} Ldet=Lk\+λsizeLsize\+λoffLoff

*   λ s i z e = 0.1 \\lambda\_\{size\} = 0.1 λsize=0.1
 *   λ o f f = 1 \\lambda\_\{off\} = 1 λoff=1

### KeyPoint Loss  L k \\text\{L\}\_k Lk ###

**penalty-reduced** pixel-wise **logistic regression** with **focal loss**

<img src="\[paper reading\] CenterNet (Object as Points).assets/image-20201105190626950.png" alt="image-![在这里插入图片描述][20201107161830643.png_pic_center]

*   Y ^ x y c \\hat\{Y\}\_\{xyc\} Y^xyc ：**predicted keypoint confidence**
 *   α = 2 , β = 4 \\alpha =2,\\beta=4 α=2,β=4

### Offset Loss  L o f f \\text\{L\}\_\{off\} Loff ###

目的：恢复由**下采样**带来的**离散化错误**（discretization error）

![在这里插入图片描述][20201107161842796.png_pic_center]

*   O ^ ∈ R W R × H R × 2 \\hat O \\in \\mathcal R^\{\\frac\{W\}\{R\}×\\frac HR×2\} O^∈RRW×RH×2 ：**predicted local offset**

**注意**：

*  仅仅对**keypoint location**（**positive**）计算
 *  **所有的类别**共享**相同的offset prediction**

### Size Loss  L s i z e \\text\{L\}\_\{size\} Lsize ###

![在这里插入图片描述][20201107161851807.png_pic_center]

*   S ^ p k ∈ R W R × H R × 2 \\hat\{S\}\_\{p\_\{k\}\} \\in \\mathcal R^\{\\frac\{W\}\{R\}×\\frac HR×2\} S^pk∈RRW×RH×2
 *   s k = ( x 2 ( k ) − x 1 ( k ) , y 2 ( k ) − y 1 ( k ) ) s\_k = \\big(x\_2^\{(k)\} - x\_1^\{(k)\}, y\_2^\{(k)\}-y\_1^\{(k)\} \\big) sk=(x2(k)−x1(k),y2(k)−y1(k))

## Use Yourself ##

……

## Related work ##

### Anchor-Based Method ###

#### Essence ####

将**detection**降级为**classification**

#### Two-Stage Method ####

1. 在**image**上放置**anchor**（同 \[One-Stage Method\](\#One-Stage Method)）
 
 即：在**low-resolution**上**dense & grid采样anchor**，分类为**foreground/background** ==> **proposal**
 
 > 具体的label：
 > 
 > * **foreground**：
 > 
 > 与**任意ground-truth box**有 **> 0.7 的IoU**
 > * **background**：
 > 
 > 与**任意ground-truth box**有 **< 0.3 的IoU**
 > * **ignored**：
 > 
 > 与**任意ground-truth box** 的**IoU** ∈ \[ 0.3 , 0.7 \] \\in \[0.3, 0.7\] ∈\[0.3,0.7\]
2. 对**anchor**进行**feature resample**

比如：

*  **RCNN**：在**image**上取**crop**
 *  **Fast-RCNN**：在**feature map**上取**crop**

#### One-Stage Method ####

1.  在**image**上放置**anchor**
2.  **直接**对**anchor**位置进行**分类**

one-stage method的一些**改进**：

*  **anchor shape prior**
 *  **different feature resolution**（e.g. **Feature Pyramid Network**）
 *  **loss re-weighting**（e.g. **Focal Loss**）

#### Post-Processing（NMS） ####

*  **Purpose**：根据**IoU**，抑制**相同instance的detections**
 *  **Drawback**：难以**differentiate**和**train**，导致绝大部分的detector**无法做到end-to-end trainable**

### KeyPoint-Based Method ###

#### Essence ####

将**detection**转化为**keypoint estimation**

其**Backbone**均为**KeyPoint Estimation Network**

#### CornerNet ####

检测**2个corner**作为**keypoints**，表示**1个bounding box**

#### ExtremeNet ####

检测 **top-most**, **left-most**, **bottom-most**, **right-most** ,**center** 作为keypints

#### Drawback ####

对**1个object检测多个keypoint**，其需要**额外的grouping stage**（导致算法速度的降低）

[Notes of Classic Detection Papers]: https://github.com/HanGuangXin/Notes-of-Classic-Detection-Papers
[20201107161727385.png_pic_center]: /images/20221120/4fe484708d484cb3a9483c633152cf00.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNjgzMDY1_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/5d2dca73607148b592eebb134133ce7a.png
[image-20201105211137786]: 
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNjgzMDY1_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/beeece08ce8949e0946ed517c1d2fad8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNjgzMDY1_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221120/d190f8d1d1cc44c283f2d5ee01e233c4.png
[2020110716181560.png_pic_center]: https://img-blog.csdnimg.cn/2020110716181560.png#pic_center
[20201107161830643.png_pic_center]: https://img-blog.csdnimg.cn/20201107161830643.png#pic_center
[20201107161842796.png_pic_center]: https://img-blog.csdnimg.cn/20201107161842796.png#pic_center
[20201107161851807.png_pic_center]: https://img-blog.csdnimg.cn/20201107161851807.png#pic_center