图像修复方法总结以及深度学习的论文汇总（更新中。。。）

以你之姓@ 2024-05-24 02:11 38阅读 0赞

### 1、图像修复总结 ###

图像修复（Image inpainting or Image complete ）的目的是在给定一个mask的情况下，填充缺失区域的像素，使其整体达到纹理和结构一致性，或者语义和视觉可信。其应用范围十分广泛，如图像复原，图像编辑，图像去噪。图像修复本身就是一个高度病态问题，修复过程带有高度的主观性，比如专家修复艺术作品。现有的大多数深度学习方法将图像修复视为一个条件生成问题，输入到输出的映射是一对多，最后生成的一个最优结果。然而该结果没有丰富的语义内容，可能生成的结果内容不一样了，但仍看似合理。

现有的图像方法可分为深度学习方法和非深度学习方法。

传统方法（非深度学习方法）可以分为如下四类：基于扩散的图像修复方法，**基于纹理合成的图像方法，基于数据驱动的图像修复方法以及基于稀疏表示的图像修复算法**。从广义上讲，传统的修复方法可分两类：基于扩散的（diffusion-based）和基于样本的（patch-based）图像修复方法。

1.  基于扩散的方法，利用待修复区域的边缘信息，确定扩散的方向，向边缘内扩散已知的信息。该类方法在修复图像的小尺度缺失效果可观，但是当缺失区域较大（如64x64，128x128的矩形块，不规则的缺失）或纹理复杂时，该类算法修复的结果存在模糊问题。此外，该算法无法对纹理进行推理，修复存在结构和纹理的不一致，内容不合理。
2.  基于纹理合成的方法，先将图像分成一个集合，设计一个匹配原则，寻找出相似度最高的块填充缺失区域。该类方法在缺失区域为背景时修复效果可以，但当缺失区域的为前景且纹理和结构复杂时，很难寻找出合适的块填充缺失区域。且搜索计算相似度高的块代价昂贵。
3.  基于数据驱动的图像修复算法，从大型的外部数据库中，以数据驱动的方式查询出相似的图像，用以填充缺失的区域。该类方法查询的计算量较大，当外部数据库中没有合理的块，修复结果自然就较差，且应用场景受限。
4.  基于稀疏表示的图像修复算法，利用图像缺失区域和完好区域具备相同的稀疏性，采集完好区域的信息，进行稀疏表示，通过重构算法重构图像，进而恢复成完整的图像。该类算法在计算稀疏系数（NP-hard问题）和迭代获取完备字典时，计算量较大，复杂度较高。

深度学习方法，按照网络结构类型分类，可以分为三类：**基于卷积自编码的图像修复算法，基于GAN的图像修复算法，基于RNN的图像修复算法**。目前应用较广的是基于卷积自编码的图像修复算法，提出了许多改进方案。

最近几年，基于深度学习的图像方法已经能从大规模数据集中学习到丰富的语义信息，然后以端到端的方式填充图像中缺失的内容，从而取得较好的修复效果。每年至少有7、8篇AB类会议。此外，现有的深度学习方法运用了传统图像修复方法的思想，比如图像内容和风格的推导，结构和纹理的推理，轮廓和边缘的推导。

下面将总结一下最近四年的图像修复方法，给出论文的下载地址，以及开源的代码。  
**备注：**  
**pdf的链接若失效，可用谷歌学术直接搜索paper名。github地址是论文的开源实现，可以根据自己实际需要（tensorflow,pytorch,keras等）在GitHub上搜索复现的项目。未开源代表没有搜索到开源版本**

### 2、深度学习论文汇总 ###

1、Pathak, Context Encoders: Feature Learning by Inpainting, 2016 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_cvpr\_2016/papers/Pathak\_Context\_Encoders\_Feature\_CVPR\_2016\_paper.pdf][http_openaccess.thecvf.com_content_cvpr_2016_papers_Pathak_Context_Encoders_Feature_CVPR_2016_paper.pdf]
 *  github:[https://github.com/BoyuanJiang/context\_encoder\_pytorch][https_github.com_BoyuanJiang_context_encoder_pytorch]

2、Iizuka， Globally and locally consistent image completion,2017 TOG.

*  pdf:[http://delivery.acm.org/10.1145/3080000/3073659/a107-iizuka.pdf?ip=113.251.221.139&id=3073659&acc=ACTIVE%20SERVICE&key=BF85BBA5741FDC6E%2E41023622363E58C3%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35&**acm**=1569199853\_5bedf657c659bd617e8007a45007480d][http_delivery.acm.org_10.1145_3080000_3073659_a107-iizuka.pdf_ip_113.251.221.139_id_3073659_acc_ACTIVE_20SERVICE_key_BF85BBA5741FDC6E_2E41023622363E58C3_2E4D4702B0C3E38B35_2E4D4702B0C3E38B35_acm_1569199853_5bedf657c659bd617e8007a45007480d]
 *  github:[https://github.com/shinseung428/GlobalLocalImageCompletion\_TF][https_github.com_shinseung428_GlobalLocalImageCompletion_TF]

3、Yang, High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis, 2017 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_cvpr\_2017/papers/Yang\_High-Resolution\_Image\_Inpainting\_CVPR\_2017\_paper.pdf][http_openaccess.thecvf.com_content_cvpr_2017_papers_Yang_High-Resolution_Image_Inpainting_CVPR_2017_paper.pdf]
 *  github:[https://github.com/leehomyc/Faster-High-Res-Neural-Inpainting][https_github.com_leehomyc_Faster-High-Res-Neural-Inpainting]

4、Gao, On-Demand Learning for Deep Image Restoration, 2017 ICCV.

*  pdf:[http://openaccess.thecvf.com/content\_ICCV\_2017/papers/Gao\_On-Demand\_Learning\_for\_ICCV\_2017\_paper.pdf][http_openaccess.thecvf.com_content_ICCV_2017_papers_Gao_On-Demand_Learning_for_ICCV_2017_paper.pdf]
 *  github:[https://github.com/rhgao/on-demand-learning][https_github.com_rhgao_on-demand-learning]

5、Li，Generative face completion, 2017 CVPR. 一篇针对人脸的图像修复。

*  pdf:[http://openaccess.thecvf.com/content\_cvpr\_2017/papers/Li\_Generative\_Face\_Completion\_CVPR\_2017\_paper.pdf][http_openaccess.thecvf.com_content_cvpr_2017_papers_Li_Generative_Face_Completion_CVPR_2017_paper.pdf]
 *  github:[https://github.com/Yijunmaverick/GenerativeFaceCompletion][https_github.com_Yijunmaverick_GenerativeFaceCompletion]

6、Yu，Generative image inpainting with contextual attention,2018 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_cvpr\_2018/papers/Yu\_Generative\_Image\_Inpainting\_CVPR\_2018\_paper.pdf][http_openaccess.thecvf.com_content_cvpr_2018_papers_Yu_Generative_Image_Inpainting_CVPR_2018_paper.pdf]
 *  github:[https://github.com/JiahuiYu/generative\_inpainting][https_github.com_JiahuiYu_generative_inpainting]

7、Song, Contextual-based Image Inpainting: Infer, Match, and Translate, 2018 ECCV.

*  pdf:[http://openaccess.thecvf.com/content\_ECCV\_2018/papers/Yuhang\_Song\_Contextual\_Based\_Image\_ECCV\_2018\_paper.pdf][http_openaccess.thecvf.com_content_ECCV_2018_papers_Yuhang_Song_Contextual_Based_Image_ECCV_2018_paper.pdf]
 *  github:未开源

8、Zhang, Semantic Image Inpainting with Progressive Generative Networks , 2018 MM.

*  pdf:[http://delivery.acm.org/10.1145/3250000/3240625/p1939-zhang.pdf?ip=113.251.221.139&id=3240625&acc=ACTIVE%20SERVICE&key=BF85BBA5741FDC6E%2E41023622363E58C3%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35&**acm**=1569201161\_266fc747cbee0404c97ba937a1f0da95][http_delivery.acm.org_10.1145_3250000_3240625_p1939-zhang.pdf_ip_113.251.221.139_id_3240625_acc_ACTIVE_20SERVICE_key_BF85BBA5741FDC6E_2E41023622363E58C3_2E4D4702B0C3E38B35_2E4D4702B0C3E38B35_acm_1569201161_266fc747cbee0404c97ba937a1f0da95]
 *  github:[https://github.com/crashmoon/Progressive-Generative-Networks][https_github.com_crashmoon_Progressive-Generative-Networks]

9、Yan, Shift-Net: Image Inpainting via Deep Feature Rearrangement, 2018 ECCV.

*  pdf:[http://openaccess.thecvf.com/content\_ECCV\_2018/papers/Zhaoyi\_Yan\_Shift-Net\_Image\_Inpainting\_ECCV\_2018\_paper.pdf][http_openaccess.thecvf.com_content_ECCV_2018_papers_Zhaoyi_Yan_Shift-Net_Image_Inpainting_ECCV_2018_paper.pdf]
 *  github:[https://github.com/Zhaoyi-Yan/Shift-Net\_pytorch][https_github.com_Zhaoyi-Yan_Shift-Net_pytorch]

10、Wang, Image Inpainting via Generative Multi-column Convolutional Neural Networks, 2018 NIPS.

*  pdf:[http://papers.nips.cc/paper/7316-image-inpainting-via-generative-multi-column-convolutional-neural-networks][http_papers.nips.cc_paper_7316-image-inpainting-via-generative-multi-column-convolutional-neural-networks]
 *  github:[https://github.com/shepnerd/inpainting\_gmcnn][https_github.com_shepnerd_inpainting_gmcnn]

11、Liu, Image Inpainting for Irregular Holes Using Partial Convolutions, 2018 ECCV.

*  pdf:[http://openaccess.thecvf.com/content\_ECCV\_2018/papers/Guilin\_Liu\_Image\_Inpainting\_for\_ECCV\_2018\_paper.pdf][http_openaccess.thecvf.com_content_ECCV_2018_papers_Guilin_Liu_Image_Inpainting_for_ECCV_2018_paper.pdf]
 *  github:[https://github.com/MathiasGruber/PConv-Keras][https_github.com_MathiasGruber_PConv-Keras]

12、Yu, Free-Form Image Inpainting with Gated Convolution, 2018 CVPR 未中.

*  pdf:[https://arxiv.org/pdf/1806.03589.pdf][https_arxiv.org_pdf_1806.03589.pdf]
 *  github:[https://github.com/avalonstrel/GatedConvolution\_pytorch][https_github.com_avalonstrel_GatedConvolution_pytorch]

13、Xiao，CISI-net: Explicit Latent Content Inference and Imitated Style Rendering for Image Inpainting，2019 AAAI.

*  pdf:[https://wvvw.aaai.org/ojs/index.php/AAAI/article/view/3805][https_wvvw.aaai.org_ojs_index.php_AAAI_article_view_3805]
 *  github:未开源

14、Sagong，PEPSI : Fast Image Inpainting with Parallel Decoding Network，2019 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_CVPR\_2019/papers/Sagong\_PEPSI\_\_Fast\_Image\_Inpainting\_With\_Parallel\_Decoding\_Network\_CVPR\_2019\_paper.pdf][http_openaccess.thecvf.com_content_CVPR_2019_papers_Sagong_PEPSI_Fast_Image_Inpainting_With_Parallel_Decoding_Network_CVPR_2019_paper.pdf]
 *  github:未开源

15、Li, Generative Image Inpainting with Submanifold Alignment, 2019 IJCAI.

*  pdf:[https://arxiv.org/pdf/1908.00211.pdf][https_arxiv.org_pdf_1908.00211.pdf]
 *  github:未开源

16、Xiong, Foreground-aware Image Inpainting, 2019 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_CVPR\_2019/papers/Xiong\_Foreground-Aware\_Image\_Inpainting\_CVPR\_2019\_paper.pdf][http_openaccess.thecvf.com_content_CVPR_2019_papers_Xiong_Foreground-Aware_Image_Inpainting_CVPR_2019_paper.pdf]
 *  github:未开源

17、Nazeri, EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning， 2019 CVPR未中.

*  pdf:[https://arxiv.org/pdf/1901.00212.pdf][https_arxiv.org_pdf_1901.00212.pdf]
 *  github:[https://github.com/knazeri/edge-connect][https_github.com_knazeri_edge-connect]

18、Zeng, Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting, 2019 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_CVPR\_2019/papers/Zeng\_Learning\_Pyramid-Context\_Encoder\_Network\_for\_High-Quality\_Image\_Inpainting\_CVPR\_2019\_paper.pdf][http_openaccess.thecvf.com_content_CVPR_2019_papers_Zeng_Learning_Pyramid-Context_Encoder_Network_for_High-Quality_Image_Inpainting_CVPR_2019_paper.pdf]
 *  github:未开源

19、Zheng, Pluralistic Image Completion, 2019 CVPR.

*  pdf:[http://openaccess.thecvf.com/content\_CVPR\_2019/papers/Zheng\_Pluralistic\_Image\_Completion\_CVPR\_2019\_paper.pdf][http_openaccess.thecvf.com_content_CVPR_2019_papers_Zheng_Pluralistic_Image_Completion_CVPR_2019_paper.pdf]
 *  github:[https://github.com/lyndonzheng/Pluralistic-Inpainting][https_github.com_lyndonzheng_Pluralistic-Inpainting]

20、Liu, Coherent Semantic Attention for Image Inpainting, 2019 CVPR.

*  pdf:[https://arxiv.org/pdf/1905.12384.pdf][https_arxiv.org_pdf_1905.12384.pdf]
 *  github:未开源

由于作者水平有限，文中若有不正确的地方，欢迎大家指出，若有任何问题，请在下方讨论。

[http_openaccess.thecvf.com_content_cvpr_2016_papers_Pathak_Context_Encoders_Feature_CVPR_2016_paper.pdf]: http://openaccess.thecvf.com/content_cvpr_2016/papers/Pathak_Context_Encoders_Feature_CVPR_2016_paper.pdf
[https_github.com_BoyuanJiang_context_encoder_pytorch]: https://github.com/BoyuanJiang/context_encoder_pytorch
[http_delivery.acm.org_10.1145_3080000_3073659_a107-iizuka.pdf_ip_113.251.221.139_id_3073659_acc_ACTIVE_20SERVICE_key_BF85BBA5741FDC6E_2E41023622363E58C3_2E4D4702B0C3E38B35_2E4D4702B0C3E38B35_acm_1569199853_5bedf657c659bd617e8007a45007480d]: http://delivery.acm.org/10.1145/3080000/3073659/a107-iizuka.pdf?ip=113.251.221.139&id=3073659&acc=ACTIVE%20SERVICE&key=BF85BBA5741FDC6E.41023622363E58C3.4D4702B0C3E38B35.4D4702B0C3E38B35&__acm__=1569199853_5bedf657c659bd617e8007a45007480d
[https_github.com_shinseung428_GlobalLocalImageCompletion_TF]: https://github.com/shinseung428/GlobalLocalImageCompletion_TF
[http_openaccess.thecvf.com_content_cvpr_2017_papers_Yang_High-Resolution_Image_Inpainting_CVPR_2017_paper.pdf]: http://openaccess.thecvf.com/content_cvpr_2017/papers/Yang_High-Resolution_Image_Inpainting_CVPR_2017_paper.pdf
[https_github.com_leehomyc_Faster-High-Res-Neural-Inpainting]: https://github.com/leehomyc/Faster-High-Res-Neural-Inpainting
[http_openaccess.thecvf.com_content_ICCV_2017_papers_Gao_On-Demand_Learning_for_ICCV_2017_paper.pdf]: http://openaccess.thecvf.com/content_ICCV_2017/papers/Gao_On-Demand_Learning_for_ICCV_2017_paper.pdf
[https_github.com_rhgao_on-demand-learning]: https://github.com/rhgao/on-demand-learning
[http_openaccess.thecvf.com_content_cvpr_2017_papers_Li_Generative_Face_Completion_CVPR_2017_paper.pdf]: http://openaccess.thecvf.com/content_cvpr_2017/papers/Li_Generative_Face_Completion_CVPR_2017_paper.pdf
[https_github.com_Yijunmaverick_GenerativeFaceCompletion]: https://github.com/Yijunmaverick/GenerativeFaceCompletion
[http_openaccess.thecvf.com_content_cvpr_2018_papers_Yu_Generative_Image_Inpainting_CVPR_2018_paper.pdf]: http://openaccess.thecvf.com/content_cvpr_2018/papers/Yu_Generative_Image_Inpainting_CVPR_2018_paper.pdf
[https_github.com_JiahuiYu_generative_inpainting]: https://github.com/JiahuiYu/generative_inpainting
[http_openaccess.thecvf.com_content_ECCV_2018_papers_Yuhang_Song_Contextual_Based_Image_ECCV_2018_paper.pdf]: http://openaccess.thecvf.com/content_ECCV_2018/papers/Yuhang_Song_Contextual_Based_Image_ECCV_2018_paper.pdf
[http_delivery.acm.org_10.1145_3250000_3240625_p1939-zhang.pdf_ip_113.251.221.139_id_3240625_acc_ACTIVE_20SERVICE_key_BF85BBA5741FDC6E_2E41023622363E58C3_2E4D4702B0C3E38B35_2E4D4702B0C3E38B35_acm_1569201161_266fc747cbee0404c97ba937a1f0da95]: http://delivery.acm.org/10.1145/3250000/3240625/p1939-zhang.pdf?ip=113.251.221.139&id=3240625&acc=ACTIVE%20SERVICE&key=BF85BBA5741FDC6E.41023622363E58C3.4D4702B0C3E38B35.4D4702B0C3E38B35&__acm__=1569201161_266fc747cbee0404c97ba937a1f0da95
[https_github.com_crashmoon_Progressive-Generative-Networks]: https://github.com/crashmoon/Progressive-Generative-Networks
[http_openaccess.thecvf.com_content_ECCV_2018_papers_Zhaoyi_Yan_Shift-Net_Image_Inpainting_ECCV_2018_paper.pdf]: http://openaccess.thecvf.com/content_ECCV_2018/papers/Zhaoyi_Yan_Shift-Net_Image_Inpainting_ECCV_2018_paper.pdf
[https_github.com_Zhaoyi-Yan_Shift-Net_pytorch]: https://github.com/Zhaoyi-Yan/Shift-Net_pytorch
[http_papers.nips.cc_paper_7316-image-inpainting-via-generative-multi-column-convolutional-neural-networks]: http://papers.nips.cc/paper/7316-image-inpainting-via-generative-multi-column-convolutional-neural-networks
[https_github.com_shepnerd_inpainting_gmcnn]: https://github.com/shepnerd/inpainting_gmcnn
[http_openaccess.thecvf.com_content_ECCV_2018_papers_Guilin_Liu_Image_Inpainting_for_ECCV_2018_paper.pdf]: http://openaccess.thecvf.com/content_ECCV_2018/papers/Guilin_Liu_Image_Inpainting_for_ECCV_2018_paper.pdf
[https_github.com_MathiasGruber_PConv-Keras]: https://github.com/MathiasGruber/PConv-Keras
[https_arxiv.org_pdf_1806.03589.pdf]: https://arxiv.org/pdf/1806.03589.pdf
[https_github.com_avalonstrel_GatedConvolution_pytorch]: https://github.com/avalonstrel/GatedConvolution_pytorch
[https_wvvw.aaai.org_ojs_index.php_AAAI_article_view_3805]: https://wvvw.aaai.org/ojs/index.php/AAAI/article/view/3805
[http_openaccess.thecvf.com_content_CVPR_2019_papers_Sagong_PEPSI_Fast_Image_Inpainting_With_Parallel_Decoding_Network_CVPR_2019_paper.pdf]: http://openaccess.thecvf.com/content_CVPR_2019/papers/Sagong_PEPSI__Fast_Image_Inpainting_With_Parallel_Decoding_Network_CVPR_2019_paper.pdf
[https_arxiv.org_pdf_1908.00211.pdf]: https://arxiv.org/pdf/1908.00211.pdf
[http_openaccess.thecvf.com_content_CVPR_2019_papers_Xiong_Foreground-Aware_Image_Inpainting_CVPR_2019_paper.pdf]: http://openaccess.thecvf.com/content_CVPR_2019/papers/Xiong_Foreground-Aware_Image_Inpainting_CVPR_2019_paper.pdf
[https_arxiv.org_pdf_1901.00212.pdf]: https://arxiv.org/pdf/1901.00212.pdf
[https_github.com_knazeri_edge-connect]: https://github.com/knazeri/edge-connect
[http_openaccess.thecvf.com_content_CVPR_2019_papers_Zeng_Learning_Pyramid-Context_Encoder_Network_for_High-Quality_Image_Inpainting_CVPR_2019_paper.pdf]: http://openaccess.thecvf.com/content_CVPR_2019/papers/Zeng_Learning_Pyramid-Context_Encoder_Network_for_High-Quality_Image_Inpainting_CVPR_2019_paper.pdf
[http_openaccess.thecvf.com_content_CVPR_2019_papers_Zheng_Pluralistic_Image_Completion_CVPR_2019_paper.pdf]: http://openaccess.thecvf.com/content_CVPR_2019/papers/Zheng_Pluralistic_Image_Completion_CVPR_2019_paper.pdf
[https_github.com_lyndonzheng_Pluralistic-Inpainting]: https://github.com/lyndonzheng/Pluralistic-Inpainting
[https_arxiv.org_pdf_1905.12384.pdf]: https://arxiv.org/pdf/1905.12384.pdf