MobileNet 在caffe训练加速方法：多GPU：解决Multi-GPU execution not available - rebuild with USE_NCCL

小鱼儿 2022-03-11 05:28 197阅读 0赞

**1、直接在如下训练指令后加：--gpu all或者 --gpu 0,1报错**

./build/tools/caffe train --solver=examples/myExample/0227\_gpu/solver\_2acc.prototxt

报错内容：Multi-GPU execution not available - rebuild with USE\_NCCL

![20190307134646445.png][]

**2、解决方法：**

（1）查看cuda和gcc版本

（cuda版本要求：cuda version>6.0，gcc 版本>=gcc 4.8）

cuda 版本   
cat /usr/local/cuda/version.txt

![20190307135200978.png][]

查看gcc版本

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc3MDE2OQ_size_16_color_FFFFFF_t_70][]

（2）安装NCCL1

cd nccl-master

make CUDA\_HOME=/usr/local/cuda-8.0/

sudo make install

sudo ldconfig

（3）caffe的makefile.config修改

添加：

![20190307161855219.png][]

> USE\_NCCL := 1
> 
> INCLUDE\_DIRS += /home/yanghuiyu/make\_cmake/nccl-master/build/include
> 
> LIBRARY\_DIRS += /home/yanghuiyu/make\_cmake/nccl-master/build/lib

makefile中要有：

![20190307162113814.png][]

> ifeq ($(USE\_NCCL), 1)
> 
>     LIBRARIES += nccl
> 
>     COMMON\_FLAGS += -DUSE\_NCCL endif

（4）caffe重新编译

sudo make clean

sudo make all -j

**3、训练速度效果的提升：提升了10倍左右**

1s2个batch ----- 1s19个batch

参考：

[https://blog.csdn.net/jmu201521121021/article/details/78654037][https_blog.csdn.net_jmu201521121021_article_details_78654037]

[https://blog.csdn.net/wudi\_X/article/details/80012764\#commentBox][https_blog.csdn.net_wudi_X_article_details_80012764_commentBox]

[20190307134646445.png]: /images/20220311/46fc9ac7fb45428bb9c80e565e1d8f3d.png
[20190307135200978.png]: /images/20220311/a105781a7c084492b44e18717f618887.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTc3MDE2OQ_size_16_color_FFFFFF_t_70]: /images/20220311/368f52a732ab4eefba57fb148eb382a4.png
[20190307161855219.png]: /images/20220311/7ad208e7196f4244ac2a0d64dc7d1263.png
[20190307162113814.png]: /images/20220311/4f26e0247529473a9951213a1791c8c2.png
[https_blog.csdn.net_jmu201521121021_article_details_78654037]: https://blog.csdn.net/jmu201521121021/article/details/78654037
[https_blog.csdn.net_wudi_X_article_details_80012764_commentBox]: https://blog.csdn.net/wudi_X/article/details/80012764#commentBox