【解决问题】Hint: If you want to see a list of allocated tensors when OOM happen

╰半夏微凉° 2023-07-06 06:13 1阅读 0赞

今天运行一个风格迁移神经网络的时候，出现以下问题：

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,64,1669,1200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
    	 [[Node: Conv2D_10 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_10-0-TransposeNHWCToNCHW-LayoutOptimizer, encoder/conv1_2/kernel/read)]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
    	 [[Node: clip_by_value/_77 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_487_clip_by_value", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

上网查了以后，说是GPU满了，可以调小 Batch\_size来解决，但是我的batch\_size本来就只有8，于是下调到4，依然不行。

于是根据其他博主的建议，指定GPU后：

import os
    os.environ["CUDA_VISIBLE_DEVICES"] = "2"#指定第二块GPU

解决了，但是由于显卡不是很大 所以运行起来特别慢。

参考：

*  [参考博客1][1]
 *  [深度学习笔记：windows+tensorflow 指定GPU占用内存（解决gpu爆炸问题）][windows_tensorflow _GPU_gpu]

[1]: https://blog.csdn.net/qq_33221533/article/details/100188050?ops_request_misc=%7B%22request_id%22:%22158184130619724811803620%22,%22scm%22:%2220140713.130056874..%22%7D&request_id=158184130619724811803620&biz_id=0&utm_source=distribute.pc_search_result.none-task
[windows_tensorflow _GPU_gpu]: https://blog.csdn.net/lvsehaiyang1993/article/details/80921228