Pytorch cifar10_tutorial.py问题BrokenPipeError: [Errno 32] Broken pipe

客官°小女子只卖身不卖艺 2022-03-29 16:14 171阅读 0赞

### 问题 ###

如果你用windows玩cifar10\_tutorial.py,一定碰到过这个问题：  
BrokenPipeError: \[Errno 32\] Broken pipe When i :"run cifar10\_tutorial.py

先给解决方案，参考：[https://github.com/pytorch/examples/issues/201][https_github.com_pytorch_examples_issues_201]

### 分析 ###

用IPython运行，事个运行过程直到报错，列出来的详细情况如下，

(pytorch) E:\APytorchDev\TutorialCode>IPython  cifar10_tutorial.py
    Files already downloaded and verified
    Files already downloaded and verified
    Files already downloaded and verified
    Files already downloaded and verified
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 105, in spawn_main
        exitcode = _main(fd)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 114, in _main
        prepare(preparation_data)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 225, in prepare
        _fixup_main_from_path(data['init_main_from_path'])
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
        run_name="__mp_main__")
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\runpy.py", line 263, in run_path
        pkg_name=pkg_name, script_name=fname)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\runpy.py", line 96, in _run_module_code
        mod_name, mod_spec, pkg_name, script_name)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "E:\APytorchDev\TutorialCode\cifar10_tutorial.py", line 162, in <module>
        for i, data in enumerate(trainloader, 0):
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
        return _DataLoaderIter(self)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
        w.start()
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\process.py", line 112, in start
        self._popen = self._Popen(self)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\context.py", line 223, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\context.py", line 322, in _Popen
        return Popen(process_obj)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
        prep_data = spawn.get_preparation_data(process_obj._name)
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
        _check_not_importing_main()
      File "C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
        is not going to be frozen to produce an executable.''')
    RuntimeError:
            An attempt has been made to start a new process before the
            current process has finished its bootstrapping phase.
    
            This probably means that you are not using fork to start your
            child processes and you have forgotten to use the proper idiom
            in the main module:
    
                if __name__ == '__main__':
                    freeze_support()
                    ...
    
            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce an executable.
    ---------------------------------------------------------------------------
    BrokenPipeError                           Traceback (most recent call last)
    E:\APytorchDev\TutorialCode\cifar10_tutorial.py in <module>
        160
        161     running_loss = 0.0
    --> 162     for i, data in enumerate(trainloader, 0):
        163         # get the inputs
        164         inputs, labels = data
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py in __iter__(self)
        817
        818     def __iter__(self):
    --> 819         return _DataLoaderIter(self)
        820
        821     def __len__(self):
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py in __init__(self, loader)
        558                 #     before it starts, and __del__ tries to join but will get:
        559                 #     AssertionError: can only join a started process.
    --> 560                 w.start()
        561                 self.index_queues.append(index_queue)
        562                 self.workers.append(w)
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\process.py in start(self)
        110                'daemonic processes are not allowed to have children'
        111         _cleanup()
    --> 112         self._popen = self._Popen(self)
        113         self._sentinel = self._popen.sentinel
        114         # Avoid a refcycle if the target function holds an indirect
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\context.py in _Popen(process_obj)
        221     @staticmethod
        222     def _Popen(process_obj):
    --> 223         return _default_context.get_context().Process._Popen(process_obj)
        224
        225 class DefaultContext(BaseContext):
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\context.py in _Popen(process_obj)
        320         def _Popen(process_obj):
        321             from .popen_spawn_win32 import Popen
    --> 322             return Popen(process_obj)
        323
        324     class SpawnContext(BaseContext):
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj)
         63             try:
         64                 reduction.dump(prep_data, to_child)
    ---> 65                 reduction.dump(process_obj, to_child)
         66             finally:
         67                 set_spawning_popen(None)
    
    C:\ProgramData\Anaconda3\envs\pytorch\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
         58 def dump(obj, file, protocol=None):
         59     '''Replacement for pickle.dump() using ForkingPickler.'''
    ---> 60     ForkingPickler(file, protocol).dump(obj)
         61
         62 #
    
    BrokenPipeError: [Errno 32] Broken pipe
    
    (pytorch) E:\APytorchDev\TutorialCode>

其实呢，报错的信息里已经提示了解决方案：

This probably means that you are not using fork to start your
            child processes and you have forgotten to use the proper idiom
            in the main module:
    
                if __name__ == '__main__':
                    freeze_support()
                    ...
    
            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce an executable.

究其原因，大致是因为windows的multiprocessing library会不停地产生子进程（child process），如果用main保护起来，子进程在运行时因为会导入这个main，（只有一个main进程），所以避免了循环生成子进程。

参考：[https://stackoverflow.com/questions/18204782/runtimeerror-on-windows-trying-python-multiprocessing/18205006\#18205006][https_stackoverflow.com_questions_18204782_runtimeerror-on-windows-trying-python-multiprocessing_18205006_18205006]

On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an `if __name__ == '__main__':` guard in the main module to avoid creating subprocesses recursively.

后记：

这个贴子写了好久了，大多都忘记什么原因了。看到有些人还是不明白怎么加这一句，我解释一下，大概意思就是在windows下，调用fork多进程的语句一定要放到if \_\_name\_\_ == '\_\_main\_\_'里面包起来，比如你原来的主程序是my\_main()，linux里可以直接调用

my\_main()

但在windows下不行，必须这样

if \_\_name\_\_ == '\_\_main\_\_'：  
    my\_main()

否则会不断创建新进程，直到程序崩溃。至于my\_main()代表的是一个函数还是一堆代码，都是一样的要求，

我把全部源码贴到下面，大概这个样子

# -*- coding: utf-8 -*-
    """
    Training a Classifier
    =====================
    
    This is it. You have seen how to define neural networks, compute loss and make
    updates to the weights of the network.
    
    Now you might be thinking,
    
    What about data?
    ----------------
    
    Generally, when you have to deal with image, text, audio or video data,
    you can use standard python packages that load data into a numpy array.
    Then you can convert this array into a ``torch.*Tensor``.
    
    -  For images, packages such as Pillow, OpenCV are useful
    -  For audio, packages such as scipy and librosa
    -  For text, either raw Python or Cython based loading, or NLTK and
       SpaCy are useful
    
    Specifically for vision, we have created a package called
    ``torchvision``, that has data loaders for common datasets such as
    Imagenet, CIFAR10, MNIST, etc. and data transformers for images, viz.,
    ``torchvision.datasets`` and ``torch.utils.data.DataLoader``.
    
    This provides a huge convenience and avoids writing boilerplate code.
    
    For this tutorial, we will use the CIFAR10 dataset.
    It has the classes: ‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’,
    ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’. The images in CIFAR-10 are of
    size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.
    
    .. figure:: /_static/img/cifar10.png
       :alt: cifar10
    
       cifar10
    
    
    Training an image classifier
    ----------------------------
    
    We will do the following steps in order:
    
    1. Load and normalizing the CIFAR10 training and test datasets using
       ``torchvision``
    2. Define a Convolutional Neural Network
    3. Define a loss function
    4. Train the network on the training data
    5. Test the network on the test data
    
    1. Loading and normalizing CIFAR10
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    
    Using ``torchvision``, it’s extremely easy to load CIFAR10.
    """
    import torch
    import torchvision
    import torchvision.transforms as transforms
    
    ########################################################################
    # The output of torchvision datasets are PILImage images of range [0, 1].
    # We transform them to Tensors of normalized range [-1, 1].
    
    transform = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    
    trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                              shuffle=True, num_workers=2)
    
    testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                           download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                             shuffle=False, num_workers=2)
    
    classes = ('plane', 'car', 'bird', 'cat',
               'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
    
    ########################################################################
    # Let us show some of the training images, for fun.
    
    import matplotlib.pyplot as plt
    import numpy as np
    
    # functions to show an image
    
    
    def imshow(img):
        img = img / 2 + 0.5     # unnormalize
        npimg = img.numpy()
        plt.imshow(np.transpose(npimg, (1, 2, 0)))
        plt.show()
    
    
    # get some random training images
    #dataiter = iter(trainloader)
    #images, labels = dataiter.next()
    
    # show images
    #imshow(torchvision.utils.make_grid(images))
    # print labels
    #print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
    
    
    ########################################################################
    # 2. Define a Convolutional Neural Network
    # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    # Copy the neural network from the Neural Networks section before and modify it to
    # take 3-channel images (instead of 1-channel images as it was defined).
    
    import torch.nn as nn
    import torch.nn.functional as F
    
    
    class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(16 * 5 * 5, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)
    
        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = x.view(-1, 16 * 5 * 5)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x
    
    
    net = Net()
    
    ########################################################################
    # 3. Define a Loss function and optimizer
    # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    # Let's use a Classification Cross-Entropy loss and SGD with momentum.
    
    import torch.optim as optim
    
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
    
    ########################################################################
    # 4. Train the network
    # ^^^^^^^^^^^^^^^^^^^^
    #
    # This is when things start to get interesting.
    # We simply have to loop over our data iterator, and feed the inputs to the
    # network and optimize.
    if __name__ == "__main__":  # this is needed on windows
        for epoch in range(2):  # loop over the dataset multiple times
    
            running_loss = 0.0
            for i, data in enumerate(trainloader, 0):
                # get the inputs
                inputs, labels = data
    
                # zero the parameter gradients
                optimizer.zero_grad()
    
                # forward + backward + optimize
                outputs = net(inputs)
                loss = criterion(outputs, labels)
                loss.backward()
                optimizer.step()
    
                # print statistics
                running_loss += loss.item()
                if i % 2000 == 1999:    # print every 2000 mini-batches
                    print('[%d, %5d] loss: %.3f' %(epoch + 1, i + 1, running_loss / 2000))
                    running_loss = 0.0
    
        print('Finished Training')
    
        ########################################################################
        # 5. Test the network on the test data
        # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        #
        # We have trained the network for 2 passes over the training dataset.
        # But we need to check if the network has learnt anything at all.
        #
        # We will check this by predicting the class label that the neural network
        # outputs, and checking it against the ground-truth. If the prediction is
        # correct, we add the sample to the list of correct predictions.
        #
        # Okay, first step. Let us display an image from the test set to get familiar.
    
        dataiter = iter(testloader)
        images, labels = dataiter.next()
    
        # print images
        imshow(torchvision.utils.make_grid(images))
        print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))
    
        ########################################################################
        # Okay, now let us see what the neural network thinks these examples above are:
    
        outputs = net(images)
    
        ########################################################################
        # The outputs are energies for the 10 classes.
        # Higher the energy for a class, the more the network
        # thinks that the image is of the particular class.
        # So, let's get the index of the highest energy:
        _, predicted = torch.max(outputs, 1)
    
        print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
                                    for j in range(4)))
    
        ########################################################################
        # The results seem pretty good.
        #
        # Let us look at how the network performs on the whole dataset.
    
        correct = 0
        total = 0
        with torch.no_grad():
            for data in testloader:
                images, labels = data
                outputs = net(images)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
    
        print('Accuracy of the network on the 10000 test images: %d %%' % (
            100 * correct / total))
    
        ########################################################################
        # That looks waaay better than chance, which is 10% accuracy (randomly picking
        # a class out of 10 classes).
        # Seems like the network learnt something.
        #
        # Hmmm, what are the classes that performed well, and the classes that did
        # not perform well:
    
        class_correct = list(0. for i in range(10))
        class_total = list(0. for i in range(10))
        with torch.no_grad():
            for data in testloader:
                images, labels = data
                outputs = net(images)
                _, predicted = torch.max(outputs, 1)
                c = (predicted == labels).squeeze()
                for i in range(4):
                    label = labels[i]
                    class_correct[label] += c[i].item()
                    class_total[label] += 1
    
    
        for i in range(10):
            print('Accuracy of %5s : %2d %%' % (
                classes[i], 100 * class_correct[i] / class_total[i]))
    
        ########################################################################
        # Okay, so what next?
        #
        # How do we run these neural networks on the GPU?
        #
        # Training on GPU
        # ----------------
        # Just like how you transfer a Tensor on to the GPU, you transfer the neural
        # net onto the GPU.
        #
        # Let's first define our device as the first visible cuda device if we have
        # CUDA available:
    
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    
        # Assume that we are on a CUDA machine, then this should print a CUDA device:
    
        print(device)
    
        ########################################################################
        # The rest of this section assumes that `device` is a CUDA device.
        #
        # Then these methods will recursively go over all modules and convert their
        # parameters and buffers to CUDA tensors:
        #
        # .. code:: python
        #
        #     net.to(device)
        #
        #
        # Remember that you will have to send the inputs and targets at every step
        # to the GPU too:
        #
        # .. code:: python
        #
        #         inputs, labels = inputs.to(device), labels.to(device)
        #
        # Why dont I notice MASSIVE speedup compared to CPU? Because your network
        # is realllly small.
        #
        # **Exercise:** Try increasing the width of your network (argument 2 of
        # the first ``nn.Conv2d``, and argument 1 of the second ``nn.Conv2d`` –
        # they need to be the same number), see what kind of speedup you get.
        #
        # **Goals achieved**:
        #
        # - Understanding PyTorch's Tensor library and neural networks at a high level.
        # - Train a small neural network to classify images
        #
        # Training on multiple GPUs
        # -------------------------
        # If you want to see even more MASSIVE speedup using all of your GPUs,
        # please check out :doc:`data_parallel_tutorial`.
        #
        # Where do I go next?
        # -------------------
        #
        # -  :doc:`Train neural nets to play video games </intermediate/reinforcement_q_learning>`
        # -  `Train a state-of-the-art ResNet network on imagenet`_
        # -  `Train a face generator using Generative Adversarial Networks`_
        # -  `Train a word-level language model using Recurrent LSTM networks`_
        # -  `More examples`_
        # -  `More tutorials`_
        # -  `Discuss PyTorch on the Forums`_
        # -  `Chat with other users on Slack`_
        #
        # .. _Train a state-of-the-art ResNet network on imagenet: https://github.com/pytorch/examples/tree/master/imagenet
        # .. _Train a face generator using Generative Adversarial Networks: https://github.com/pytorch/examples/tree/master/dcgan
        # .. _Train a word-level language model using Recurrent LSTM networks: https://github.com/pytorch/examples/tree/master/word_language_model
        # .. _More examples: https://github.com/pytorch/examples
        # .. _More tutorials: https://github.com/pytorch/tutorials
        # .. _Discuss PyTorch on the Forums: https://discuss.pytorch.org/
        # .. _Chat with other users on Slack: https://pytorch.slack.com/messages/beginner/

[https_github.com_pytorch_examples_issues_201]: https://github.com/pytorch/examples/issues/201
[https_stackoverflow.com_questions_18204782_runtimeerror-on-windows-trying-python-multiprocessing_18205006_18205006]: https://stackoverflow.com/questions/18204782/runtimeerror-on-windows-trying-python-multiprocessing/18205006#18205006