python3 多进程共享变量实现方法

青旅半醒 2021-07-28 14:52 372阅读 0赞

今天同事反映一个问题让帮忙看一下：多进程共用一个变量，在一个进程中修改后，在另外的进程中并没有产生修改。

### 一、错误的实现方式 ###

最初以为是没添加global声明导致修改未生效，但实际操作发现global方式在多进程中也只能读不能写。错误示例代码如下：

import multiprocessing
    
    # 声明一个全局变量
    share_var = ["start flag"]
    
    def sub_process(process_name):
        # 企图像单个进程那样通过global声明使用全局变量
        global share_var
        share_var.append(process_name)
        # 但是很可惜，在多进程中这样引用只能读，修改其他进程不会同步改变
        for item in share_var:
            print(f"{process_name}-{item}")
        pass
    
    def main_process():
        process_list = []
        # 创建进程1
        process_name = "process 1"
        tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,))
        process_list.append(tmp_process)
        # 创建进程2
        process_name = "process 2"
        tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,))
        process_list.append(tmp_process)
        # 启动所有进程
        for process in process_list:
            process.start()
        for process in process_list:
            process.join()
    
    if __name__ == "__main__":
        main_process()

执行结果如下，可以看到进程1中的修改未表现在进程2中（不过要注意，和多线程一样，如果运算量再大一点进程1并不一定比进程2先执行）：

![20191210113310430.png][]

### 二、共享普通类型变量实现方法 ###

import multiprocessing
    
    # 不能将共享变量和共享锁定义成全局变量然后通过global引用那样会报错，只能传过来
    def sub_process(process_name,share_var,share_lock):
        # 获取锁
        share_lock.acquire()
        share_var.append(process_name)
        # 释放锁
        share_lock.release()
        for item in share_var:
            print(f"{process_name}-{item}")
        pass
    
    def main_process():
        # 单个值声明方式。typecode是进制类型，value是初始值
        # share_var = multiprocessing.Manager().Value(typecode, value)
        # 数组声明方式。typecode是数组变量中的变量类型，sequence是数组初始值
        # share_var = multiprocessing.Manager().Array(typecode, sequence)
        # 字典声明方式
        # share_var = multiprocessing.Manager().dict()
        # 列表声明方式
        share_var = multiprocessing.Manager().list()
        share_var.append("start flag")
        # 声明一个共享锁
        share_lock = multiprocessing.Manager().Lock()
        process_list = []
    
        process_name = "process 1"
        tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,share_var,share_lock))
        process_list.append(tmp_process)
    
        process_name = "process 2"
        tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,share_var,share_lock))
        process_list.append(tmp_process)
    
        for process in process_list:
            process.start()
        for process in process_list:
            process.join()
    
    if __name__ == "__main__":
        main_process()

执行结果如下，可以看到进程1中的修改已表现在进程2中（不过要注意，和多线程一样，如果运算量再大一点进程1并不一定比进程2先执行）：![20191210113405677.png][]

typecode如果是数值或单个字符，可为以下类型（注意有引号）：

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:middle;">Type Code</td> 
   <td style="vertical-align:middle;">C Type</td> 
   <td style="vertical-align:middle;">Python Type</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'c'</td> 
   <td style="vertical-align:middle;">char</td> 
   <td style="vertical-align:middle;">character</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'b'</td> 
   <td style="vertical-align:middle;">signed char</td> 
   <td style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'B'</td> 
   <td style="vertical-align:middle;">unsigned char</td> 
   <td style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'u'</td> 
   <td style="vertical-align:middle;">Py_UNICODE</td> 
   <td style="vertical-align:middle;">unicode character</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'h'</td> 
   <td style="vertical-align:middle;">signed short</td> 
   <td style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'H'</td> 
   <td style="vertical-align:middle;">unsigned short</td> 
   <td style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'i'</td> 
   <td style="vertical-align:middle;">signed int</td> 
   <td style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:middle;">'I'</td> 
   <td colspan="1" style="vertical-align:middle;">unsigned int</td> 
   <td colspan="1" style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:middle;">'l'</td> 
   <td colspan="1" style="vertical-align:middle;">signed long</td> 
   <td colspan="1" style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'L'</td> 
   <td style="vertical-align:middle;">unsigned long</td> 
   <td style="vertical-align:middle;">int</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;">'f'</td> 
   <td style="vertical-align:middle;">float</td> 
   <td style="vertical-align:middle;">float</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:middle;">'d'</td> 
   <td colspan="1" style="vertical-align:middle;">double</td> 
   <td colspan="1" style="vertical-align:middle;">flo</td> 
  </tr> 
 </tbody> 
</table>

如果是字符串类型，typecode可为以下第一列形式（注意无引号）：

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:middle;"><strong>ctypes type</strong></td> 
   <td style="vertical-align:middle;"><strong>C type</strong></td> 
   <td style="vertical-align:middle;"><strong>Python type</strong></td> 
  </tr> 
  <tr> 
   <td style="vertical-align:middle;"> <p>c_bool</p> </td> 
   <td style="vertical-align:middle;">_Bool</td> 
   <td style="vertical-align:middle;">bool (1)</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:top;">char</td> 
   <td style="vertical-align:top;">&nbsp;char</td> 
   <td style="vertical-align:top;">1-character string</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:top;">c_wchar</td> 
   <td style="vertical-align:top;">wchar_t</td> 
   <td style="vertical-align:top;">1-character unicode string</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_byte</td> 
   <td colspan="1" style="vertical-align:top;">char</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_ubyte</td> 
   <td colspan="1" style="vertical-align:top;">unsigned char</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td style="vertical-align:top;">c_short</td> 
   <td style="vertical-align:top;">short</td> 
   <td style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_ushort</td> 
   <td colspan="1" style="vertical-align:top;">unsigned short</td> 
   <td style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_int</td> 
   <td colspan="1" style="vertical-align:top;">int</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_uint</td> 
   <td colspan="1" style="vertical-align:top;">unsigned in</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_long</td> 
   <td colspan="1" style="vertical-align:top;">long</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_ulong</td> 
   <td colspan="1" style="vertical-align:top;">unsigned long</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_longlong</td> 
   <td colspan="1" style="vertical-align:top;">__int64 or long long</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_ulonglong</td> 
   <td colspan="1" style="vertical-align:top;">unsigned __int64 or unsigned long long</td> 
   <td colspan="1" style="vertical-align:top;">int/long</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_float</td> 
   <td colspan="1" style="vertical-align:top;">float</td> 
   <td colspan="1" style="vertical-align:top;">float</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_double</td> 
   <td colspan="1" style="vertical-align:top;">double</td> 
   <td colspan="1" style="vertical-align:top;">float</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_longdouble</td> 
   <td colspan="1" style="vertical-align:top;">long double</td> 
   <td colspan="1" style="vertical-align:top;">float</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_char_p</td> 
   <td colspan="1" style="vertical-align:top;">char * (NUL terminated)</td> 
   <td colspan="1" style="vertical-align:top;">string or None</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;"> <p>c_wchar_p</p> </td> 
   <td colspan="1" style="vertical-align:top;">wchar_t * (NUL terminated)</td> 
   <td colspan="1" style="vertical-align:top;">unicode or None</td> 
  </tr> 
  <tr> 
   <td colspan="1" style="vertical-align:top;">c_void_p</td> 
   <td colspan="1" style="vertical-align:top;">void *</td> 
   <td colspan="1" style="vertical-align:top;">int/long or None</td> 
  </tr> 
 </tbody> 
</table>

### 三、共享实例化对象实现方法 ###

同事还想共享一个文件对象，然后问上边的方法是不是只能共享字典、列表，没法共享对象。

回头一看，Value和Array中typecode要求是c语言中存在的类型，其他只有dict()和list()方法没有其他方法，所以似乎上边的方法共享实例化对象是不行的。

import multiprocessing
    import threading
    
    # 实例化一个全局文件对象
    file_obj = open("1.txt","a")
    share_lock = threading.Lock()
    
    def sub_process(process_name):
        global file_obj,share_lock
        share_lock.acquire()
        file_obj.writelines(f"{process_name}")
        share_lock.release()
        pass
    
    def main_process():
        process_list = []
        # 创建进程1
        process_name = "process 1"
        tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,))
        process_list.append(tmp_process)
        # 创建进程2
        process_name = "process 2"
        tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,))
        process_list.append(tmp_process)
        # 启动所有进程
        for process in process_list:
            process.start()
        for process in process_list:
            process.join()
    
    if __name__ == "__main__":
        main_process()

### 3.2 共享需要修改实例化对象实现方法----使用BaseManager ###

global方式不能修改变量（如要修改其成员变量），在大多时候也是可以了，但总让人觉得不是一种完美的实现方法。有没有可以修改的实现方法呢，答案是有的，可以使用BaseManager。示例代码如下。

import multiprocessing
    from multiprocessing.managers import BaseManager
    import threading
    
    # 锁可以通过global也可以在Process中传无所谓
    share_lock = threading.Lock()
    
    # 定义一个要共享实例化对象的类
    class Test():
        def __init__(self):
            self.test_list = ["start flag"]
    
        def test_function(self,arg):
            self.test_list.append(arg)
    
        def print_test_list(self):
            for item in self.test_list:
                print(f"{item}")
    
    def sub_process(process_name,obj):
        global share_lock
        share_lock.acquire()
        obj.test_function(f"{process_name}")
        share_lock.release()
        obj.print_test_list()
        pass
    
    def main_process():
        # 如果是想注册open方法这样操作
        # manager = BaseManager()
        # # 一定要在start前注册，不然就注册无效
        # manager.register('open', open)
        # manager.start()
        # obj = manager.open("1.txt","a")
    
        # 为了更加直接我们直接以一个Test类的实例化对象来演示
        manager = BaseManager()
        # 一定要在start前注册，不然就注册无效
        manager.register('Test', Test)
        manager.start()
        obj = manager.Test()
    
        process_list = []
        # 创建进程1
        process_name = "process 1"
        tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,obj))
        process_list.append(tmp_process)
        # 创建进程2
        process_name = "process 2"
        tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,obj))
        process_list.append(tmp_process)
        # 启动所有进程
        for process in process_list:
            process.start()
        for process in process_list:
            process.join()
    
    
    if __name__ == "__main__":
        main_process()

执行结果如下，可以看到进程1中的修改已表现在进程2中（不过要注意，和多线程一样，如果运算量再大一点进程1并不一定比进程2先执行）：

![format_png][]

[20191210113310430.png]: /images/20210728/0e90f4d90e604737a61d815d7ac15fa9.png
[20191210113405677.png]: /images/20210728/fc625890df37474fad4f87014219a080.png
[format_png]: /images/20210728/3cec917443494ce680f5f070c803f81c.png