本文实例讲述了Python multiprocess pool模块报错pickling error问题解决方法。分享给大家供大家参考，具体如下：
问题
之前在调用class内的函数用multiprocessing模块的pool函数进行多线程处理的时候报了以下下错误信息:
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
查了下官方文档发现python默认只能pickle以下的类型:
- None, True, and False
- integers, floating point numbers, complex numbers
- strings, bytes, bytearrays
- tuples, lists, sets, and dictionaries containing only picklable objects
- functions defined at the top level of a module (using def, not lambda)
- built-in functions defined at the top level of a module
- classes that are defined at the top level of a module
- instances of such classes whose dict or the result of calling getstate() is picklable (see section -
- Pickling Class Instances for details).
函数只能pickle在顶层定义的函数，很明显的class内的函数无法被pickle因此会报错。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
import multiprocessingdef work(): # top-level 函数 print "work!"class Foo(): def work(self): # 非top-level函数 print "work"pool1 = multiprocessing.Pool(processes=4)foo = Foo()pool1.apply_async(foo.work)pool1.close()pool1.join()# 此时报错pool2 = multiprocessing.Pool(processes=4)pool2.apply_async(work)pool2.close()pool2.join()# 此时工作正常 |
解决方案
调用pathos包下的multiprocessing模块代替原生的multiprocessing。pathos中multiprocessing是用dill包改写过的，dill包可以将几乎所有python的类型都serialize，因此都可以被pickle。或者也可以自己用dill写一个（有点重复造轮子之嫌啊）
参考
1. https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function
2. https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled
3. https://github.com/uqfoundation/pathos
希望本文所述对大家Python程序设计有所帮助。
原文链接：https://www.cnblogs.com/arkenstone/p/7901129.html
