Python多进程机制实例详解_Python

本文实例讲述了Python 多进程机制。分享给大家供大家参考。具体如下：

在以前只是接触过PYTHON的多线程机制，今天搜了一下多进程，相关文章好像不是特别多。看了几篇，小试了一把。程序如下，主要内容就是通过PRODUCER读一个本地文件，一行一行的放到队列中去。然后会有相应的WORKER从队列中取出这些行。

									import multiprocessing

									import os

									import sys

									import Queue

									import time

									def writeQ(q,obj):

									    q.put(obj,True,None)

									    print "put size: ",q.qsize()

									def readQ(q):

									    ret = q.get(True,1)

									    print "get size: ",q.qsize()

									    return ret

									def producer(q):

									    time.sleep(5)  #让进行休息几秒 方便ps命令看到相关内容

									    pid = os.getpid()

									    handle_file = '/home/dwapp/joe.wangh/test/multiprocess/datafile'

									    with open(handle_file,'r') as f:   #with...as... 这个用法今天也是第一次看到的

									        for line in f:

									            print "producer <" ,pid , "> is doing: ",line

									            writeQ(q,line.strip())

									    q.close()

									def worker(q):

									    time.sleep(5)  #让进行休息几秒 方便ps命令看到相关内容

									    pid = os.getpid()

									    empty_count = 0

									    while True:

									        try:

									            task = readQ(q)

									            print "worker <" , pid , "> is doing: " ,task

									            '''

									            如果这里不休眠的话 一般情况下所有行都会被同一个子进程读取到 为了使实验效果更加清楚 在这里让每个进程读取完

									一行内容时候休眠5s 这样就可以让其他的进程到队列中进行读取

									            '''

									            time.sleep(5)  

									        except Queue.Empty:

									            empty_count += 1

									            if empty_count == 3:

									                print "queue is empty, quit"

									                q.close()

									                sys.exit(0)

									def main():

									    concurrence = 3

									    q = multiprocessing.Queue(10)

									    funcs = [producer , worker]

									    for i in range(concurrence-1):

									        funcs.append(worker)

									    for item in funcs:

									        print str(item)

									    nfuncs = range( len(funcs) )

									    processes = []    

									    for i in nfuncs:

									        p = multiprocessing.Process(target=funcs[i] , args=(q,))

									        processes.append(p)

									    print "concurrence worker is : ",concurrence," working start"

									    for i in nfuncs:

									        processes[i].start()

									    for i in nfuncs:

									        processes[i].join()

									    print "all DONE"

									if __name__ == '__main__':

									    main()

实验结果如下：

									dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>python 1.py

									<function producer at 0xb7b9141c>

									<function worker at 0xb7b91454>

									<function worker at 0xb7b91454>

									<function worker at 0xb7b91454>

									concurrence worker is : 3 working start

									producer < 28320 > is doing: line 1

									put size: 1

									producer < 28320 > is doing: line 2

									put size: 2

									producer < 28320 > is doing: line 3

									put size: 3

									producer < 28320 > is doing: line 4

									put size: 3

									producer < 28320 > is doing: line 5

									get size: 3

									put size: 4

									worker < 28321 > is doing: line 1

									get size: 3

									worker < 28322 > is doing: line 2

									get size: 2

									worker < 28323 > is doing: line 3

									get size: 1

									worker < 28321 > is doing: line 4

									get size: 0

									worker < 28322 > is doing: line 5

									queue is empty, quit

									queue is empty, quit

									queue is empty, quit

									all DONE

程序运行期间在另外一个窗口进行ps命令可以观测到一些进程的信息

									dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python

									dwapp  13735 11830 0 Nov20 pts/12  00:00:05 python

									dwapp  28319 27481 8 14:04 pts/0  00:00:00 python 1.py

									dwapp  28320 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28321 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28322 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28323 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28325 27849 0 14:04 pts/13  00:00:00 grep python

									dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python

									dwapp  13735 11830 0 Nov20 pts/12  00:00:05 python     #此时28320进程 也就是PRODUCER进程已经结束

									dwapp  28319 27481 1 14:04 pts/0  00:00:00 python 1.py

									dwapp  28321 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28322 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28323 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28328 27849 0 14:04 pts/13  00:00:00 grep python

									dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python

									dwapp  13735 11830 0 Nov20 pts/12  00:00:05 python

									dwapp  28319 27481 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28321 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28322 28319 0 14:04 pts/0  00:00:00 python 1.py

									dwapp  28323 28319 0 14:04 pts/0  00:00:00 [python] <defunct>  #这里应该是代表28323进程（WORKER）已经运行结束了

									dwapp  28331 27849 0 14:04 pts/13  00:00:00 grep python

									dwapp@pttest1:/home/dwapp/joe.wangh/test/multiprocess>ps -ef | grep python

									dwapp  13735 11830 0 Nov20 pts/12  00:00:05 python

									dwapp  28337 27849 0 14:05 pts/13  00:00:00 grep python

希望本文所述对大家的Python程序设计有所帮助。

Python多进程机制实例详解

相关文章

热门资讯