python 多进程并行编程 ProcessPoolExecutor的实现

yipeiwu_com5年前Python基础

使用 ProcessPoolExecutor

from concurrent.futures import ProcessPoolExecutor, as_completed
import random

斐波那契数列

当 n 大于 30 时抛出异常

def fib(n):
  if n > 30:
    raise Exception('can not > 30, now %s' % n)
  if n <= 2:
    return 1
  return fib(n-1) + fib(n-2)

准备数组

nums = [random.randint(0, 33) for _ in range(0, 10)]
'''
[13, 17, 0, 22, 19, 33, 7, 12, 8, 16]
'''

方案一:submit

submit 输出结果按照子进程执行结束的先后顺序,不可控

 with ProcessPoolExecutor(max_workers=3) as executor:
    futures = {executor.submit(fib, n):n for n in nums}
    for f in as_completed(futures):
      try:
        print('fib(%s) result is %s.' % (futures[f], f.result()))
      except Exception as e:
        print(e)
'''
fib(13) result is 233.
fib(17) result is 1597.
fib(0) result is 1.
fib(22) result is 17711.
fib(19) result is 4181.
can not > 30, now 33
fib(7) result is 13.
fib(12) result is 144.
fib(8) result is 21.
fib(16) result is 987.

'''

等价写法:

 with ProcessPoolExecutor(max_workers=3) as executor:
    futures = {}
    for n in nums:
      job = executor.submit(fib, n)
      futures[job] = n

    for job in as_completed(futures):
      try:
        re = job.result()
        n = futures[job]
        print('fib(%s) result is %s.' % (n, re))
      except Exception as e:
        print(e)
'''
fib(13) result is 233.
fib(17) result is 1597.
fib(0) result is 1.
fib(22) result is 17711.
can not > 30, now 33
fib(7) result is 13.
fib(19) result is 4181.
fib(8) result is 21.
fib(12) result is 144.
fib(16) result is 987.
'''

方案二:map

map 输出结果按照输入数组的顺序

缺点:某一子进程异常会导致整体中断

 with ProcessPoolExecutor(max_workers=3) as executor:
    try:
      results = executor.map(fib, nums)
      for num, result in zip(nums, results):
        print('fib(%s) result is %s.' % (num, result))
    except Exception as e:
      print(e)
'''
fib(13) result is 233.
fib(17) result is 1597.
fib(0) result is 1.
fib(22) result is 17711.
fib(19) result is 4181.
can not > 30, now 33
'''

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持【听图阁-专注于Python设计】。

相关文章

详解Python 中sys.stdin.readline()的用法

之前在Python中输入都是用的input(),但是看到大家都用sys.stdin.readline(),没办法那我也得用. python3中使用sys.stdin.readline()...

python儿童学游戏编程知识点总结

python爬虫基本告一段落,琢磨搞点其他的,正好在网上看到一个帖子,一个外国13岁小朋友用python写的下棋程序,内容详细,也有意思,拿来练手。 13岁啊。。 我这年纪还在敲 dir...

Python jieba库用法及实例解析

Python jieba库用法及实例解析

1、jieba库基本介绍 (1)、jieba库概述 jieba是优秀的中文分词第三方库 - 中文文本需要通过分词获得单个的词语 - jieba是优秀的中文分词第三方库,需要额...

Python使用稀疏矩阵节省内存实例

推荐系统中经常需要处理类似user_id, item_id, rating这样的数据,其实就是数学里面的稀疏矩阵,scipy中提供了sparse模块来解决这个问题,但scipy.spar...

Django中的FBV和CBV用法详解

Django中的FBV和CBV用法详解

FBV FBV,即 func base views,函数视图,在视图里使用函数处理请求。 以用户注册代码为例, 使用两个函数完成注册 初级注册代码 def register(req...