Python中运行并行任务技巧

yipeiwu_com6年前Python基础

示例

标准线程多进程,生产者/消费者示例:
Worker越多,问题越大

复制代码 代码如下:

# -*- coding: utf8 -*-

import os
import time
import Queue
import threading
from PIL import Image

def create_thumbnail(filename, size=(128, 128)):
    try:
        fp, fmt = filename.rsplit('.', 1)
        im = Image.open(filename)
        im.thumbnail(size, Image.ANTIALIAS)
        im.save((fp + '_'+'x'.join(str(i) for i in size) + '.'+fmt), im.format)
        return '%s thumbnail success!' % filename
    except Exception:
        return '%s thumbnail failed!' % filename


def get_image_paths(folder):
    return [os.path.join(folder, f) for f in os.listdir(folder) if 'png' in f]


class Consumer(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self._queue = queue

    def run(self):
        while True:
            content = self._queue.get()
            if isinstance(content, str) and content == 'quit':
                break
            respone = create_thumbnail(content)
        print 'Bye bye!'


def Producer():
    filenames = get_image_paths('images')
    queue = Queue.Queue()
    worker_threads = build_worker_pool(queue, 4)
    start_time = time.time()

    for filename in filenames:
        queue.put(filename)
    for worker in worker_threads:
        queue.put('quit')
    for worker in worker_threads:
        worker.join()

    print time.time() - start_time


def build_worker_pool(queue, size):
    workers = []
    for _ in range(size):
        worker = Consumer(queue)
        worker.start()
        workers.append(worker)
    return workers


if __name__ == '__main__':
    Producer()

map

Map能够处理集合按顺序遍历,最终将调用产生的结果保存在一个简单的集合当中。

复制代码 代码如下:

# -*- coding: utf8 -*-

import os
import time
from multiprocessing import Pool
from PIL import Image

def create_thumbnail(filename, size=(128, 128)):
    try:
        fp, fmt = filename.rsplit('.', 1)
        im = Image.open(filename)
        im.thumbnail(size, Image.ANTIALIAS)
        im.save((fp + '_'+'x'.join(str(i) for i in size) + '.'+fmt), im.format)
        return '%s thumbnail success!' % filename
    except Exception:
        return '%s thumbnail failed!' % filename


def get_image_paths(folder):
    return [os.path.join(folder, f) for f in os.listdir(folder) if 'png' in f]


def main():
    filenames = get_image_paths('images')
    start_time = time.time()
   
    pool = Pool(4)
    pool.map(create_thumbnail, filenames)
    pool.close()
    pool.join()

    print time.time() - start_time


if __name__ == '__main__':
    main()

相关文章

Python装饰器的执行过程实例分析

本文实例分析了Python装饰器的执行过程。分享给大家供大家参考,具体如下: 今天看到一句话:装饰器其实就是对闭包的使用,仔细想想,其实就是这回事,今天又看了下闭包,基本上算是弄明白了闭...

python根据经纬度计算距离示例

复制代码 代码如下:/** * 计算两点之间距离 * @param _lat1 - start纬度 * @param _lon1 - start经度&nbs...

python redis 删除key脚本的实例

单机模式 代码片段 安装 pip install redis import redis r = redis.Redis(host='192.168.1.3', port=6188,d...

使用Python实现微信提醒备忘录功能

使用Python实现微信提醒备忘录功能

最近工作比较繁杂,经常忘事,有时候记了备忘录结果却忘记看备忘录,但是微信是每天都会看的,于是就想到写 一个基于微信的提醒系统。总体思路是将待办事项记录到在线记事本,通过建立定时任务,每天...

Python3.5 Pandas模块之Series用法实例分析

Python3.5 Pandas模块之Series用法实例分析

本文实例讲述了Python3.5 Pandas模块之Series用法。分享给大家供大家参考,具体如下: 1、Pandas模块引入与基本数据结构 2、Series的创建...