Python自定义scrapy中间模块避免重复采集的方法

yipeiwu_com5年前Python基础

本文实例讲述了Python自定义scrapy中间模块避免重复采集的方法。分享给大家供大家参考。具体如下:

from scrapy import log
from scrapy.http import Request
from scrapy.item import BaseItem
from scrapy.utils.request import request_fingerprint
from myproject.items import MyItem
class IgnoreVisitedItems(object):
  """Middleware to ignore re-visiting item pages if they
  were already visited before. 
  The requests to be filtered by have a meta['filter_visited']
  flag enabled and optionally define an id to use 
  for identifying them, which defaults the request fingerprint,
  although you'd want to use the item id,
  if you already have it beforehand to make it more robust.
  """
  FILTER_VISITED = 'filter_visited'
  VISITED_ID = 'visited_id'
  CONTEXT_KEY = 'visited_ids'
  def process_spider_output(self, response, result, spider):
    context = getattr(spider, 'context', {})
    visited_ids = context.setdefault(self.CONTEXT_KEY, {})
    ret = []
    for x in result:
      visited = False
      if isinstance(x, Request):
        if self.FILTER_VISITED in x.meta:
          visit_id = self._visited_id(x)
          if visit_id in visited_ids:
            log.msg("Ignoring already visited: %s" % x.url,
                level=log.INFO, spider=spider)
            visited = True
      elif isinstance(x, BaseItem):
        visit_id = self._visited_id(response.request)
        if visit_id:
          visited_ids[visit_id] = True
          x['visit_id'] = visit_id
          x['visit_status'] = 'new'
      if visited:
        ret.append(MyItem(visit_id=visit_id, visit_status='old'))
      else:
        ret.append(x)
    return ret
  def _visited_id(self, request):
    return request.meta.get(self.VISITED_ID) or request_fingerprint(request)

希望本文所述对大家的Python程序设计有所帮助。

相关文章

Python3安装Pillow与PIL的方法

关于Pillow与PIL PIL(Python Imaging Library)是Python一个强大方便的图像处理库,名气也比较大。不过只支持到Python 2.7。 PIL官方网站:...

python求众数问题实例

本文实例讲述了python求众数问题的方法,是一个比较典型的应用。分享给大家供大家参考。具体如下: 问题描述: 多重集中重数最大的元素称为众数...就是一个可以有重复元素的集合,在这个集...

Python通过future处理并发问题

Python通过future处理并发问题

future初识 通过下面脚本来对future进行一个初步了解: 例子1:普通通过循环的方式 import os import time import sys import re...

使用PDB模式调试Python程序介绍

以前在windows下一直用的idel带的功能调试python程序,在linux下没调试过。(很多时候只是print)就从网上查找一下~ 方法: 复制代码 代码如下: python -m...

Python实现的简单读写csv文件操作示例

Python实现的简单读写csv文件操作示例

本文实例讲述了Python实现的简单读写csv文件操作。分享给大家供大家参考,具体如下: python中有一个读写csv文件的包,直接import csv即可 新建test.csv 1....