Python自定义scrapy中间模块避免重复采集的方法

yipeiwu_com5年前Python基础

本文实例讲述了Python自定义scrapy中间模块避免重复采集的方法。分享给大家供大家参考。具体如下:

from scrapy import log
from scrapy.http import Request
from scrapy.item import BaseItem
from scrapy.utils.request import request_fingerprint
from myproject.items import MyItem
class IgnoreVisitedItems(object):
  """Middleware to ignore re-visiting item pages if they
  were already visited before. 
  The requests to be filtered by have a meta['filter_visited']
  flag enabled and optionally define an id to use 
  for identifying them, which defaults the request fingerprint,
  although you'd want to use the item id,
  if you already have it beforehand to make it more robust.
  """
  FILTER_VISITED = 'filter_visited'
  VISITED_ID = 'visited_id'
  CONTEXT_KEY = 'visited_ids'
  def process_spider_output(self, response, result, spider):
    context = getattr(spider, 'context', {})
    visited_ids = context.setdefault(self.CONTEXT_KEY, {})
    ret = []
    for x in result:
      visited = False
      if isinstance(x, Request):
        if self.FILTER_VISITED in x.meta:
          visit_id = self._visited_id(x)
          if visit_id in visited_ids:
            log.msg("Ignoring already visited: %s" % x.url,
                level=log.INFO, spider=spider)
            visited = True
      elif isinstance(x, BaseItem):
        visit_id = self._visited_id(response.request)
        if visit_id:
          visited_ids[visit_id] = True
          x['visit_id'] = visit_id
          x['visit_status'] = 'new'
      if visited:
        ret.append(MyItem(visit_id=visit_id, visit_status='old'))
      else:
        ret.append(x)
    return ret
  def _visited_id(self, request):
    return request.meta.get(self.VISITED_ID) or request_fingerprint(request)

希望本文所述对大家的Python程序设计有所帮助。

相关文章

python 检查数据中是否有缺失值,删除缺失值的方式

# 检查数据中是否有缺失值 np.isnan(train).any() Flase:表示对应特征的特征值中无缺失值 True:表示有缺失值 通常情况下删除行,使用参数axis =...

Python实现二维数组输出为图片

对于二维数组,img_mask [[ 0 0 0 ..., 7 7 7] [ 0 0 0 ..., 7 7 7] [ 0 0 0 ..., 7 7 7] ..., [266...

wxPython 入门教程

wxPython 入门教程

这篇文章是关于 wxPython,但 wxPython 实际是两件事物的组合体:Python 脚本语言和 GUI 功能的 wxWindows 库(关于 wxWindows 的介绍,请参阅...

PyQT5 emit 和 connect的用法详解

PyQT5 emit 和 connect的用法详解

对于PyQT4, PyQT5在一些使用上有着比较明显的变化有很大的变化,让人惊讶是在emit和connect上的一些变化比较有意思,相信也是QT为了更好的和Python相结合做的改进。...

Python中统计函数运行耗时的方法

本文实例讲述了Python中统计函数运行耗时的方法。分享给大家供大家参考。具体实现方法如下: import time def time_me(fn): def _wrapper(...