基于scrapy实现的简单蜘蛛采集程序

yipeiwu_com7年前Python基础

本文实例讲述了基于scrapy实现的简单蜘蛛采集程序。分享给大家供大家参考。具体如下:

# Standard Python library imports
# 3rd party imports
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
# My imports
from poetry_analysis.items import PoetryAnalysisItem
HTML_FILE_NAME = r'.+\.html'
class PoetryParser(object):
  """
  Provides common parsing method for poems formatted this one specific way.
  """
  date_pattern = r'(\d{2} \w{3,9} \d{4})'
 
  def parse_poem(self, response):
    hxs = HtmlXPathSelector(response)
    item = PoetryAnalysisItem()
    # All poetry text is in pre tags
    text = hxs.select('//pre/text()').extract()
    item['text'] = ''.join(text)
    item['url'] = response.url
    # head/title contains title - a poem by author
    title_text = hxs.select('//head/title/text()').extract()[0]
    item['title'], item['author'] = title_text.split(' - ')
    item['author'] = item['author'].replace('a poem by', '')
    for key in ['title', 'author']:
      item[key] = item[key].strip()
    item['date'] = hxs.select("//p[@class='small']/text()").re(date_pattern)
    return item
class PoetrySpider(CrawlSpider, PoetryParser):
  name = 'example.com_poetry'
  allowed_domains = ['www.example.com']
  root_path = 'someuser/poetry/'
  start_urls = ['http://www.example.com/someuser/poetry/recent/',
         'http://www.example.com/someuser/poetry/less_recent/']
  rules = [Rule(SgmlLinkExtractor(allow=[start_urls[0] + HTML_FILE_NAME]),
                  callback='parse_poem'),
       Rule(SgmlLinkExtractor(allow=[start_urls[1] + HTML_FILE_NAME]),
                  callback='parse_poem')]

希望本文所述对大家的Python程序设计有所帮助。

相关文章

Python中垃圾回收和del语句详解

Python中的垃圾回收算法是采用引用计数, 当一个对象的引用计数为0时, Python的垃圾回收机制就会将对象回收 a = "larry" b = a larry这个字符串对象,...

浅谈python的dataframe与series的创建方法

如下所示: # -*- coding: utf-8 -*- import numpy as np import pandas as pd def main(): s = pd.S...

Django后台获取前端post上传的文件方法

Django后台获取前端post上传的文件方法

由于初学Django所以对于其特有的表单模板和models中的filefiled并不是特别熟悉。因此在本次开发中还是依照以往自己在html中写好图片提交表单: 注意在<form&...

Python将主机名转换为IP地址的方法

关于主机名转IP地址只记住两点即可: 1、英特网协议一直都是用4字节的IP将包转发给目的地; 2、至于主机名是如何转换成IP地址的,这是操作系统关心的事情; import socke...

Python在OpenCV里实现极坐标变换功能

Python在OpenCV里实现极坐标变换功能

在中学里学习过直角坐标系,也叫做笛卡尔坐标系,它是正交坐标系,不过也学习过极坐标系,这种坐标系比较适合大炮发射的场合。极坐标系的定义如下: 在 平面内取一个定点O, 叫极点,引一条射线O...