基于scrapy实现的简单蜘蛛采集程序

yipeiwu_com7年前Python基础

本文实例讲述了基于scrapy实现的简单蜘蛛采集程序。分享给大家供大家参考。具体如下:

# Standard Python library imports
# 3rd party imports
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
# My imports
from poetry_analysis.items import PoetryAnalysisItem
HTML_FILE_NAME = r'.+\.html'
class PoetryParser(object):
  """
  Provides common parsing method for poems formatted this one specific way.
  """
  date_pattern = r'(\d{2} \w{3,9} \d{4})'
 
  def parse_poem(self, response):
    hxs = HtmlXPathSelector(response)
    item = PoetryAnalysisItem()
    # All poetry text is in pre tags
    text = hxs.select('//pre/text()').extract()
    item['text'] = ''.join(text)
    item['url'] = response.url
    # head/title contains title - a poem by author
    title_text = hxs.select('//head/title/text()').extract()[0]
    item['title'], item['author'] = title_text.split(' - ')
    item['author'] = item['author'].replace('a poem by', '')
    for key in ['title', 'author']:
      item[key] = item[key].strip()
    item['date'] = hxs.select("//p[@class='small']/text()").re(date_pattern)
    return item
class PoetrySpider(CrawlSpider, PoetryParser):
  name = 'example.com_poetry'
  allowed_domains = ['www.example.com']
  root_path = 'someuser/poetry/'
  start_urls = ['http://www.example.com/someuser/poetry/recent/',
         'http://www.example.com/someuser/poetry/less_recent/']
  rules = [Rule(SgmlLinkExtractor(allow=[start_urls[0] + HTML_FILE_NAME]),
                  callback='parse_poem'),
       Rule(SgmlLinkExtractor(allow=[start_urls[1] + HTML_FILE_NAME]),
                  callback='parse_poem')]

希望本文所述对大家的Python程序设计有所帮助。

相关文章

python使用rsa非对称加密过程解析

python使用rsa非对称加密过程解析

这篇文章主要介绍了python使用rsa非对称加密过程解析,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友可以参考下 1、安装rsa 支持pyth...

Python进程间通信 multiProcessing Queue队列实现详解

一、进程间通信 IPC(Inter-Process Communication) IPC机制:实现进程之间通讯 管道:pipe 基于共享的内存空间 队列:pipe+锁的概念--->...

在python中将list分段并保存为array类型的方法

如下所示: list=[1,2,3,4,5,6,7,8,9,0,11,0,13,14,15,16,17,18,19,20] #把list分为长度为5的4段 for j in rang...

python与mysql数据库交互的实现

python与mysql数据库交互的实现

1、安装pymysql库 如果你想要使用python操作MySQL数据库,就必须先要安装pymysql库,这个库的安装很简单,直接使用pip install pymysql;进行安装。...

python 不同方式读取文件速度不同的实例

1、按行读取较慢较耗时: srcFiles = open('inputFile.txt', 'r') for file_path in srcFiles: file_path...