scrapy-deltafetch实现增量爬取

Dear 丶 2022-12-21 11:59 131阅读 0赞

详情:[https://blog.csdn.net/zsl10/article/details/52885597][https_blog.csdn.net_zsl10_article_details_52885597]

安装:Berkeley DB

\# cd /usr/local/src

\# wget http://download.oracle.com/berkeley-db/db-4.7.25.NC.tar.gz

\# tar zxvf db-4.7.25.NC.tar.gz \# cd build\_unix

\# ../dist/configure

\# make&&make install

安装bsddb3

` pip install bsddb3`

安装scrapy-deltafetch

# ` pip install scrapy-deltafetch` #

# 安装scrapy-magicfields #

# ` pip install scrapy-magicfields` #

settings.py设置

SPIDER\_MIDDLEWARES = \{  ‘scrapy\_deltafetch.DeltaFetch’: 100  *\}  *

*    DELTAFETCH\_ENABLED = True*

*运行爬虫项目*

*scrapy crawl* spider\_name

# *如果想重新爬取之前已经爬取过的链接，可以通过重置DeltaFetch的缓存来实现      scrapy crawl spider\_name -a deltafetch\_reset=1* #

前言

在之前的文章中我们都是对目标站点进行全量爬取，只要爬虫run起来就会对所有的链接都爬取一遍，这其实是很傻的做法，因为很多情况下我们并不需要爬取已经爬过的链接，除非你需要定期更新这个链接对应页面上的数据。好了，回归正题，本文介绍scrapy使用scrapy-deltafetch这个插件来实现增量爬取，这里以爬取【美食杰】上的菜谱信息为例。

正文

安装scrapy-deltafetch

pip install scrapy-deltafetch

如安装过程报错`Can't find a local Berkeley DB installation.`请参考：[http://jinbitou.net/2018/01/27/2579.html][http_jinbitou.net_2018_01_27_2579.html]

新建项目和爬虫

$ scrapy startproject meishijie PycharmProjects/meishijie
    $ cd PycharmProjects/meishijie
    $ scrapy genspider meishi meishij.net

settings.py

用Pycharm打开生成的项目，编辑`settings.py`，添加如下内容：

> SPIDER\_MIDDLEWARES = \{    
> ‘scrapy\_deltafetch.DeltaFetch’: 100    
> \}    
> DELTAFETCH\_ENABLED = True

meishi.py

编辑爬虫文件`meishi.py`，因为是测试我就不写太多逻辑了，大家知道就好。

# -*- coding: utf-8 -*-
    import scrapy
    
    class MeishiSpider(scrapy.Spider):
        name = 'meishi'
        allowed_domains = ['meishij.net']
        start_urls = ['http://www.meishij.net/yaoshanshiliao/jibingtiaoli/weiyan/']
    
        def parse(self, response):
            for ms in response.xpath("//div[contains(@class,'i_w')]"):
                item = {}
                title = ms.xpath("div/div/strong/text()").extract_first()
                hot = ms.xpath("div/div/span/text()").extract_first()
                item["title"] = title
                item["hot"] = hot
                yield item
            next_page = response.xpath("//a[@class='next']/@href").extract_first()
            print("下一页：", next_page)
            if next_page:
                yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

如上，`items.py`，`pipelines.py`，`middlewares.py`不做任何改动，保持默认就好。

运行

运行如下命令就可以看到scrapy已经愉快地跑起来了

$ scrapy crawl meishi

*  1

很快，所有链接已经爬取完毕，查看运行日志

2018-01-27 14:11:18 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'deltafetch/stored': 1500,
    'downloader/request_bytes': 25530,
    'downloader/request_count': 76,
    'downloader/request_method_count/GET': 76,
    'downloader/response_bytes': 1280237,
    'downloader/response_count': 76,
    'downloader/response_status_count/200': 76,
    'finish_reason': 'finished',
    'finish_time': datetime.datetime(2018, 1, 27, 6, 11, 18, 772936),
    'item_scraped_count': 1500,
    'log_count/DEBUG': 1577,
    'log_count/INFO': 7,
    'memusage/max': 52551680,
    'memusage/startup': 52551680,
    'request_depth_max': 74,
    'response_received_count': 76,
    'scheduler/dequeued': 75,
    'scheduler/dequeued/memory': 75,
    'scheduler/enqueued': 75,
    'scheduler/enqueued/memory': 75,
    'start_time': datetime.datetime(2018, 1, 27, 6, 10, 55, 141746)}
    2018-01-27 14:11:18 [scrapy.core.engine] INFO: Spider closed (finished)

可以知道，scrapy共发起了包括入口在内的76次请求，爬取了1500个item，deltafetch存储了这1500个item对应请求的指纹信息。

测试增量爬取

再次运行`scrapy crawl meishi`命令

2018-01-27 14:27:39 [scrapy_deltafetch.middleware] INFO: Ignoring already visited: <GET http://www.meishij.net/shiliao.php?st=3&cid=178&sortby=update&page=2>
    2018-01-27 14:27:39 [scrapy.core.engine] INFO: Closing spider (finished)
    2018-01-27 14:27:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'deltafetch/skipped': 1,
    'deltafetch/stored': 20,
    'downloader/request_bytes': 471,
    'downloader/request_count': 2,
    'downloader/request_method_count/GET': 2,
    'downloader/response_bytes': 17879,
    'downloader/response_count': 2,
    'downloader/response_status_count/200': 2,
    'finish_reason': 'finished',
    'finish_time': datetime.datetime(2018, 1, 27, 6, 27, 39, 940365),
    'item_scraped_count': 20,
    'log_count/DEBUG': 23,
    'log_count/INFO': 8,
    'memusage/max': 52420608,
    'memusage/startup': 52420608,
    'request_depth_max': 1,
    'response_received_count': 2,
    'scheduler/dequeued': 1,
    'scheduler/dequeued/memory': 1,
    'scheduler/enqueued': 1,
    'scheduler/enqueued/memory': 1,
    'start_time': datetime.datetime(2018, 1, 27, 6, 27, 39, 547330)}
    2018-01-27 14:27:39 [scrapy.core.engine] INFO: Spider closed (finished)

*  可以看到scrapy除了入口的请求外，之前请求过的链接都已经跳过，Done！

## 补充 ##

如果想重新爬取之前已经爬取过的链接，可以通过重置DeltaFetch的缓存来实现，具体做法是给你的爬虫传一个参数`deltafetch_reset`，例如：

$ scrapy crawl meishi -a deltafetch_reset=1

参考：[http://jinbitou.net/2018/01/27/2581.html][http_jinbitou.net_2018_01_27_2581.html]

[https_blog.csdn.net_zsl10_article_details_52885597]: https://blog.csdn.net/zsl10/article/details/52885597
[http_jinbitou.net_2018_01_27_2579.html]: http://jinbitou.net/2018/01/27/2579.html
[http_jinbitou.net_2018_01_27_2581.html]: http://jinbitou.net/2018/01/27/2581.html