爬虫利器Scrapy框架：1:概要介绍
Scrapy是使用python实现的一个web抓取框架，这篇文章将对Scrapy的概要、安装进行说明，并结合scrapy shell获取页面的title的简单示例来获取scrapy的直观使用感受。
概要信息
Scrapy是使用python实现的一个web抓取框架, 非常适合用于网站数据爬取、结构化数据提取等操作，相较于通用搜索为目的的Apache Nutch更加小巧和灵活，概要信息如下表所示：
| 项目 | 说明 |
|---|---|
| 官网 | https://scrapy.org/ |
| 开源/闭源 | 开源 |
| 源码管理地址 | https://github.com/scrapy/scrapy |
| 开发语言 | python |
| 当前稳定版本 | 1.13.0 （2019/03/19） |
安装
使用pip即可直接安装Scrapy，执行命令如下所示：
执行命令：pip install scrapy
本文使用python3和python并存的环境，使用pip3进行安装, 安装日志如下所示：
liumiaocn:scrapy liumiao$ pip3 install scrapyCollecting scrapyDownloading...省略Successfully built protego PyDispatcher zope.interfaceInstalling collected packages: six, pycparser, cffi, cryptography, pyasn1, pyasn1-modules, attrs, service-identity, protego, cssselect, pyOpenSSL, w3lib, PyDispatcher, incremental, constantly, Automat, PyHamcrest, zope.interface, idna, hyperlink, Twisted, lxml, parsel, queuelib, scrapySuccessfully installed Automat-20.2.0 PyDispatcher-2.0.5 PyHamcrest-2.0.2 Twisted-20.3.0 attrs-19.3.0 cffi-1.14.0 constantly-15.1.0 cryptography-2.8 cssselect-1.1.0 hyperlink-19.0.0 idna-2.9 incremental-17.5.0 lxml-4.5.0 parsel-1.5.2 protego-0.1.16 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 queuelib-1.5.0 scrapy-2.0.1 service-identity-18.1.0 six-1.14.0 w3lib-1.21.0 zope.interface-5.0.1liumiaocn:scrapy liumiao$
版本确认
liumiaocn:scrapy liumiao$ scrapy -hScrapy 2.0.1 - no active projectUsage:scrapy <command> [options] [args]Available commands:bench Run quick benchmark testfetch Fetch a URL using the Scrapy downloadergenspider Generate new spider using pre-defined templatesrunspider Run a self-contained spider (without creating a project)settings Get settings valuesshell Interactive scraping consolestartproject Create new projectversion Print Scrapy versionview Open URL in browser, as seen by Scrapy[ more ] More commands available when run from project directoryUse "scrapy <command> -h" to see more info about a commandliumiaocn:scrapy liumiao$
获取页面的标题信息
爬虫实际上是对HTML进行的处理，最为简单的确认Scrapy的功能的示例方式是通过scrapy shell来进行，scrapy shell提供了一种交互式的方式进行数据的抓取，也可以用于抓取的调试。
示例说明：希望获取Scrapy官网主页的标题信息，页面如下所示
步骤1: 执行scrapy shell
执行如下示例命令：
执行命令：scrapy shell https://scrapy.org/
liumiaocn:scrapy liumiao$ scrapy shell https://scrapy.org/2020-03-28 05:38:09 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: scrapybot)2020-03-28 05:38:09 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.5 (default, Nov 1 2019, 02:16:32) - [Clang 11.0.0 (clang-1100.0.33.8)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Darwin-19.2.0-x86_64-i386-64bit2020-03-28 05:38:09 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor2020-03-28 05:38:09 [scrapy.crawler] INFO: Overridden settings:{'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter','LOGSTATS_INTERVAL': 0}2020-03-28 05:38:09 [scrapy.extensions.telnet] INFO: Telnet Password: 5e36afd357190e932020-03-28 05:38:09 [scrapy.middleware] INFO: Enabled extensions:['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage']2020-03-28 05:38:09 [scrapy.middleware] INFO: Enabled downloader middlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']2020-03-28 05:38:09 [scrapy.middleware] INFO: Enabled spider middlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']2020-03-28 05:38:09 [scrapy.middleware] INFO: Enabled item pipelines:[]2020-03-28 05:38:09 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:60232020-03-28 05:38:09 [scrapy.core.engine] INFO: Spider opened2020-03-28 05:38:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://scrapy.org/> (referer: None)[s] Available Scrapy objects:[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)[s] crawler <scrapy.crawler.Crawler object at 0x1073043d0>[s] item {}[s] request <GET https://scrapy.org/>[s] response <200 https://scrapy.org/>[s] settings <scrapy.settings.Settings object at 0x1075d05d0>[s] spider <DefaultSpider 'default' at 0x107acbd90>[s] Useful shortcuts:[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)[s] fetch(req) Fetch a scrapy.Request and update local objects[s] shelp() Shell help (print this help)[s] view(response) View response in a browser>>>
步骤2: 通过response.css获取title
输入response.css(‘title’)，回车即可看到输出的信息中的title内容
>>> response.css('title')[<Selector xpath='descendant-or-self::title' data='<title>Scrapy | A Fast and Powerful S...'>]>>>
进一步获取title的详细信息
>>> response.css('title').extract_first()'<title>Scrapy | A Fast and Powerful Scraping and Web Crawling Framework</title>'>>>
怼烎@
迷南。
水深无声
小灰灰
忘是亡心i
朱雀
还没有评论，来说两句吧...