Scrapy安装及项目启动血泪辛酸史

深碍√TFBOYSˉ_ 2022-03-26 04:06 321阅读 0赞

## 导读 ##

作为python新手，更是一名scrapy的菜鸟。在这里简单的吐槽Scrapy官方文档两句，对于官方文档的指南更多的需要是参考而不是照搬，如果你放弃了前者选择了后者，就会像我一样，在前期阶段超级想放弃Scrapy的学习，还真是没入门就想放弃，特别的尴尬。

#### 简单叙述下压垮你信心的几个大坑 ####

*  使用pip安装scrapy
 *  依赖报错信息百度铁定搜不到，CSDN、segmentfalt搜了同样的报错问题没人回答
 *  安装好scrapy后，启动项目报错
 *  报错后不知道是代码的问题，配置的问题，还是安装过程中出现问题
 *  报错信息收到了github的issue倒是有人问，就是没人回答
 *  环境配置的路途中，还真是路漫漫，其修远兮。有人带真的很重要

### 一、安装启动 ###

1.  windows下强烈建议使用 Anaconda 这个第三方软件安装  
    Anaconda和文档下载地址：[http://www.scrapyd.cn/doc/124.html][http_www.scrapyd.cn_doc_124.html]  
    注：如果你不小心已经使用 `pip install scrapy` 但是报错了,那么骚年别慌！还有补救的方法：  
    （1）[《pip安装scrapy所需要的所有依赖》][pip_scrapy]  
    （2）[《pip install twisted 报错看这个》][pip install twisted]
2.  linux下载安装就会简单很多，至少不会有一些很奇怪的错误，试着使用`pip install scrapy`就可以了，如果遇到问题上面的几个方法就可以解决
3.  启动项目：`scrapy crawl spider_name`报错如下：

D:\me\python\scrapy\douban\douban\spiders>scrapy crawl douban_spider
    2019-01-15 14:34:28 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: douban)
    2019-01-15 14:34:28 [scrapy.utils.log] INFO: Versions: lxml 4.3.0.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.5rc1 (v3.6.5rc1:f03c5148cf, Mar 14 2018, 03:12:11) [MSC v.1913 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018), cryptography 2.4.2, Platform Windows-10-10.0.16299-SP0
    2019-01-15 14:34:28 [scrapy.crawler] INFO: Overridden settings: {
        'BOT_NAME': 'douban', 'DOWNLOAD_DELAY': 0.5, 'NEWSPIDER_MODULE': 'douban.spiders', 'SPIDER_MODULES': ['douban.spiders']}
    2019-01-15 14:34:28 [scrapy.middleware] INFO: Enabled extensions:
    ['scrapy.extensions.corestats.CoreStats',
     'scrapy.extensions.telnet.TelnetConsole',
     'scrapy.extensions.logstats.LogStats']
    2019-01-15 14:34:29 [scrapy.middleware] INFO: Enabled downloader middlewares:
    ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
     'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
     'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
     'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
     'scrapy.downloadermiddlewares.retry.RetryMiddleware',
     'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
     'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
     'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
     'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
     'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
     'scrapy.downloadermiddlewares.stats.DownloaderStats']
    2019-01-15 14:34:29 [scrapy.middleware] INFO: Enabled spider middlewares:
    ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
     'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
     'scrapy.spidermiddlewares.referer.RefererMiddleware',
     'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
     'scrapy.spidermiddlewares.depth.DepthMiddleware']
    2019-01-15 14:34:29 [scrapy.middleware] INFO: Enabled item pipelines:
    []
    2019-01-15 14:34:29 [scrapy.core.engine] INFO: Spider opened
    2019-01-15 14:34:29 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2019-01-15 14:34:29 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
    2019-01-15 14:34:30 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/top250> (referer: None)
    2019-01-15 14:34:30 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not allowed
    2019-01-15 14:34:30 [scrapy.core.engine] INFO: Closing spider (finished)
    2019-01-15 14:34:30 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {
        'downloader/request_bytes': 221,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 248,
     'downloader/response_count': 1,
     'downloader/response_status_count/403': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2019, 1, 15, 6, 34, 30, 269071),
     'httperror/response_ignored_count': 1,
     'httperror/response_ignored_status_count/403': 1,
     'log_count/DEBUG': 2,
     'log_count/INFO': 8,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2019, 1, 15, 6, 34, 29, 539961)}
    2019-01-15 14:34:30 [scrapy.core.engine] INFO: Spider closed (finished)

解决方案：（找了好久，各种瞎改`settings.py`中的配置，最终解决方案还是在慕课网找到了，慕课网棒棒哒！！！）

[《scrapy crawl splider\_name 报错 看这个视频》][scrapy crawl splider_name _]

### 二、将项目导入PyCharm IDE ###

pycharm的基本配置和使用以及将scrapy项目导入到pycharm中可以参考如下文档，我就不重复造轮子了：

*  [pycharm使用及设置方法][pycharm]
 *  [Pycharm+Scrapy安装并且初始化项目][Pycharm_Scrapy]

### 三、结语 ###

*  还好没放弃，痛苦踩坑之后随便往哪里瞅瞅，遍地都是彩虹！

[http_www.scrapyd.cn_doc_124.html]: http://www.scrapyd.cn/doc/124.html
[pip_scrapy]: http://www.scrapyd.cn/doc/123.html
[pip install twisted]: https://blog.csdn.net/a19990412/article/details/78849881
[scrapy crawl splider_name _]: https://www.imooc.com/video/17517
[pycharm]: https://blog.csdn.net/yanyangjie/article/details/78259561
[Pycharm_Scrapy]: https://blog.csdn.net/skullfang/article/details/78607942