CentOS使用scrapy-splash

男娘i 2022-05-04 16:56 161阅读 0赞

准备工作

*  先完成简单scrapy项目
 *  安装docker
    
     *  win下下载安装包安装
     *  mac下下载安装包安装(尝试使用brew安装，安装启动过程非常复杂，最后选择使用安装包直接安装)
     *  centos7下运行：
        
        yum install docker
 *  redhat运行：
    
        yum install --setopt=obsoletes=0 docker-ce-17.03.2.ce-1.el7.centos.x86_64 docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch
 *  安装 scrapy-splash
    
        pip install scrapy-splash
 *  启动docker服务
    
     *  centos7
        
        service docker start
     *  win下直接打开应用
     *  mac下直接打开应用
 *  拉取镜像
    
        docker pull scrapinghub/splash
 *  运行镜像
    
        docker run -p 8050:8050 scrapinghub/splash
 *  配置splash服务（以下操作全部在settings.py）：
    
     *  添加splash服务器地址：
        
        SPLASH\_URL = ‘http://localhost:8050’
     *  将splash middleware添加到DOWNLOADER\_MIDDLEWARE中：
        
            DOWNLOADER_MIDDLEWARES = {
                'scrapy_splash.SplashCookiesMiddleware': 723,
                'scrapy_splash.SplashMiddleware': 725,
                'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
            }
     *  Enable SplashDeduplicateArgsMiddleware:
        
            SPIDER_MIDDLEWARES = {
                'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
            }
     *  Set a custom DUPEFILTER\_CLASS:
        
            DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
     *  a custom cache storage backend:
        
            HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
 *  例子
    
        import json, scrapy
        
        lass MySpider(scrapy.Spider):
           name = 'example'
           allowed_domains = ['example.com']
           start_urls = ["http://example.com", "http://example.com/foo"]
        
           def start_requests(self):
             for url in self.start_urls:
               yield SplashRequest(url, self.parse, args={'wait': 0.5})
        
           def parse(self, response):
               # ...

欢迎关注我的公众号，最近在更新知乎爬虫系列

![我的公众号][20190127004149980.jpg]

[20190127004149980.jpg]: /images/20220504/6d7eb7a765964822a1d57ab06b157d9f.png