scrapy爬虫框架你还不会吗？简单使用爬虫框架采集网站数据

淩亂°似流年 2023-01-03 04:50 124阅读 0赞

# 前言 #

本文的文字及图片过滤网络，可以学习，交流使用，不具有任何商业用途，如有问题请及时联系我们以作处理。

本篇文章就使用python爬虫框架scrapy采集网站的一些数据。

# 基本开发环境 #

*  Python 3.6
 *  pycharm

**如何安装scrapy**

在cmd命令行当中 **pip install scrapy**就可以安装了。但是一般情况都会出现网络超时的情况。

建议切换国内常规源安装 **pip install -i国内常规地址****包名**

例如：

pip install -i  https://mirrors.aliyun.com/pypi/simple/ scrapy

**国内常用源别名地址：**

清华：https://pypi.tuna.tsinghua.edu.cn/simple
    阿里云：http://mirrors.aliyun.com/pypi/simple/
    中国科技大学 https://pypi.mirrors.ustc.edu.cn/simple/
    华中理工大学：http://pypi.hustunique.com/
    山东理工大学：http://pypi.sdutlinux.org/ 
    豆瓣：http://pypi.douban.com/simple/

**你可能会出现的报错：**

在安装Scrapy的过程中可能会遇到VC ++等错误，可以安装删除模块的离线包

![aa2d76ac20821010810311ef7132dd0e.png][]

**Scrapy如何爬取网站数据**

本篇文章以豆瓣电影Top250的数据为例，讲解一下scrapy框架爬取数据的基本流程。

![8f4a5137c84df1d81584faffcd16cfdc.png][]

豆瓣Top250这个数据就不过多分析，静态网站，网页结构十分适合写爬取，所以很多基础入门的爬虫案例都是以豆瓣电影数据以及猫眼电影数据为例的。

# Scrapy的爬虫项目的创建流程 #

**1.创建一个爬虫项目**

在Pycharm中选择Terminal在Local里面输入

**scrapy startproject +（项目名字<独一无二>）**

![5eb84db9d1e03d174b30a200b2b67b89.png][]

**2.cd切换到爬虫项目目录**

![42d97071ad243f01b59f332b20dd4b94.png][]

**3.创建爬虫文件**

**scrapy genspider（+爬虫文件的名字<独一无二的>）（+域名限制）**

![ffd8bf1c9aa16950a1a07ce5cbde0393.png][]

![afa9a296dfe116339bb47d9a8f740e00.png][]

这就对于scrapy的项目创建以及爬虫文件创建完成了。

# Scrapy的爬虫代码编写 #

**1，在settings.py文件中关闭robots协议默认是True**

![f7746af4eee4c0d6a1098cb4dad9d4d4.png][]

**2，在爬虫文件下修改起始网址**

start_urls = ['https://movie.douban.com/top250?filter=']

把start\_urls改成豆瓣导航网址的链接，也就是你爬取数据的第一页的url地址

**3，写解析数据的业务逻辑**

爬取内容如下：

![96901d9e717d3cc6577fe11920da24cf.png][]

**douban\_info.py**

import scrapy
    
    
    from ..items import DoubanItem
    
    
    
    
    class DoubanInfoSpider(scrapy.Spider):
        name = 'douban_info'
        allowed_domains = ['douban.com']
        start_urls = ['https://movie.douban.com/top250?start=0&filter=']
    
    
        def parse(self, response):
            lis = response.css('.grid_view li')
            print(lis)
            for li in lis:
                title = li.css('.hd span:nth-child(1)::text').get()
                movie_info = li.css('.bd p::text').getall()
                info = ''.join(movie_info).strip()
                score = li.css('.rating_num::text').get()
                number = li.css('.star span:nth-child(4)::text').get()
                summary = li.css('.inq::text').get()
                print(title)
                yield DoubanItem(title=title, info=info, score=score, number=number, summary=summary)
    
    
            href = response.css('#content .next a::attr(href)').get()
            if href:
                next_url = 'https://movie.douban.com/top250' + href
                yield scrapy.Request(url=next_url, callback=self.parse)

**itmes.py**

import scrapy
    
    
    
    
    class DoubanItem(scrapy.Item):
        # define the fields for your item here like:
        title = scrapy.Field()
        info = scrapy.Field()
        score = scrapy.Field()
        number = scrapy.Field()
        summary = scrapy.Field()

**middlewares.py**

import faker
    
    
    
    
    def get_cookies():
        """获取cookies的函数"""
        headers = {
            'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'}
    
    
        response = requests.get(url='https://movie.douban.com/top250?start=0&filter=',
                                headers=headers)
        return response.cookies.get_dict()
    
    
    
    
    def get_proxies():
        """代理请求的函数"""
        proxy_data = requests.get(url='http://127.0.0.1:5000/get/').json()
        return proxy_data['proxy']
    
    
    
    
    class HeadersDownloaderMiddleware:
        """headers中间件"""
    
    
        def process_request(self, request, spider):
            # 可以拿到请求体
            fake = faker.Faker()
            # request.headers  拿到请求头, 请求头是一个字典
            request.headers.update(
                {
                    'user-agent': fake.user_agent(),
                }
            )
            return None
    
    
    class CookieDownloaderMiddleware:
        """cookie中间件"""
    
    
        def process_request(self, request, spider):
            # request.cookies 设置请求的cookies, 是字典
            # get_cookies()  调用获取cookies的方法
            request.cookies.update(get_cookies())
            return None
    
    
    class ProxyDownloaderMiddleware:
        """代理中间件"""
    
    
        def process_request(self, request, spider):
            # 获取请求的 meta , 字典
            request.meta['proxy'] = get_proxies()
            return None

**pipelines.py**

import csv
    
    
    
    
    class DoubanPipeline:
        def __init__(self):
            self.file = open('douban.csv', mode='a', encoding='utf-8', newline='')
            self.csv_file = csv.DictWriter(self.file, fieldnames=['title', 'info', 'score', 'number', 'summary'])
            self.csv_file.writeheader()
    
    
        def process_item(self, item, spider):
            dit = dict(item)
            dit['info'] = dit['info'].replace('\n', "").strip()
            self.csv_file.writerow(dit)
            return item
    
    
    
    
        def spider_closed(self, spider) -> None:
            self.file.close()

**setting.py**

# Scrapy settings for douban project
    #
    # For simplicity, this file contains only settings considered important or
    # commonly used. You can find more settings consulting the documentation:
    #
    #     https://docs.scrapy.org/en/latest/topics/settings.html
    #     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
    #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
    
    
    BOT_NAME = 'douban'
    
    
    SPIDER_MODULES = ['douban.spiders']
    NEWSPIDER_MODULE = 'douban.spiders'
    
    
    
    
    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    #USER_AGENT = 'douban (+http://www.yourdomain.com)'
    
    
    # Obey robots.txt rules
    ROBOTSTXT_OBEY = False
    
    
    # Configure maximum concurrent requests performed by Scrapy (default: 16)
    #CONCURRENT_REQUESTS = 32
    
    
    # Configure a delay for requests for the same website (default: 0)
    # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
    # See also autothrottle settings and docs
    #DOWNLOAD_DELAY = 3
    # The download delay setting will honor only one of:
    #CONCURRENT_REQUESTS_PER_DOMAIN = 16
    #CONCURRENT_REQUESTS_PER_IP = 16
    
    
    # Disable cookies (enabled by default)
    #COOKIES_ENABLED = False
    
    
    # Disable Telnet Console (enabled by default)
    #TELNETCONSOLE_ENABLED = False
    
    
    # Override the default request headers:
    #DEFAULT_REQUEST_HEADERS = {
    #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    #   'Accept-Language': 'en',
    #}
    
    
    # Enable or disable spider middlewares
    # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
    # SPIDER_MIDDLEWARES = {
    #    'douban.middlewares.DoubanSpiderMiddleware': 543,
    # }
    
    
    # Enable or disable downloader middlewares
    # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
    DOWNLOADER_MIDDLEWARES = {
       'douban.middlewares.HeadersDownloaderMiddleware': 543,
    }
    
    
    # Enable or disable extensions
    # See https://docs.scrapy.org/en/latest/topics/extensions.html
    #EXTENSIONS = {
    #    'scrapy.extensions.telnet.TelnetConsole': None,
    #}
    
    
    # Configure item pipelines
    # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
    ITEM_PIPELINES = {
       'douban.pipelines.DoubanPipeline': 300,
    }
    
    
    # Enable and configure the AutoThrottle extension (disabled by default)
    # See https://docs.scrapy.org/en/latest/topics/autothrottle.html
    #AUTOTHROTTLE_ENABLED = True
    # The initial download delay
    #AUTOTHROTTLE_START_DELAY = 5
    # The maximum download delay to be set in case of high latencies
    #AUTOTHROTTLE_MAX_DELAY = 60
    # The average number of requests Scrapy should be sending in parallel to
    # each remote server
    #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
    # Enable showing throttling stats for every response received:
    #AUTOTHROTTLE_DEBUG = False
    
    
    # Enable and configure HTTP caching (disabled by default)
    # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
    #HTTPCACHE_ENABLED = True
    #HTTPCACHE_EXPIRATION_SECS = 0
    #HTTPCACHE_DIR = 'httpcache'
    #HTTPCACHE_IGNORE_HTTP_CODES = []
    #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

**4，运行爬虫程序**

![ef0ab2824e687b5ab1c9061daa91ace1.png][]

输入命令 **scrapy crawl +爬虫文件名**

![df5d805770e6bbe37064e76b73bc0a5c.png][]

![4ffdccd35189005988d655b6c89d200c.png][]

[aa2d76ac20821010810311ef7132dd0e.png]: /images/20221119/926ec5de51c544c2a90f3dd1ed4edd4a.png
[8f4a5137c84df1d81584faffcd16cfdc.png]: /images/20221119/eff9dbc316e4471791fc2a5b415c8b19.png
[5eb84db9d1e03d174b30a200b2b67b89.png]: /images/20221119/a61c4e2630c540fe8df436be1219d58a.png
[42d97071ad243f01b59f332b20dd4b94.png]: /images/20221119/ecabf95f230f4aea84c22630a288a006.png
[ffd8bf1c9aa16950a1a07ce5cbde0393.png]: /images/20221119/45bb81d27c4a442d98e0ef125f07593c.png
[afa9a296dfe116339bb47d9a8f740e00.png]: /images/20221119/60bef76b4f3848729f5f0f4f90f1a481.png
[f7746af4eee4c0d6a1098cb4dad9d4d4.png]: /images/20221119/93947c2ac17442fb802c97b1e0f0b619.png
[96901d9e717d3cc6577fe11920da24cf.png]: /images/20221119/45beda133d7045b4b1f55b2be09ef750.png
[ef0ab2824e687b5ab1c9061daa91ace1.png]: /images/20221119/323e931a00714982832df94bb449a67d.png
[df5d805770e6bbe37064e76b73bc0a5c.png]: /images/20221119/bcc272faadd8445f92fdbc603f5ebfe7.png
[4ffdccd35189005988d655b6c89d200c.png]: /images/20221119/09baab1c98c949bda8d82de817e1d6c1.png