基于selenium+scrapy爬取复仇者联盟4豆瓣影评数据

朱雀 2022-02-09 13:41 251阅读 0赞

# 基于selenium+scrapy爬取复仇者联盟4豆瓣影评数据 #

参考资料：

*  黑马程序员爬虫教程
 *  静觅爬虫教程
 *  [mac下anaconda安装selenium+PhantomJS][mac_anaconda_selenium_PhantomJS]
 *  [scrapy下载中间件结合selenium抓取全国空气质量检测数据][scrapy_selenium]
 *  [使用xpath的轴(Axis)进行元素定位][xpath_Axis]

*以下内容只用于学习使用，请勿用于商业用途.*

五一放假看了《复仇者联盟4》，对影片内容不是很懂，所以写个爬虫，抓取下影评，加深对电影的理解。

一开始打算使用简单的python工具库来爬取数据。电脑系统是mac，安装了Anaconda和python3.7。查看以前写的python爬虫笔记，都是使用urllib2来爬取数据。而Anaconda默认安装的是urllib3，所以就研究下urllib3。

## urllib3学习 ##

urllib3是一个功能强大，条理清晰，用于HTTP客户端的Python库，许多Python的原生系统已经开始使用urllib3。

urllib3提供了很多python标准库里所没有的重要特性：

1.  线程安全
2.  连接池
3.  客户端SSL / TLS验证
4.  文件分部编码上传
5.  协助处理重复请求和HTTP重定位
6.  支持压缩编码
7.  支持HTTP和SOCKS代理
8.  100 % 测试覆盖率

def askURL(url, fields):
        '''
        使用urllib3得到页面全部内容
        :param url: 请求的url
        :param fields: 请求参数
        :return: 网页内容
        '''
    
        # 创建一个PoolManager实例来生成请求,由该实例对象处理与线程池的连接以及线程安全的所有细节，不需要任何人为操作
        http = urllib3.PoolManager()
        # 通过request()方法创建一个请求，返回一个HTTPResponse对象
        response = http.request('GET', url, fields)
        # print(response.status) # 打印响应码
        html = response.data # 获取网页内容
        # print(html)
        return html

研究了urllib3后，就使用它去爬取复仇4的影评，结果发现：

1.  短评的内容到start为220，即21页的时候，就必须要登录才能查看；
2.  影评的内容默认是折叠的，需要要手动点击展开才能查看完整内容；

鉴于以上我原因，我只好使用selenium结合scrapy框架来实现影评爬虫。

## mac下anaconda安装selenium+PhantomJS ##

在Anaconda中查找selenium，点击安装即可。

在[phantomJS官网][phantomJS]下载压缩包，解压后，拷贝bin/phantomjs文件到Anaconda安装python3的目录anaconda3/bin下。

## scrapy框架学习 ##

Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架。 可以应用在包括数据挖掘，信息处理或存储历史数据等一系列的程序中。  
其最初是为了页面抓取 (更确切来说, 网络抓取 )所设计的， 也可以应用在获取API所返回的数据(例如 Amazon Associates Web Services ) 或者通用的网络爬虫。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。

Scrapy 使用了 Twisted 异步网络库来处理网络通讯。整体架构大致如下

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZpZ2h0aW5nX05vMQ_size_16_color_FFFFFF_t_70]

**Scrapy主要包括了以下组件：**

1.  引擎(Scrapy): 用来处理整个系统的数据流处理, 触发事务(框架核心)
2.  调度器(Scheduler): 用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址
3.  下载器(Downloader): 用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的)
4.  爬虫(Spiders): 爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面
5.  项目管道(Pipeline): 负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。
6.  下载器中间件(Downloader Middlewares): 位于Scrapy引擎和下载器之间的框架，主要是处理Scrapy引擎与下载器之间的请求及响应。
7.  爬虫中间件(Spider Middlewares): 介于Scrapy引擎和爬虫之间的框架，主要工作是处理蜘蛛的响应输入和请求输出。
8.  调度中间件(Scheduler Middewares): 介于Scrapy引擎和调度之间的中间件，从Scrapy引擎发送到调度的请求和响应。

**Scrapy运行流程大概如下：**

*  首先，引擎从调度器中取出一个链接(URL)用于接下来的抓取
 *  引擎把URL封装成一个请求(Request)传给下载器，下载器把资源下载下来，并封装成应答包(Response)
 *  然后，爬虫解析Response
 *  若是解析出实体（Item）,则交给实体管道进行进一步的处理。
 *  若是解析出的是链接（URL）,则把URL交给Scheduler等待抓取

**安装**

在Anaconda中搜索scrapy并安装即可。

## 代码实现 ##

新建个scrapy爬虫工程

`scrapy startproject doubanscrapy`

进入爬虫工程目录并新建爬虫类

cd doubanscrapy
    scrapy genspider doubanmovie movie.douban.com

启动爬虫使用命令

`scrapy crawl doubanmovie`

下面我们开始编写代码来对复仇4的影评数据进行爬取。

### 利用selenium模拟表单登录 ###

首先，我们需要在settings.py中开启Cookie

`COOKIES_ENABLED = True`

然后，我们要启用下载中间件

DOWNLOADER_MIDDLEWARES = {
        'doubanscrapy.middlewares.DoubanscrapyDownloaderMiddleware': 543,
    }

接着，在middlewares.py中去修改DoubanscrapyDownloaderMiddleware类的process\_request函数实现。

def process_request(self, request, spider):
            if (request.url == 'https://accounts.douban.com/passport/login?source=movie'):
                '''
                利用selenium，模拟表单登录
                '''
                self.driver = webdriver.PhantomJS()
                self.driver.get(request.url)
                # 选择账号密码登录
                self.driver.find_element_by_class_name('account-tab-account').click()
                # 输入账号密码
                self.driver.find_element_by_id('username').send_keys('username')
                self.driver.find_element_by_id('password').send_keys('password')
                # 模拟点击登录
                self.driver.find_element_by_class_name('btn-account').click()
                # 等待3秒
                time.sleep(3)
                # 获取请求后得到的源码
                html = self.driver.page_source
                # spider.logger.info(html)
                # 关闭浏览器
                self.driver.quit()
    
                # 构造一个请求的结果，将浏览器访问得到的结果构造成response，并返回给引擎
                response = http.HtmlResponse(url=request.url, body=html, request=request, encoding='utf-8')
    
                return response

注意，这里通过判断url是否是登录的url，然后登录操作，其他url是不需要去调用登录操作的。

豆瓣登录默认采用的是手机验证码的登录方式，所以我们获取到登录页面的HTML内容后，要先通过selenium选择使用账号密码方式登录，然后在表单中输入用户名和密码，接着点击登录按钮，最后等待3秒，就可以得到登录成功后的请求页面内容了。

最后，我们在DoubanmovieSpider类中设置start\_urls为登录的url，这样启动爬虫第一个爬取的url就是登录url，然后中间件就会模拟登录，然后就会返回登录成功的response，最后我们去调用Request去爬取影评url。

class DoubanmovieSpider(scrapy.Spider):
        name = 'doubanmovie'
        allowed_domains = ['movie.douban.com']
        start_urls = ['https://accounts.douban.com/passport/login?source=movie']
        base_url = "https://movie.douban.com/subject/26100958/reviews"
    
        # 处理start_urls里的登录url的响应内容
        def parse(self, response):
            # 爬取复仇者联盟4的影评信息数据
            yield scrapy.Request(self.base_url, callback=self.parse_movie)

### 爬取影评数据 ###

首先，由于豆瓣影评的内容可能是被折叠，或者是未完整展开的，我们需要在DoubanscrapyDownloaderMiddleware类的process\_request函数中利用selenium，模拟用户点击查看被折叠的影评，并模拟用户点击展开按钮，获取完整的影评内容HTML。

def process_request(self, request, spider):
        if (request.url == 'https://accounts.douban.com/passport/login?source=movie'):
            ...
        elif (request.url.startswith('https://movie.douban.com/subject/26100958/reviews')):
            '''
            利用selenium，模拟用户查看影评
            '''
            self.driver = webdriver.PhantomJS()
            self.driver.get(request.url)
    
            # 是否存在被隐藏的影评
            fold_hd = self.driver.find_elements_by_class_name('fold-hd')
    
            if len(fold_hd) != 0:
                # 展开影评列表
                self.driver.find_element_by_class_name('btn-unfold').click()
                time.sleep(1) 
    
            # 展开所有影评内容
            reviews = self.driver.find_elements_by_class_name('unfold')
            for review in reviews:
                review.click()
            # 等待5秒
            time.sleep(5)
            # 获取请求后得到的源码
            html = self.driver.page_source
            # spider.logger.info(html)
            # 关闭浏览器
            self.driver.quit()
    
            # 构造一个请求的结果，将浏览器访问得到的结果构造成response，并返回给引擎
            response = http.HtmlResponse(url=request.url, body=html, request=request, encoding='utf-8')
    
            return response
        else:
            return None

注意，有折叠的话，需要sleep1秒等待浏览器反应后再去展开影评内容。

然后我们需要编写个item类去保存我们需要的影评信息。

class DoubanMovieReviewItem(scrapy.Item):
        # define the fields for your item here like:
        title = scrapy.Field()
        # 作者
        author = scrapy.Field()
        # 推荐度
        rating = scrapy.Field()
        # 点赞数
        vote = scrapy.Field()
        # 回应数
        reply = scrapy.Field()
        # 影评内容
        content = scrapy.Field()
        # 影评日期
        reviewTime = scrapy.Field()

接着，修改pipeline，将获取到的影评信息保存到json文件中。

class DoubanscrapyPipeline(object):
        def __init__(self):
            self.file = open('movie.json', 'w', encoding='utf-8')
            self.file.write("[\n")
    
        def process_item(self, item, spider):
            content = json.dumps(dict(item), ensure_ascii=False) + ",\n"
            self.file.write(content)
            return item
    
        def close_spider(self, spider):
            self.file.write("]\n")
            self.file.close()

最后，我们编写DoubanmovieSpider中的parse\_movie函数来解析影评内容，抽取出我们想要的信息。

# 处理响应内容
        def parse_movie(self, response):
            # self.log(response)
            contents = response.xpath("//div[@class='main review-item']")
            # 遍历所有的影评
            for content in contents:
                item = DoubanMovieReviewItem()
                main_hd = content.xpath("./header")
                # 影评作者
                item['author'] = main_hd.xpath("./a[@class='name']/text()").extract()[0]
                # 影评推荐力度
                rating = main_hd.xpath("./span[@title]/@title").extract()
                if len(rating) != 0:
                    item['rating'] = rating[0]
                # 影评时间
                item['reviewTime'] = main_hd.xpath("./span[@class='main-meta']/text()").extract()[0]
                main_bd = content.xpath("./div[@class='main-bd']")
                # 影评标题
                item['title'] = main_bd.xpath("./h2/a/text()").extract()[0]
                # 影评内容
                item['content'] = main_bd.xpath("./descendant::div[@id='review-content']/*").extract()
                action = main_bd.xpath("./div[3]")
                # 点赞数
                item['vote'] = action.xpath("./a[1]/span/text()").extract()[0].strip()
                # 回应数
                reply = action.xpath("./a[3]/text()").extract()[0]
                if reply:
                    item['reply'] = reply.replace("回应", "")
                # self.log(item)
                yield item
    
            # 爬取剩余页面
            if self.offset == 0:
                # 获取总页数
                total_page = response.xpath("//div[@class='paginator']/span[@class='thispage']/@data-total-page").extract()[
                    0]
                # self.log(total_page)
                total_page = int(total_page)
    
                # 设置总页数
                self.page = total_page
    
                # 设置只爬取前60条影评
                if (self.page > 3):
                    self.page = 3
    
                if total_page > 1:
                    self.offset = self.offset + 20
                    url = self.base_url + '?start=' + str(self.offset)
                    # self.log(url)
                    yield scrapy.Request(url, callback=self.parse_movie)
            else:
                if self.offset / 20 != self.page - 1:
                    self.offset = self.offset + 20
                    url = self.base_url + '?start=' + str(self.offset)
                    # self.log(url)
                    yield scrapy.Request(url, callback=self.parse_movie)

注意：我这边可能是因为网络原因，所以爬取的过程老是中途就没有反应了，所以上述代码中设置page为3，只爬取前60条数据。

爬取结果如下图所示：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZpZ2h0aW5nX05vMQ_size_16_color_FFFFFF_t_70 1]

### 完整代码 ###

[settings.py][]

# -*- coding: utf-8 -*-
    
    # Scrapy settings for doubanscrapy project
    #
    # For simplicity, this file contains only settings considered important or
    # commonly used. You can find more settings consulting the documentation:
    #
    #     https://doc.scrapy.org/en/latest/topics/settings.html
    #     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
    #     https://doc.scrapy.org/en/latest/topics/spider-middleware.html
    
    BOT_NAME = 'doubanscrapy'
    
    SPIDER_MODULES = ['doubanscrapy.spiders']
    NEWSPIDER_MODULE = 'doubanscrapy.spiders'
    
    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    # USER_AGENT = 'doubanscrapy (+http://www.yourdomain.com)'
    
    # Obey robots.txt rules
    ROBOTSTXT_OBEY = True
    
    # Configure maximum concurrent requests performed by Scrapy (default: 16)
    # CONCURRENT_REQUESTS = 32
    
    # Configure a delay for requests for the same website (default: 0)
    # See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
    # See also autothrottle settings and docs
    # DOWNLOAD_DELAY = 3
    # The download delay setting will honor only one of:
    # CONCURRENT_REQUESTS_PER_DOMAIN = 16
    # CONCURRENT_REQUESTS_PER_IP = 16
    
    # Disable cookies (enabled by default)
    COOKIES_ENABLED = True
    
    # Disable Telnet Console (enabled by default)
    # TELNETCONSOLE_ENABLED = False
    
    # Override the default request headers:
    # DEFAULT_REQUEST_HEADERS = {
    #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    #   'Accept-Language': 'en',
    # }
    
    # Enable or disable spider middlewares
    # See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
    # SPIDER_MIDDLEWARES = {
    #    'doubanscrapy.middlewares.DoubanscrapySpiderMiddleware': 543,
    # }
    
    # Enable or disable downloader middlewares
    # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
    DOWNLOADER_MIDDLEWARES = {
        'doubanscrapy.middlewares.RandomUserAgent': 100,
        # 'doubanscrapy.middlewares.RandomProxy': 200,
        'doubanscrapy.middlewares.DoubanscrapyDownloaderMiddleware': 543,
    }
    
    USER_AGENTS = [
        'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)',
        'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)',
        'Opera/9.27 (Windows NT 5.2; U; zh-cn)',
        'Opera/8.0 (Macintosh; PPC Mac OS X; U; en)',
        'Mozilla/5.0 (Macintosh; PPC Mac OS X; U; en) Opera 8.0',
        'Mozilla/5.0 (Linux; U; Android 4.0.3; zh-cn; M032 Build/IML74K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30',
        'Mozilla/5.0 (Windows; U; Windows NT 5.2) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13'
    ]
    
    PROXIES = [
        {"ip_port": "121.42.140.113:16816", "user_passwd": ""},
        {"ip_prot": "58.246.96.211:8080", "user_passwd": ""},
        {"ip_prot": "58.253.238.242:80", "user_passwd": ""}
    ]
    
    # Enable or disable extensions
    # See https://doc.scrapy.org/en/latest/topics/extensions.html
    # EXTENSIONS = {
    #    'scrapy.extensions.telnet.TelnetConsole': None,
    # }
    
    # Configure item pipelines
    # See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
    ITEM_PIPELINES = {
        'doubanscrapy.pipelines.DoubanscrapyPipeline': 300,
    }
    
    # Enable and configure the AutoThrottle extension (disabled by default)
    # See https://doc.scrapy.org/en/latest/topics/autothrottle.html
    # AUTOTHROTTLE_ENABLED = True
    # The initial download delay
    # AUTOTHROTTLE_START_DELAY = 5
    # The maximum download delay to be set in case of high latencies
    # AUTOTHROTTLE_MAX_DELAY = 60
    # The average number of requests Scrapy should be sending in parallel to
    # each remote server
    # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
    # Enable showing throttling stats for every response received:
    # AUTOTHROTTLE_DEBUG = False
    
    # Enable and configure HTTP caching (disabled by default)
    # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
    # HTTPCACHE_ENABLED = True
    # HTTPCACHE_EXPIRATION_SECS = 0
    # HTTPCACHE_DIR = 'httpcache'
    # HTTPCACHE_IGNORE_HTTP_CODES = []
    # HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

[middlewares.py][]

#!/usr/bin/env python
    # -*- coding: utf-8 -*-
    
    # Define here the models for your spider middleware
    #
    # See documentation in:
    # https://doc.scrapy.org/en/latest/topics/spider-middleware.html
    
    from scrapy import signals, http
    from selenium import webdriver
    import time
    
    import random
    import base64
    
    from doubanscrapy.settings import USER_AGENTS
    from doubanscrapy.settings import PROXIES
    
    
    class DoubanscrapySpiderMiddleware(object):
        # Not all methods need to be defined. If a method is not defined,
        # scrapy acts as if the spider middleware does not modify the
        # passed objects.
    
        @classmethod
        def from_crawler(cls, crawler):
            # This method is used by Scrapy to create your spiders.
            s = cls()
            crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
            return s
    
        def process_spider_input(self, response, spider):
            # Called for each response that goes through the spider
            # middleware and into the spider.
    
            # Should return None or raise an exception.
            return None
    
        def process_spider_output(self, response, result, spider):
            # Called with the results returned from the Spider, after
            # it has processed the response.
    
            # Must return an iterable of Request, dict or Item objects.
            for i in result:
                yield i
    
        def process_spider_exception(self, response, exception, spider):
            # Called when a spider or process_spider_input() method
            # (from other spider middleware) raises an exception.
    
            # Should return either None or an iterable of Response, dict
            # or Item objects.
            pass
    
        def process_start_requests(self, start_requests, spider):
            # Called with the start requests of the spider, and works
            # similarly to the process_spider_output() method, except
            # that it doesn’t have a response associated.
    
            # Must return only requests (not items).
            for r in start_requests:
                yield r
    
        def spider_opened(self, spider):
            spider.logger.info('Spider opened: %s' % spider.name)
    
    
    class DoubanscrapyDownloaderMiddleware(object):
        # Not all methods need to be defined. If a method is not defined,
        # scrapy acts as if the downloader middleware does not modify the
        # passed objects.
    
        @classmethod
        def from_crawler(cls, crawler):
            # This method is used by Scrapy to create your spiders.
            s = cls()
            crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
            return s
    
        def process_request(self, request, spider):
            # Called for each request that goes through the downloader
            # middleware.
    
            # Must either:
            # - return None: continue processing this request
            # - or return a Response object
            # - or return a Request object
            # - or raise IgnoreRequest: process_exception() methods of
            #   installed downloader middleware will be called
            if (request.url == 'https://accounts.douban.com/passport/login?source=movie'):
                '''
                利用selenium，模拟表单登录
                '''
                self.driver = webdriver.PhantomJS()
                self.driver.get(request.url)
                # 选择账号密码登录
                self.driver.find_element_by_class_name('account-tab-account').click()
                # 输入账号密码
                self.driver.find_element_by_id('username').send_keys('18819490304')
                self.driver.find_element_by_id('password').send_keys('5354PPmm')
                # 模拟点击登录
                self.driver.find_element_by_class_name('btn-account').click()
                # 等待3秒
                time.sleep(3)
                # 获取请求后得到的源码
                html = self.driver.page_source
                # spider.logger.info(html)
                # 关闭浏览器
                self.driver.quit()
    
                # 构造一个请求的结果，将浏览器访问得到的结果构造成response，并返回给引擎
                response = http.HtmlResponse(url=request.url, body=html, request=request, encoding='utf-8')
    
                return response
            elif (request.url.startswith('https://movie.douban.com/subject/26100958/reviews')):
                '''
                利用selenium，模拟用户查看影评
                '''
                self.driver = webdriver.PhantomJS()
                self.driver.get(request.url)
    
                # 是否存在被隐藏的影评
                fold_hd = self.driver.find_elements_by_class_name('fold-hd')
    
                if len(fold_hd) != 0:
                    # 展开影评列表
                    self.driver.find_element_by_class_name('btn-unfold').click()
                    time.sleep(1)
    
                # 展开所有影评内容
                reviews = self.driver.find_elements_by_class_name('unfold')
                for review in reviews:
                    review.click()
                # 等待5秒
                time.sleep(5)
                # 获取请求后得到的源码
                html = self.driver.page_source
                # spider.logger.info(html)
                # 关闭浏览器
                self.driver.quit()
    
                # 构造一个请求的结果，将浏览器访问得到的结果构造成response，并返回给引擎
                response = http.HtmlResponse(url=request.url, body=html, request=request, encoding='utf-8')
    
                return response
            else:
                return None
    
        def process_response(self, request, response, spider):
            # Called with the response returned from the downloader.
    
            # Must either;
            # - return a Response object
            # - return a Request object
            # - or raise IgnoreRequest
            return response
    
        def process_exception(self, request, exception, spider):
            # Called when a download handler or a process_request()
            # (from other downloader middleware) raises an exception.
    
            # Must either:
            # - return None: continue processing this exception
            # - return a Response object: stops process_exception() chain
            # - return a Request object: stops process_exception() chain
            pass
    
        def spider_opened(self, spider):
            spider.logger.info('Spider opened: %s' % spider.name)
    
    
    # 随机的User-Agent
    class RandomUserAgent(object):
        def process_request(self, request, spider):
            useragent = random.choice(USER_AGENTS)
            # print useragent
            request.headers.setdefault("User-Agent", useragent)
    
    
    # class RandomProxy(object):
    #     def process_request(self, request, spider):
    #         proxy = random.choice(PROXIES)
    #         # 没有代理账户验证的代理使用方式
    #         request.meta['proxy'] = "http://" + proxy['ip_port']

[items.py][]

# -*- coding: utf-8 -*-
    
    # Define here the models for your scraped items
    #
    # See documentation in:
    # https://doc.scrapy.org/en/latest/topics/items.html
    
    import scrapy
    
    
    class DoubanMovieReviewItem(scrapy.Item):
        # define the fields for your item here like:
        title = scrapy.Field()
        # 作者
        author = scrapy.Field()
        # 推荐度
        rating = scrapy.Field()
        # 点赞数
        vote = scrapy.Field()
        # 回应数
        reply = scrapy.Field()
        # 影评内容
        content = scrapy.Field()
        # 影评日期
        reviewTime = scrapy.Field()

[pipelines.py][]

# -*- coding: utf-8 -*-
    
    import json
    
    # Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
    
    
    class DoubanscrapyPipeline(object):
        def __init__(self):
            self.file = open('movie.json', 'w', encoding='utf-8')
            self.file.write("[\n")
    
        def process_item(self, item, spider):
            content = json.dumps(dict(item), ensure_ascii=False) + ",\n"
            self.file.write(content)
            return item
    
        def close_spider(self, spider):
            self.file.write("]\n")
            self.file.close()

[doubanmovie.py][]

# -*- coding: utf-8 -*-
    import scrapy
    from doubanscrapy.items import DoubanMovieReviewItem
    
    
    class DoubanmovieSpider(scrapy.Spider):
        name = 'doubanmovie'
        allowed_domains = ['movie.douban.com']
        start_urls = ['https://accounts.douban.com/passport/login?source=movie']
        base_url = "https://movie.douban.com/subject/26100958/reviews"
        offset = 0
        page = 1
    
        # 处理start_urls里的登录url的响应内容
        def parse(self, response):
            # 爬取复仇者联盟4的影评信息数据
            yield scrapy.Request(self.base_url, callback=self.parse_movie)
    
        # 处理响应内容
        def parse_movie(self, response):
            # self.log(response)
            contents = response.xpath("//div[@class='main review-item']")
            # 遍历所有的影评
            for content in contents:
                item = DoubanMovieReviewItem()
                main_hd = content.xpath("./header")
                # 影评作者
                item['author'] = main_hd.xpath("./a[@class='name']/text()").extract()[0]
                # 影评推荐力度
                rating = main_hd.xpath("./span[@title]/@title").extract()
                if len(rating) != 0:
                    item['rating'] = rating[0]
                # 影评时间
                item['reviewTime'] = main_hd.xpath("./span[@class='main-meta']/text()").extract()[0]
                main_bd = content.xpath("./div[@class='main-bd']")
                # 影评标题
                item['title'] = main_bd.xpath("./h2/a/text()").extract()[0]
                # 影评内容
                item['content'] = main_bd.xpath("./descendant::div[@id='review-content']/*").extract()
                action = main_bd.xpath("./div[3]")
                # 点赞数
                item['vote'] = action.xpath("./a[1]/span/text()").extract()[0].strip()
                # 回应数
                reply = action.xpath("./a[3]/text()").extract()[0]
                if reply:
                    item['reply'] = reply.replace("回应", "")
                # self.log(item)
                yield item
    
            # 爬取剩余页面
            if self.offset == 0:
                # 获取总页数
                total_page = response.xpath("//div[@class='paginator']/span[@class='thispage']/@data-total-page").extract()[
                    0]
                # self.log(total_page)
                total_page = int(total_page)
    
                # 设置总页数
                self.page = total_page
    
                # 设置只爬取前60条影评
                if (self.page > 3):
                    self.page = 3
    
                if total_page > 1:
                    self.offset = self.offset + 20
                    url = self.base_url + '?start=' + str(self.offset)
                    # self.log(url)
                    yield scrapy.Request(url, callback=self.parse_movie)
            else:
                if self.offset / 20 != self.page - 1:
                    self.offset = self.offset + 20
                    url = self.base_url + '?start=' + str(self.offset)
                    # self.log(url)
                    yield scrapy.Request(url, callback=self.parse_movie)

[mac_anaconda_selenium_PhantomJS]: https://blog.csdn.net/lilong117194/article/details/83277075
[scrapy_selenium]: https://www.cnblogs.com/pythoner6833/p/9049498.html
[xpath_Axis]: https://www.jianshu.com/p/fcfc889aba43
[phantomJS]: http://phantomjs.org/download.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZpZ2h0aW5nX05vMQ_size_16_color_FFFFFF_t_70]: /images/20220206/1ae2232384344e43bbb3b68e238d9382.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZpZ2h0aW5nX05vMQ_size_16_color_FFFFFF_t_70 1]: /images/20220206/0658b29ddfdc4aed96166099b176001c.png
[settings.py]: http://settings.py
[middlewares.py]: http://middlewares.py
[items.py]: http://items.py
[pipelines.py]: http://pipelines.py
[doubanmovie.py]: http://doubanmovie.py