Python3 Scrapy框架学习一：爬取猫眼Top100榜

逃离我推掉我的手 2022-05-09 06:14 237阅读 0赞

以下操作基于Windows平台。

打开CMD命令提示框：

输入 如下命令：

![70][]

打开项目里的items.py文件，定义如下变量，用于存储。

class MaoyanItem(scrapy.Item):
        # define the fields for your item here like:
        # name = scrapy.Field()
        movie = scrapy.Field()
        actor = scrapy.Field()
        release = scrapy.Field()
        score = scrapy.Field()

然后打开项目里的spiders文件夹内的maoyanTop100.py文件。

# -*- coding: utf-8 -*-
    import time
    import scrapy
    from scrapy.http import Request
    from maoyan.items import MaoyanItem
    
    
    class Maoyantop100Spider(scrapy.Spider):
        name = 'maoyanTop100'
        #allowed_domains = ['maoyan.com/board/4']
        #start_urls = ['http://maoyan.com/board/4']
        
        def start_requests(self):
             for offset in range(0,100,10):
                self.url = 'http://maoyan.com/board/4?offset={}'.format(offset)
                yield Request(self.url,callback = self.parse)
                time.sleep(1)
    
        def parse(self, response):
            context = response.css('dd')   #分析得知所有的电影item均在该标签内
            for info in context:
                item = MaoyanItem()
                item['movie'] = info.css('p.name a::text').extract_first().strip()
                item['actor'] = info.css('.star::text').extract_first().strip()
                item['release'] = info.css('.releasetime::text').extract_first().strip()
                score = info.css('i.integer::text').extract_first().strip()
                score += info.css('i.fraction::text').extract_first().strip()
                item['score'] = score
                yield item

打开settings.py文件，加入如下两行代码：

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'
    FEED_EXPORT_ENCODING = 'gbk'

然后在命令提示符输入:

scrapy crawl maoyanTop100 -o maoyan.csv

爬取结束后会生成一个csv文件，效果图如下：

![70 1][]

欢迎关注本人微信公众号,会分享更多的干货:

![20200607142700196.png][]

[70]: /images/20220509/362f811bb186466bb955b94eb84340b4.png
[70 1]: /images/20220509/ab92187b4d394eb987e3538f8401de1d.png
[20200607142700196.png]: /images/20220509/8b3ca21e68344a3589ffe39e1fc8f19e.png