使用scrapy框架爬取数据并存到mongo数据库

青旅半醒 2022-01-30 05:19 402阅读 0赞

以爬取淘车网的二手车信息为例，将车的信息爬取出来并存到MongoDB数据库中

首先创建如图所示的目录：

进入当前目录下命令行：

创建项目：scrapy startproject day0514

然后cd day0514  进入当前项目

创建爬虫程序：scrapy genspider 程序名 域名

scrapy genspider TaoChe taoche.com

启动项目：scrapy crawl 项目名称

scrapy crawl day0514

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70][]

首先在items.py文件中定义好要存储的字段：

import scrapy
    
    
    class Day0514Item(scrapy.Item):
        # define the fields for your item here like:
        # name = scrapy.Field()
        # 列表页
        name = scrapy.Field()  # 标题
        registered_date = scrapy.Field()  # 上牌时间
        mileage = scrapy.Field()  # 车程
        city = scrapy.Field()  # 车程
        price = scrapy.Field()  # 原价
        new_price = scrapy.Field()  # 售价
        detail_url = scrapy.Field()  # 详情页链接
    
        # 详情页内容
        displacement = scrapy.Field()  # 排量
        transmission = scrapy.Field()  # 变速箱
        brand_type = scrapy.Field()  # 品牌型号
        loc_of_lic = scrapy.Field()  # 牌照所在地
        oil_wear = scrapy.Field()  # 油耗
        engine = scrapy.Field()  # 发动机
        three_high = scrapy.Field()  # 长宽高
        drive_type = scrapy.Field()  # 驱动方式
        body_way = scrapy.Field()  # 车身类型
        che_level = scrapy.Field()  # 车辆级别
        trunk_cap = scrapy.Field()  # 后备箱容量
        p_time = scrapy.Field()  # 爬取时间
        ip = scrapy.Field()

爬虫程序TaoChe.py程序如图所示：

# -*- coding: utf-8 -*-
    import scrapy
    from scrapy_redis.spiders import RedisSpider
    import time
    from day0514.items import Day0514Item
    class TaocheSpider(RedisSpider):
        name = 'TaoChe'
        # allowed_domains = ['taoche.com']
        # start_urls = ['http://taoche.com/']
    
        redis_key = "taoche:start_urls"
    
        # def parse(self, response):
        #     with open('taoCheUrl.txt','a',encoding='utf-8') as fp:
        #         fp.write(response.url+'\n')
    
        def parse(self, response):
            # 从首页获取最大页面page
            max_page = response.xpath('//div[@class="paging-box the-pages"]/div/a[last()-1]/text()').extract()
            max_page = self.get_value(max_page)
    
            # 列表页单页
            for i in range(1,int(max_page)+1):
                url = response.url + '?page=%d#pagetag'%(i)
                yield scrapy.Request(url=url,callback=self.parse_1)
    
        # 列表页解析函数
        def parse_1(self,response):
    
            # 获取整个车辆信息
            car_info_list = response.xpath('//ul[@class="gongge_ul"]/li')
            for car in car_info_list:
    
                # 标题
                name =  car.xpath('./div[@class="gongge_main"]/a/span/text()').extract()
                name = self.get_value(name)
                # 车程
                mileage =  car.xpath('./div[2]/p/i[2]/text()').extract() # 标题
                mileage = self.get_value(mileage)
                # 城市
                city = car.xpath('./div[2]/p/i[3]/span/text()').extract()
                city = self.get_value([i.strip() for i in city])
    
                # 原价
                price = car.xpath('./div[2]/div[1]/i[3]/text()').extract()
                price = self.get_value(price)
                # 现价
                new_price = car.xpath('./div[2]/div[1]/i[2]//text()').extract()
                new_price = ''.join(new_price)
                # 上牌时间
                registered_date = car.xpath('./div[2]/p/i[1]/text()').extract()
                registered_date = self.get_value(registered_date)
                # 详情页链接
                detail_url = car.xpath('./div[2]/a/@href').extract()
                detail_url = 'https:'+self.get_value(detail_url)
    
                # 爬取时间
                p_time = time.strftime("%Y-%m-%d %X", time.localtime())
                # 实例化
                item = Day0514Item()
                item['name'] = name
                item['mileage'] = mileage
                item['city'] = city
                item['price'] = price
                item['new_price'] = new_price
                item['registered_date'] = registered_date
                item['detail_url'] = detail_url
                item['p_time'] = p_time
    
                yield scrapy.Request(
                    url=detail_url,
                    callback=self.parse_detail,
                    meta={'data':item},
                    encoding = 'utf-8',
                    dont_filter=True
                )
    
        # 详情页解析数据
        def parse_detail(self, response):
            print(response.url)
    
            li_box = response.xpath('//div[@class="row parameter-configure"]//div[2]/ul')[0]
            # 排量
            displacement = li_box.xpath('./li[1]//text()').extract()
            displacement = ''.join([i.strip() for i in displacement])
            # 油耗
            oil_wear = li_box.xpath('./li[2]//text()').extract()
            oil_wear = ''.join(oil_wear)
    
            # 长宽高
            three_high = li_box.xpath('./li[3]//text()').extract()
            three_high = ''.join(three_high)
            # 车身类型
            body_way = li_box.xpath('./li[4]//text()').extract()
            body_way = ''.join(body_way)
            #驱动方式
            trunk_cap = li_box.xpath('./li[5]//text()').extract()
            trunk_cap = ''.join(trunk_cap)
    
            ul = response.xpath('//div[@class="row parameter-configure"]//div[1]/ul')[0]
            # 品牌型号
            brand_type = ul.xpath('./li[1]/span//text()').extract()
            brand_type = ''.join(brand_type)
            # 牌照所在地
            loc_of_lic = ul.xpath('./li[2]//text()').extract()
            loc_of_lic = ''.join([i.strip() for i in loc_of_lic])
            # 发动机
            engine = ul.xpath('./li[3]//text()').extract()
            engine = ''.join(engine)
            # 驱动方式
            drive_type = ul.xpath('./li[4]//text()').extract()
            drive_type = ''.join(drive_type)
            # 车辆级别
            che_level = ul.xpath('./li[5]//text()').extract()
            che_level = ''.join([i.strip()  for i in che_level])
            # 变速箱
            transmission = response.xpath('//div[@class="summary-attrs"]/dl[3]//text()').extract()
            transmission = ''.join([i.strip() for i in transmission])
    
            # 爬取时间
            p_time = time.strftime("%Y-%m-%d %X", time.localtime())
    
    
            item = response.meta['data']
            item['displacement'] = displacement
            item['oil_wear'] = oil_wear
            item['three_high'] = three_high
            item['body_way'] = body_way
            item['trunk_cap'] = trunk_cap
            item['brand_type'] = brand_type
            item['loc_of_lic'] = loc_of_lic
            item['engine'] = engine
            item['drive_type'] = drive_type
            item['che_level'] = che_level
            item['transmission'] = transmission
            item['p_time'] = p_time
            item['ip'] = '10.10.65.168'
            yield item
    
        
        # 判空
        def get_value(self,value):
            if value:
                value = value[0]
            else:
                value = 1
            return value

不要忘了settings.py配置文件中的各项设置：

![20190518100003432.png][]

将对应的管道创建的类名打开，以便执行数据的存储操作

![2019051810161594.png][]

最后是pipelines.py文件执行连接mongo数据库并进行数据存储的操作

import pymongo
    
    class Day0514Pipeline(object):
        # def process_item(self, item, spider):
        #     return item
        def __init__(self):
             # 链接数据库
            self.client = pymongo.MongoClient(host='10.10.65.227',port=27017)
            # 创建库
            self.db = self.client['TaoCheWang1']
            # 创建表
            self.table = self.db['cars']
    
        def process_item(self, item, spider):
            # with open('taoche.txt', 'a', encoding='utf-8') as fp:
            #     fp.write(json.dumps(dict(item), ensure_ascii=False) + '\n')
            # 往表中插入字段
            self.table.insert(dict(item))
            return item

在项目执行之前要确保mongo数据库处于运行状态

在桌面，右击“计算机”，选择“管理”，选择“服务和应用”，如下图

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70 1][]

如图所示表示mongodb数据库处于运行状态。

运行项目

通过mongo可视化工具Robo 3T 1.2.1查看数据情况:

点击：file-->Connect-->Connect 创建与MongoDB数据库的连接

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70 2][]

然后就可以查看数据库了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70 3][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70]: /images/20220130/360e06be88bb45d4940720da7b54e726.png
[20190518100003432.png]: /images/20220130/68952f6a54ce47c8b5ccfc8b36c17b70.png
[2019051810161594.png]: /images/20220130/84718d9d597948e0a248ca562b403546.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70 1]: /images/20220130/0d820435f8a1461984735c340b532988.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70 2]: /images/20220130/16a16ff17a7847659937475b50339ae0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDIzOTU0MQ_size_16_color_FFFFFF_t_70 3]: /images/20220130/9aafcee817684171a6bba46f12bc6ae7.png