Python3[爬虫实战] 爬虫之scrapy爬取爱上程序网存MongoDB（android模块）

柔情只为你懂 2021-06-11 15:10 559阅读 0赞

### 爱上程序网 ###

> （[http://www.aichengxu.com/android][http_www.aichengxu.com_android]）

缘由：这个网站是在工作中谷歌找问题找出来的，然后发现里面的文章挺多的， 毕竟自己平时比较喜欢看技术文章，什么都想懂，什么都懂得不深入，这不，想要转爬虫工作的，现在还在继续android开发中。。

废话不多说。

来个数据库的结果：  
![这里写图片描述][SouthEast]

--------------------

为什么暂时是这些呢？因为用的循环用了10000次，可能还会多，数据爬取到了2013年了，想必这个网站确实是建站比较久了，可能遍历的话还会有更多吧。在第二次爬取数据的时候一共爬取了24万左右的数据，可能还有多吧。

这里总结一下经验：  
之前在pycharm中直接在文件夹中进行写scrapy工程导入模块一直出错，原因是导入不进去。可以看以下截图对比：

![这里写图片描述][SouthEast 1]

#### 可以看出来下面那个项目在工程中没有变黑，所以在spider中导入items里面的类，是一直导入报错的，说是no module named ‘’xxx‘’ ####

### 解决方法： ###

> 需要我们在file菜单栏中重新open。。然后导入该项目，再关联到该工程中，add xxx的 然后项目就变成黑色了， 这样子导包就正常了。

抓取的数据主要是：

#### **阅读量，标题，标题链接（内容详情页），内容，时间** ####

--------------------

![这里写图片描述][SouthEast 2]

#### 使用scrapy 进行数据的爬取，在parse（）方法中用的还不是很熟练，一边调，一边用把。 ####

##### 步骤一： #####

settings文件的配置：

# -*- coding: utf-8 -*-
    
    # Scrapy settings for aichengxu project
    #
    # For simplicity, this file contains only settings considered important or
    # commonly used. You can find more settings consulting the documentation:
    #
    # http://doc.scrapy.org/en/latest/topics/settings.html
    # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
    # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
    
    BOT_NAME = 'aichengxu'
    
    SPIDER_MODULES = ['aichengxu.spiders']
    NEWSPIDER_MODULE = 'aichengxu.spiders'
    
    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    # USER_AGENT = 'aichengxu (+http://www.yourdomain.com)'
    
    # Obey robots.txt rules
    ROBOTSTXT_OBEY = False
    
    # Configure maximum concurrent requests performed by Scrapy (default: 16)
    # CONCURRENT_REQUESTS = 32
    
    # Configure a delay for requests for the same website (default: 0)
    # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
    # See also autothrottle settings and docs
    # DOWNLOAD_DELAY = 3
    # The download delay setting will honor only one of:
    # CONCURRENT_REQUESTS_PER_DOMAIN = 16
    # CONCURRENT_REQUESTS_PER_IP = 16
    
    # Disable cookies (enabled by default)
    # COOKIES_ENABLED = False
    
    # Disable Telnet Console (enabled by default)
    # TELNETCONSOLE_ENABLED = False
    
    # Override the default request headers:
    DEFAULT_REQUEST_HEADERS = {
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Accept-Language': 'zh-CN,zh;q=0.8',
        'Cache-Control': 'max-age=0',
        'Connection': 'keep-alive',
        'Cookie': 'ras=24656333; cids_AC31=24656333',
        'Host': 'www.aichengxu.com',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36',
    }
    
    # 配置mongoDB
    MONGO_HOST = "127.0.0.1"  # 主机IP
    MONGO_PORT = 27017  # 端口号
    MONGO_DB = "aichengxu2"  # 库名
    MONGO_COLL = "ai_chengxu"  # collection
    
    # Enable or disable spider middlewares
    # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
    # SPIDER_MIDDLEWARES = { 
    # 'aichengxu.middlewares.AichengxuSpiderMiddleware': 543,
    # }
    
    # Enable or disable downloader middlewares
    # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
    # DOWNLOADER_MIDDLEWARES = { 
    # 'aichengxu.middlewares.MyCustomDownloaderMiddleware': 543,
    # 'aichengxu.middlewares.MyCustomDownloaderMiddleware': 543,
    # 'aichengxu.middlewares.MyCustomDownloaderMiddleware': 543,
    # }
    
    # Enable or disable extensions
    # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
    # EXTENSIONS = { 
    # 'scrapy.extensions.telnet.TelnetConsole': None,
    # }
    
    # Configure item pipelines
    # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
    ITEM_PIPELINES = {
        'aichengxu.pipelines.AichengxuPipeline': 300,
        'aichengxu.pipelines.DuoDuoMongo': 300,
        'aichengxu.pipelines.JsonWritePipline': 300,
    }
    
    # Enable and configure the AutoThrottle extension (disabled by default)
    # See http://doc.scrapy.org/en/latest/topics/autothrottle.html
    # AUTOTHROTTLE_ENABLED = True
    # The initial download delay
    # AUTOTHROTTLE_START_DELAY = 5
    # The maximum download delay to be set in case of high latencies
    # AUTOTHROTTLE_MAX_DELAY = 60
    # The average number of requests Scrapy should be sending in parallel to
    # each remote server
    # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
    # Enable showing throttling stats for every response received:
    # AUTOTHROTTLE_DEBUG = False
    
    # Enable and configure HTTP caching (disabled by default)
    # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
    # HTTPCACHE_ENABLED = True
    # HTTPCACHE_EXPIRATION_SECS = 0
    # HTTPCACHE_DIR = 'httpcache'
    # HTTPCACHE_IGNORE_HTTP_CODES = []
    # HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

当然有用的也就那几个没有注释掉的，这是一种习惯吧，不然后面有时候都在想为什么scrapy没有跑起来，或者直接报错，呵呵。

#### 步骤二： ####

进行items里面的定义：

# -*- coding: utf-8 -*-
    
    # Define here the models for your scraped items
    #
    # See documentation in:
    # http://doc.scrapy.org/en/latest/topics/items.html
    
    import scrapy
    
    
    class AichengxuItem(scrapy.Item):
        # define the fields for your item here like:
        # name = scrapy.Field()
        pass
    
    class androidItem(scrapy.Item):
        # 阅读量
        count = scrapy.Field()
        # 标题
        title = scrapy.Field()
        # 链接
        titleLink = scrapy.Field()
        # 描述
        desc = scrapy.Field()
        # 时间
        time = scrapy.Field()

#### 步骤三： ####

spiders里面的代码编写

# -*- coding: utf-8 -*-
    # @Time : 2017/8/25 21:54
    # @Author : 蛇崽
    # @Email : 17193337679@163.com
    # @File : aichengxuspider.py 爱程序网 www.aichengxu.com
    
    import scrapy
    from aichengxu.items import androidItem
    import logging
    class aiChengxu(scrapy.Spider):
    
        name = 'aichengxu'
    
        allowed_domains = ['www.aichengxu.com']
    
        start_urls = ["http://www.aichengxu.com/android/{}/".format(n) for n in range(1,10000)]
    
        def parse(self, response):
            node_list = response.xpath("//*[@class='item-box']")
            print('nodelist',node_list)
            for node in node_list:
                android_item = androidItem()
                count = node.xpath("./div[@class='views']/text()").extract()
                title_link = node.xpath("./div[@class='bd']/h3/a/@href").extract()
                title = node.xpath("./div[@class='bd']/h3/a/text()").extract()
                desc = node.xpath("./div[@class='bd']/div[@class='desc']/text()").extract()
                time = node.xpath("./div[@class='bd']/div[@class='item-source']/span[2]").extract()
                print(count,title,title_link,desc,time)
                android_item['title'] = title
                android_item['titleLink'] = title_link
                android_item['desc'] = desc
                android_item['count'] = count
                android_item['time'] = time
                yield android_item

这里对yield这个关键字说一下自己的看法：就是返回该条目，并继续执行下一个任务，或者说，把item返回去，然后继续下一条爬虫吧。

#### 步骤四： ####

piplines代码：

# -*- coding: utf-8 -*-
    
    # Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
    import json
    
    import pymongo
    from scrapy.conf import settings
    
    class AichengxuPipeline(object):
        def process_item(self, item, spider):
            return item
    class DuoDuoMongo(object):
        def __init__(self):
            self.client = pymongo.MongoClient(host=settings['MONGO_HOST'], port=settings['MONGO_PORT'])
            self.db = self.client[settings['MONGO_DB']]
            self.post = self.db[settings['MONGO_COLL']]
    
        def process_item(self, item, spider):
            postItem = dict(item)
            self.post.insert(postItem)
            return item
    
    # 写入json文件
    class JsonWritePipline(object):
        def __init__(self):
            self.file = open('爱上程序网2.json','w',encoding='utf-8')
    
        def process_item(self,item,spider):
            line  = json.dumps(dict(item),ensure_ascii=False)+"\n"
            self.file.write(line)
            return item
    
        def spider_closed(self,spider):
            self.file.close()

这里主要写了两种类型的存储，一个是mogodb，一个是json文件的存储

这样，代码就爬完了，然后就这样子结束了，代码久了不写，生疏了许多。

> 有一个疑问：这里爬取了24万的数据，没有在中途因为cookie或者代理的原因被静止，可能在settings里面设置了请求头的缘故吧，`DEFAULT_REQUEST_HEADERS`

再说一下这个网站为什么自己为什么想爬，总感觉自己对技术驱动性很强吧，当然啦，android并不是我最喜欢的吧，可能或者有些畏惧它，android适配很恶心的。机型，屏幕什么的。

这个网站还有很多其他条目：  
![这里写图片描述][SouthEast 3]

如果全部爬取下来，老实说，我没有进行爬取过，有想法的小伙伴可是试一试，最后把代码上传到我的github上：

[我的github][github]

[http_www.aichengxu.com_android]: http://www.aichengxu.com/android
[SouthEast]: /images/20210529/e3f82dd875a94fff8fb22ec136faeb62.png
[SouthEast 1]: /images/20210529/a9bee2f41c4045c9b9e9f5397dade8e3.png
[SouthEast 2]: /images/20210529/cc73e39c0a444cfd9c3adbcbac105ed7.png
[SouthEast 3]: /images/20210529/540c29efd49e486da25397a2c010f808.png
[github]: https://github.com/643435675/PyStudy