scrapy获取网页内容和图片下载

系统管理员 2022-05-17 01:13 273阅读 0赞

首先在终端命令行菜单键+R输入cmd进入命令行

然后再终端输入cd+空格 把储存的地址拉获取:

在存储的路径下创建一个小的项目输入scrapy startproject QiuShi,然后输入cd QiuShi进入项目下再然后输入创建scrapy genspider qiushi qiushibaike.com

创建好最后的结果是如下图:

![70][]

我们直接进入主题:点击qiushi.py进入写代码界面:

![70 1][]

![70 2][]

![70 3][]

获取热图里面的所有内容和图片下载;完整代码:

# -*- coding: utf-8 -*-
    import scrapy
    from  .. items import QiushiItem
    class QiushiSpider(scrapy.Spider):
        name = 'qiushi'
        allowed_domains = ['qiushibaike.com']
        start_urls = ['https://www.qiushibaike.com/imgrank']
    
        def parse(self, response):
            # 获取文本内容
            content_list = response.xpath('//div[@class="content"]/span/text()').extract()
            # 获取图片链接
            thumb_list = response.xpath('//div[@class="thumb"]/a/img/@src').extract()
          
    
            item = QiushiItem()
            for content in content_list:
       
                # 去掉换行
                out_list = content.strip()
                # 内容保存txt格式
                with open('baike.txt', 'a', encoding='utf-8')as f:
                    f.write(out_list)
                    f.write('\n')
    
            for thumb in thumb_list:
                #拼接图片链接
                url = 'https:' + thumb
               
                item['url'] = [url]
                yield item
            # 获取下一页的链接
            next_url=response.xpath('//ul[@class="pagination"]/li[last()]/a/@href').extract()
        
            if len(next_url) != 0:
                # 拼接下一页网址
                nextUrl = 'http://www.qiushibaike.com' + next_url[0]
              
                yield scrapy.Request(url=nextUrl,callback=self.parse)

我们先进入items.py:

![70 4][]

![70 5][]

接着我们进入settings.py配置管道:

![70 6][]

![70 7][]

![70 8][]

![70 9][]

配置好以后我们在终端输入scrapy crawl qiushi来运行:

输出的结果:

![70 10][]

![70 11][]

![70 12][]

这样我们就把热图所有爬取完了...

[70]: /images/20220517/62ccb6fab4814f70adddee14e76c8852.png
[70 1]: /images/20220517/d42e9386592b4e81a8691ea91b0d47b0.png
[70 2]: /images/20220517/9cdb6d663aa04c0bbbcca51177e40298.png
[70 3]: /images/20220517/5dcbb8d806144c39a28607f51ae5566c.png
[70 4]: /images/20220517/3fc3751e6c2842729c6aba19901035df.png
[70 5]: /images/20220517/e89378a9dad242f79690bcd3a9556471.png
[70 6]: /images/20220517/c7fe068bd87b4a57826cfc0dd4c2a79e.png
[70 7]: /images/20220517/98453bf873154ef498ce5e7dc047f855.png
[70 8]: /images/20220517/00c03a969630454aa4ede3a0492b4cd1.png
[70 9]: /images/20220517/f13779e1a7fd4e9a9ecd3ee967ddcc50.png
[70 10]: /images/20220517/9986bba675354ce4b2e7e3541f0a08d0.png
[70 11]: /images/20220517/d9c38a3dda184a97b3d029508c04130a.png
[70 12]: /images/20220517/8c48a57ac9ff49f1aacc90bf32db7930.png