爬取猫眼电影前100名（获取图片并记录名字跟排名）

今天药忘吃喽~ 2022-05-22 04:23 180阅读 0赞

## 简述 ##

代码实现上，这个可以说是不难的。但是需要注意的是，这里的有一些细节得注意一下。

*  协程的，gevent.joinall() 需要的是一个可以迭代的对象，但是内容必须是gevent.spawn()这个类
 *  这里有些小东西，比如在network上得到的包的内的数据（也就是直接爬取得到的数据，其实不是完全的数据，会有一些对于源码的部分的修改）。这种修改导致一开始，如果是看着**检查**的话，就会有bug。

## 代码 ##

import requests
    import re
    import gevent
    from gevent import monkey
    
    monkey.patch_all(select=False)
    
    
    def getPicture(url, name):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'
        }
        res = requests.get(url, headers=headers)
        if res.status_code == 200:
            with open('./picture/' + name + '.png', 'wb') as f:
                f.write(res.content)
        else:
            print(url)
    
    
    def get_one_page(url):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'
        }
        res = requests.get(url, headers=headers)
        if res.status_code == 200:
            pattern = re.compile('<dd>.*?board-index-.*?>(.*?)</i>.*?data-src="(.*?)".*?alt="(.*?)"', re.S)
            data = re.findall(pattern, res.text)
            waitlist = [gevent.spawn(getPicture, d[1], d[0] + '-' + d[2]) for d in data]
            gevent.joinall(waitlist)
        else:
            print(url)
    
    
    if __name__ == '__main__':
        url = 'http://maoyan.com/board/4?offset=%d'
        waitlist = [gevent.spawn(get_one_page, url % (d * 10)) for d in range(10)]
        gevent.joinall(waitlist)

import os
    import requests
    from lxml import etree
    import gevent
    from gevent import monkey
    
    monkey.patch_all(select=False)
    
    def getPicture(url, name):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'
        }
        res = requests.get(url, headers=headers)
        if res.status_code == 200:
            with open('./picture/' + name + '.png', 'wb') as f:
                f.write(res.content)
        else:
            print(url)
    
    def get_one_page(url):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'
        }
        res = requests.get(url, headers=headers)
        if res.status_code == 200:
            html = etree.HTML(res.text)
            i = html.xpath('//i[contains(@class, "board")]/text()')
            title = html.xpath('//a[contains(@class, "image")]/@title')
            img = html.xpath('//img/@data-src')
            waitlist = [gevent.spawn(getPicture, img[t], i[t] + '-' + title[t]) for t in range(10)]
            gevent.joinall(waitlist)
        else:
            print(url)
    
    def createfile():
        if not os.path.exists('./picture/'):
            os.mkdir('./picture')
    
    if __name__ == '__main__':
        createfile()
        url = 'http://maoyan.com/board/4?offset=%d'
        waitlist = [gevent.spawn(get_one_page, url % (d * 10)) for d in range(10)]
        gevent.joinall(waitlist)