python爬虫开发之使用Python爬虫库requests多线程抓取猫眼电影TOP100实例

你的名字 2023-07-16 03:49 20阅读 0赞

这篇文章主要介绍了python爬虫开发之使用Python爬虫库requests多线程抓取猫眼电影TOP100实例,需要的朋友可以参考下  
使用Python爬虫库requests多线程抓取猫眼电影TOP100思路：

查看网页源代码  
抓取单页内容  
正则表达式提取信息  
猫眼TOP100所有信息写入文件  
多线程抓取  
运行平台：windows  
Python版本：Python 3.7.  
IDE:Sublime Text  
浏览器：Chrome浏览器  
1.查看猫眼电影TOP100网页原代码  
按F12查看网页源代码发现每一个电影的信息都在“

”标签之中。 ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70]  
点开之后，信息如下： ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 1]  
2.抓取单页内容  
在浏览器中打开猫眼电影网站，点击“榜单”，再点击“TOP100榜”如下图： ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 2]  
接下来通过以下代码获取网页源代码：

#-*-coding:utf-8-*-
    import requests
    from requests.exceptions import RequestException
      
    #猫眼电影网站有反爬虫措施，设置headers后可以爬取
    headers = { 
        'Content-Type': 'text/plain; charset=UTF-8',
        'Origin':'https://maoyan.com',
        'Referer':'https://maoyan.com/board/4',
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
        }
      
    #爬取网页源代码
    def get_one_page(url,headers):
        try:
            response =requests.get(url,headers =headers)
            if response.status_code == 200:
                return response.text
            return None
        except RequestsException:
            return None
      
    def main():
        url = "https://maoyan.com/board/4"
        html = get_one_page(url,headers)
        print(html)
      
    if __name__ == '__main__':
        main()

执行结果如下：![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 3]  
3.正则表达式提取信息  
上图标示信息即为要提取的信息，代码实现如下：

#-*-coding:utf-8-*-
 import requests
 import re
 from requests.exceptions import RequestException
 
 #猫眼电影网站有反爬虫措施，设置headers后可以爬取
 headers = { 
 'Content-Type': 'text/plain; charset=UTF-8',
 'Origin':'https://maoyan.com',
 'Referer':'https://maoyan.com/board/4',
 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
 }
 
 #爬取网页源代码
 def get_one_page(url,headers):
 try:
 response =requests.get(url,headers =headers)
 if response.status_code == 200:
 return response.text
 return None
 except RequestsException:
 return None
 
 #正则表达式提取信息
 def parse_one_page(html):
 pattern = re.compile('<dd>.*?board-index.*?>(\d+).*?data-src="(.*?)".*?name"><a'
 +'.*?>(.*?)</a>.*?star">(.*?).*?releasetime">(.*?).*?integer">(.*?).*?fraction">(.*?).*?</dd>',re.S)
 items = re.findall(pattern,html)
 for item in items:
 yield{ 
 'index':item[0],
 'image':item[1],
 'title':item[2],
 'actor':item[3].strip()[3:],
 'time':item[4].strip()[5:],
 'score':item[5]+item[6]
 }
 
 def main():
 url = "https://maoyan.com/board/4"
 html = get_one_page(url,headers)
 for item in parse_one_page(html):
 print(item)
 
 if __name__ == '__main__':
 main()

执行结果如下：![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 4]  
4.猫眼TOP100所有信息写入文件  
上边代码实现单页的信息抓取，要想爬取100个电影的信息，先观察每一页url的变化，点开每一页我们会发现url进行变化，原url后面多了‘？offset=0’,且offset的值变化从0,10,20，变化如下：![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 5]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 6]  
代码实现如下：

#-*-coding:utf-8-*-
 import requests
 import re
 import json
 import os
 from requests.exceptions import RequestException
 
 #猫眼电影网站有反爬虫措施，设置headers后可以爬取
 headers = { 
 'Content-Type': 'text/plain; charset=UTF-8',
 'Origin':'https://maoyan.com',
 'Referer':'https://maoyan.com/board/4',
 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
 }
 
 #爬取网页源代码
 def get_one_page(url,headers):
 try:
 response =requests.get(url,headers =headers)
 if response.status_code == 200:
 return response.text
 return None
 except RequestsException:
 return None
 
 #正则表达式提取信息
 def parse_one_page(html):
 pattern = re.compile('<dd>.*?board-index.*?>(\d+).*?data-src="(.*?)".*?name"><a'
 +'.*?>(.*?)</a>.*?star">(.*?).*?releasetime">(.*?).*?integer">(.*?).*?fraction">(.*?).*?</dd>',re.S)
 items = re.findall(pattern,html)
 for item in items:
 yield{ 
 'index':item[0],
 'image':item[1],
 'title':item[2],
 'actor':item[3].strip()[3:],
 'time':item[4].strip()[5:],
 'score':item[5]+item[6]
 }
 #猫眼TOP100所有信息写入文件
 def write_to_file(content):
 #encoding ='utf-8',ensure_ascii =False,使写入文件的代码显示为中文
 with open('result.txt','a',encoding ='utf-8') as f:
 f.write(json.dumps(content,ensure_ascii =False)+'\n')
 f.close()
 #下载电影封面
 def save_image_file(url,path):
 
 jd = requests.get(url)
 if jd.status_code == 200:
 with open(path,'wb') as f:
 f.write(jd.content)
 f.close()
 
 def main(offset):
 url = "https://maoyan.com/board/4?offset="+str(offset)
 html = get_one_page(url,headers)
 if not os.path.exists('covers'):
 os.mkdir('covers') 
 for item in parse_one_page(html):
 print(item)
 write_to_file(item)
 save_image_file(item['image'],'covers/'+item['title']+'.jpg')
 
 if __name__ == '__main__':
 #对每一页信息进行爬取
 for i in range(10):
 main(i*10)

爬取结果如下：![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 7]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 8]  
5.多线程抓取  
进行比较，发现多线程爬取时间明显较快：![在这里插入图片描述][20200317214319821.png]  
多线程：  
![在这里插入图片描述][20200317214329957.png]  
以下为完整代码：

#-*-coding:utf-8-*-
 import requests
 import re
 import json
 import os
 from requests.exceptions import RequestException
 from multiprocessing import Pool
 #猫眼电影网站有反爬虫措施，设置headers后可以爬取
 headers = { 
 'Content-Type': 'text/plain; charset=UTF-8',
 'Origin':'https://maoyan.com',
 'Referer':'https://maoyan.com/board/4',
 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
 }
 
 #爬取网页源代码
 def get_one_page(url,headers):
 try:
 response =requests.get(url,headers =headers)
 if response.status_code == 200:
 return response.text
 return None
 except RequestsException:
 return None
 
 #正则表达式提取信息
 def parse_one_page(html):
 pattern = re.compile('<dd>.*?board-index.*?>(\d+).*?data-src="(.*?)".*?name"><a'
 +'.*?>(.*?)</a>.*?star">(.*?).*?releasetime">(.*?).*?integer">(.*?).*?fraction">(.*?).*?</dd>',re.S)
 items = re.findall(pattern,html)
 for item in items:
 yield{ 
 'index':item[0],
 'image':item[1],
 'title':item[2],
 'actor':item[3].strip()[3:],
 'time':item[4].strip()[5:],
 'score':item[5]+item[6]
 }
 #猫眼TOP100所有信息写入文件
 def write_to_file(content):
 #encoding ='utf-8',ensure_ascii =False,使写入文件的代码显示为中文
 with open('result.txt','a',encoding ='utf-8') as f:
 f.write(json.dumps(content,ensure_ascii =False)+'\n')
 f.close()
 #下载电影封面
 def save_image_file(url,path):
 
 jd = requests.get(url)
 if jd.status_code == 200:
 with open(path,'wb') as f:
 f.write(jd.content)
 f.close()
 
 def main(offset):
 url = "https://maoyan.com/board/4?offset="+str(offset)
 html = get_one_page(url,headers)
 if not os.path.exists('covers'):
 os.mkdir('covers') 
 for item in parse_one_page(html):
 print(item)
 write_to_file(item)
 save_image_file(item['image'],'covers/'+item['title']+'.jpg')
 
 if __name__ == '__main__':
 #对每一页信息进行爬取
 pool = Pool()
 pool.map(main,[i*10 for i in range(10)])
 pool.close()
 pool.join()

最后给大家推荐我们的Python学习扣qun：913066266 ，看看前辈们是如何学习的！从基础的python脚本到web开发、爬虫、django、数据挖掘等【PDF，实战源码】，零基础到项目实战的资料都有整理。送给每一位python的小伙伴！每天都有大牛定时讲解Python技术，分享一些学习的方法和需要注意的小细节，[点击加入][Link 1]我们的 python学习者聚集地  
本文主要讲解了使用Python爬虫库requests多线程抓取猫眼电影TOP100数据的实例

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70]: /images/20230528/79acf0f0af2b456dad9e072b24f6028a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 1]: /images/20230528/320fbdfd96684c518f8814f06fecd54b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 2]: /images/20230528/544ae279f75c402682705f70d6e2f4a6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 3]: /images/20230528/c1710586bd2a49d2b3bae3cae891964a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 4]: /images/20230528/ef98d8ba9c0741efa0a2369389cb36b9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 5]: /images/20230528/f829cc5adaf04381a87900e6319426e0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 6]: /images/20230528/4d86dd94acbd41fe80b0333930d76f66.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 7]: /images/20230528/f6d7cd7b2c93468c8661a62d8db1132a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhb3h1bjEy_size_16_color_FFFFFF_t_70 8]: /images/20230528/dafc1ec0159c4c42b5d37e7d1848591e.png
[20200317214319821.png]: /images/20230528/be45700dd56347209169b60af6562628.png
[20200317214329957.png]: /images/20230528/60157d65b4f449dfa24a6971e9f61c13.png
[Link 1]: https://jq.qq.com/?_wv=1027&k=5JIjRvv