【python实现网络爬虫（12）】JSON解析之爬取腾讯新闻

深藏阁楼爱情的钟 2023-05-28 15:44 136阅读 0赞

**目标网址：[腾讯新闻][Link 1]，页面如下**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70]

##### 1. 寻找`json`接口 #####

在目标页面点击鼠标右键进行’检查’，然后选择`'Network'`，再点击’网页刷新’按钮，接着在右下区域内弹出的内容上选择具有`pull_url`标识的文件，最后点击`'Preview'`选项即可。图解如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70 1]  
比如就以当前这个页面来看，获取这个`json`的接口就是点击`'Preview'`旁边的`'Headers'`，选择’Request URL:'后面的网址，就为请求数据的接口。如下  
![在这里插入图片描述][2020031618181712.png]

##### 2. 尝试获取数据 #####

import requests
    import json
    
    def get_json():
    	url = 'https://i.match.qq.com/ninja/fragcontent?pull_urls=news_top_2018&callback=__jp0'
    	html = requests.get(url)
    	print(html.text)
    
    get_json()

–> 输出结果为：（可以看到最后输出的结果并不符合json转python数组的格式要求，可以尝试把数据格式转为一致）

__jp0(
    [{ "title":"\u4e60\u8fd1\u5e73\u56de\u4fe1\u52c9\u52b1\u5317\u5927\u63f4\u9102\u533b\u7597\u961f90\u540e\u515a\u5458","url":"https:\/\/new.qq.com\/omn\/20200316\/20200316A0CI9E00.html","article_id":"20200316a0ci9e00","comment_id":"0","group":"0"},
    { "title":"\u4e60\u8fd1\u5e73\u6c42\u662f\u6587\u7ae0\u91cc\u7684\u5173\u952e","url":"https:\/\/new.qq.com\/omn\/TWF20200\/TWF2020031604303300.html","article_id":"twf2020031604303300","comment_id":"0","group":1}
    ,{ "title":"\u82f1\u96c4\uff0c\u5fc5\u987b\u8ba9\u4f60\u4eec\u767b\u201c\u53f0\u201d\u4eae\u76f8\uff01","url":"https:\/\/new.qq.com\/omn\/20200315\/20200315A0GHBK00.html","article_id":"20200315a0ghbk00","comment_id":"0","group":1},
    { "title":"\u5b9a\u5411\u964d\u51c6\u7a33\u4fe1\u5fc3 \u5229\u7387\u4e0b\u884c\u6709\u7a7a\u95f4","url":"https:\/\/new.qq.com\/omn\/20200316\/20200316A049RE00.html","article_id":"20200316a049re00","comment_id":"0","group":2},
    { "title":"\u75ab\u60c5\u5f71\u54cd\u4e0b\u7684\u4e2d\u56fd\u7ecf\u6d4e\u89c2\u5bdf","url":"https:\/\/new.qq.com\/omn\/20200316\/20200316A04HX200.html","article_id":"20200316a04hx200","comment_id":"0","group":2},
    { "title":"\u5927\u533b\u5f20\u4f2f\u793c\u2014\u2014\u4ed6\u662f\u957f\u8005\uff0c\u662f\u7236\u4eb2\uff0c\u66f4\u662f\u5171\u4ea7\u515a\u5458\uff01","url":"https:\/\/new.qq.com\/omn\/20200316\/20200316A0DS7U00.html","article_id":"20200316a0ds7u00","comment_id":"0","group":"0"},
    { "title":"\u4ed6\u4eec\u662f\u8bb0\u8005\uff0c\u4ed6\u4eec\u662f\u6218\u58eb","url":"https:\/\/new.qq.com\/omn\/20200316\/20200316A07XO800.html","article_id":"20200316a07xo800","comment_id":"0","group":3},
    { "title":"\u6218\u201c\u75ab\u201d\u4e2d\u7684\u9752\u6625\u4e4b\u6b4c","url":"https:\/\/new.qq.com\/omn\/20200316\/20200316A07ZYG00.html","article_id":"20200316a07zyg00","comment_id":"0","group":3},
    { "title":"\u3010\u6218\u201c\u75ab\u201d\u8bf4\u7406\u3011\u4ee5\u4eba\u6c11\u4e3a\u4e2d\u5fc3\uff1a\u75ab\u60c5\u9632\u63a7\u7684\u4ef7\u503c\u903b\u8f91","url":"https:\/\/new.qq.com\/rain\/a\/20200316A07IIW00","article_id":"20200316A07IIW00","comment_id":"0","group":"0"},
    { "title":"\u575a\u6301\u5411\u79d1\u5b66\u8981\u7b54\u6848\u8981\u65b9\u6cd5","url":"https:\/\/new.qq.com\/omn\/20200315\/20200315A0FJZO00.html","article_id":"20200315a0fjzo00","comment_id":"0","group":4},
    ])

尝试转变数据（去掉列表两侧的多余字符）

print(json.loads((html.text)[6:-2]))

–> 输出结果为：（这里就是进行字符串的切片，将数据转为合适的数据，上下数据一致，均为13条）  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70 2]

##### 3. 获取标题和url #####

for i in json.loads((html.text)[6:-2]):
    		# print(i)
    		print('新闻标题为：\n{}\n'.format(i['title']))
    		print('新闻链接为：\n{}\n----------\n'.format(i['url']))

–> 输出结果为：（获取的内容和原网页的内容是一致的）  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70 3]

##### 4. 全部代码 #####

import requests
    import json
    
    def get_json():
    	url = 'https://i.match.qq.com/ninja/fragcontent?pull_urls=news_top_2018&callback=__jp0'
    	html = requests.get(url)
    	# print(html.text)
    	# print(json.loads((html.text)[6:-2]))
    	for i in json.loads((html.text)[6:-2]):
    		# print(i)
    		print('新闻标题为：\n{}\n'.format(i['title']))
    		print('新闻链接为：\n{}\n----------\n'.format(i['url']))
    
    get_json()

[Link 1]: https://news.qq.com/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70]: /images/20230528/36ccea9fe6874d6caf7c9a023618d5e1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70 1]: /images/20230528/4a5ff6ba6f4f4d4288f1fafaefc97dda.png
[2020031618181712.png]: /images/20230528/c8ed0bce133f4d5fbe774c85aefe77f8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70 2]: /images/20230528/5dd14700146148589477f48e22ee988f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x5c184Mjg_size_16_color_FFFFFF_t_70 3]: /images/20230528/3596bbf6c39d4b96ba587550c09d6ebd.png