【爬虫】对某某贴吧主页的爬虫分析+源码

逃离我推掉我的手 2024-03-17 00:39 24阅读 0赞

## 1. 网站分析 ##

想要的内容有标题、时间和帖子跳转链接

![e3bc472c475f4b0bb5522cbd4c2fc6c5.png][]

查看网站源代码，发现想要的内容就在里面，那就好办了，直接上正则，当然beautifulsoup也不是不可以

## ![a8e231e082aa4b8ab0725c21a842f357.png][] ##

## 2. Python源码 ##

import requests
    import re
    from prettytable import PrettyTable
    
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0'
    }
    x = PrettyTable(["标题", "时间", "链接"])
    x.align["标题"] = "l"
    // 这里会整合三页内容，想要多少页，就把101改成50*(页数-1)+1
    for i in range(0, 101, 50):
        resp = requests.get(f'https://tieba.baidu.com/f?kw=%E5%8D%9A%E5%B0%94%E7%89%B9&ie=utf-8&pn={i}', headers=headers)
        with open(file='1.html', mode='w', encoding='utf-8') as f:
            f.write(resp.text)
        obj = re.compile('<a rel="noopener" href="/p/(?P<url>.*?)" title="(?P<title>.*?)".*?<span class="pull-right is_show_create_time" title="创建时间">(?P<time>.*?)</span>', re.S)
        title = obj.finditer(resp.text)
        for i in title:
            x.add_row([i.group('title'), i.group('time'), 'https://tieba.baidu.com/p/' + i.group('url')])
    print(x)

## 3. 效果展示 ##

![11d5232f9eef45588b5fb958f27c941d.png][]

> ps：好久没玩爬虫了，但是在网页版逛吧时看到很多广告让我很不爽，故写下此脚本。

[e3bc472c475f4b0bb5522cbd4c2fc6c5.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/15/455b552d845d44c382c48a8f30151561.png
[a8e231e082aa4b8ab0725c21a842f357.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/15/f643d47e62e641259d2b156384c34935.png
[11d5232f9eef45588b5fb958f27c941d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/15/e57cd2419c484c63b3bb2c617b1acfd6.png