登录并爬取评论和用户内容

ゝ一纸荒年。 2022-01-31 13:21 183阅读 0赞

学习目标

## \*\*4.1 \*\* 实战大项目 ##

1.  实战大项目：模拟登录丁香园，并抓取论坛页面所有的人员基本信息与回复帖子内容。
2.  丁香园论坛：[http://www.dxy.cn/bbs/thread/626626\#626626][http_www.dxy.cn_bbs_thread_626626_626626] 。
3.  参考资料：[https://blog.csdn.net/naonao77/article/details/88316754][https_blog.csdn.net_naonao77_article_details_88316754] (Task7的参考资料，任务结束之后再发出来)

1、打开丁香园登录地址是二维码，无法输入账号密码登录

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70][]

2、用了个简单的方法

使用selenium点击返回电脑登录，进入账号密码登录页面

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70 1][]

3、后面就一样的方式去登陆，再获取内容，附代码

from selenium import webdriver
    import time
    from lxml import etree
    import requests
    
    def login():
        browser = webdriver.Chrome()
        browser.get('https://auth.dxy.cn/accounts/login')
        #点击返回电脑登录
        browser.find_element_by_xpath('/html/body/div[2]/div[2]/div[1]/a[2]').click()
        #使用账号密码登录
        username = browser.find_element_by_name('username')
        username.send_keys('*******')
        password = browser.find_element_by_name('password')
        password.send_keys('********')
        browser.find_element_by_xpath('//*[@id="user"]/div[1]/div[3]/button').click()
        time.sleep(5)
        cookie = browser.get_cookies()
        cookie_dict = {i['name']:i['value'] for i in cookie}
        return cookie_dict
    
    def get_content():
        cookies = login()
        url = 'http://www.dxy.cn/bbs/thread/626626#626626'
        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}
        response = requests.get(url=url,headers=headers,cookies=cookies)
        html = etree.HTML(response.text)
        contents = html.xpath('//td[@class="postbody"]')#element类型列表
        users = html.xpath('//div[@class="auth"]')
        for i in range(0,len(contents)):
            content1 = contents[i].xpath('string(.)').strip()
            user1 = users[i].xpath('string(.)')
            print(user1+':'+content1)
            print('='*100)
            result = user1+':'+content1
            file_content = open('testing.txt','w',encoding="utf-8")
            file_content.write(result+"\n")
            file_content.write('*'*80 +"\n")
            file_content.close()
    
    if __name__ == '__main__':
        get_content()

[http_www.dxy.cn_bbs_thread_626626_626626]: http://www.dxy.cn/bbs/thread/626626#626626
[https_blog.csdn.net_naonao77_article_details_88316754]: https://blog.csdn.net/naonao77/article/details/88316754
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70]: /images/20220131/241be4f2d2c24463ac24df28b99b1047.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70 1]: /images/20220131/60040422b8bc43559b19c7c23cddc35c.png