python爬虫爬取腾讯新闻

红太狼 2021-07-27 01:32 882阅读 0赞

python爬虫爬取腾讯新闻

话不多说,直接上代码!

  1. import requests
  2. from bs4 import BeautifulSoup
  3. def getHTMLText(url):
  4. try:
  5. r = requests.get(url, timeout = 30)
  6. r.raise_for_status()
  7. #r.encoding = 'utf-8'
  8. return r.text
  9. except:
  10. return ""
  11. def getContent(url):
  12. html = getHTMLText(url)
  13. # print(html)
  14. soup = BeautifulSoup(html, "html.parser")
  15. title = soup.select("div.hd > h1")
  16. print(title[0].get_text())
  17. time = soup.select("div.a_Info > span.a_time")
  18. print(time[0].string)
  19. author = soup.select("div.qq_articleFt > div.qq_toolWrap > div.qq_editor")
  20. print(author[0].get_text())
  21. paras = soup.select("div.Cnt-Main-Article-QQ > p.text")
  22. for para in paras:
  23. if len(para) > 0:
  24. print(para.get_text())
  25. print()
  26. fo = open("text.txt", "w+")
  27. fo.writelines(title[0].get_text() + "\n")
  28. fo.writelines(time[0].get_text() + "\n")
  29. for para in paras:
  30. if len(para) > 0:
  31. fo.writelines(para.get_text() + "\n\n")
  32. fo.writelines(author[0].get_text() + '\n')
  33. fo.close()
  34. article = {
  35. 'Title' : title[0].get_text(),
  36. 'Time' : time[0].get_text(),
  37. 'Paragraph' : paras,
  38. 'Author' : author[0].get_text()
  39. }
  40. print(article)
  41. def main():
  42. url = "http://news.qq.com/a/20170504/012032.htm"
  43. getContent(url);
  44. main()

运行结果:
在这里插入图片描述

发表评论

表情:
评论列表 (有 0 条评论,882人围观)

还没有评论,来说两句吧...

相关阅读