爬取全部的校园新闻

╰+哭是因爲堅強的太久メ 2021-12-21 01:34 365阅读 0赞

作业要求来自于[https://edu.cnblogs.com/campus/gzcc/GZCC-16SE2/homework/2941][https_edu.cnblogs.com_campus_gzcc_GZCC-16SE2_homework_2941]

**1.从新闻url获取新闻详情： 字典,anews**

具体代码如下：

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

anews

结果如下：

![1480585-20190410002346103-140253195.png][]

**2.从列表页的url获取新闻url：列表append(字典) alist**

具体代码如下：

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

import pandas
    import requests
    from bs4 import BeautifulSoup
    from datetime import datetime
    import re
    
    def click(url):
        id = re.findall('(\d{1,5})',url)[-1]#返回所有匹配的字符串的字符串列表的最后一个
        clickUrl = 'http://oa.gzcc.cn/api.php?op=count&id={}&modelid=80'.format(id)
        resClick = requests.get(clickUrl)
        newsClick = int(resClick.text.split('.html')[-1].lstrip("('").rstrip("');"))
        return newsClick
    
    def newsdt(showinfo):
        newsDate = showinfo.split()[0].split(':')[1]
        newsTime = showinfo.split()[1]
        newsDT = newsDate+' '+newsTime
        dt = datetime.strptime(newsDT,'%Y-%m-%d %H:%M:%S')#转换成datetime类型
        return dt
    
    def anews(url):
        newsDetail = {}
        res = requests.get(url)
        res.encoding = 'utf-8'
        soup = BeautifulSoup(res.text,'html.parser')
        newsDetail['newsTitle'] = soup.select('.show-title')[0].text#题目
        showinfo = soup.select('.show-info')[0].text
        newsDetail['newsDT'] = newsdt(showinfo)#时间
        newsDetail['newsClick'] = click(newsUrl)#点击次数
        return newsDetail
    
    def alist(url):
        res = requests.get(listUrl)
        res.encoding = 'utf-8'
        soup = BeautifulSoup(res.text, 'html.parser')
        newsList = []
        for news in soup.select('li'):#获取li元素
            if len(news.select('.news-list-title'))>0:#如果存在新闻题目
                newsUrl = news.select('a')[0]['href']#获取新闻的链接
                newsDesc = news.select('.news-list-description')[0].text#获取摘要文本
                newsDict = anews(newsUrl)#通过链接获取题目时间点击数
                newsDict['description'] = newsDesc
                newsList.append(newsDict)#把每个新闻的信息放进字典扩展到列表里
        return newsList
    listUrl = 'http://news.gzcc.cn/html/xiaoyuanxinwen/'
    print(alist(listUrl))

alist

结果如下：

![1480585-20190410004602995-1640085002.png][]

**3.生成所页列表页的url并获取全部新闻 ：列表extend(列表) allnews**

\*每个同学爬学号尾数开始的10个列表页

具体代码如下：

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

def alist(url):
        res = requests.get(listUrl)
        res.encoding = 'utf-8'
        soup = BeautifulSoup(res.text, 'html.parser')
        newsList = []
        for news in soup.select('li'):#获取li元素
            if len(news.select('.news-list-title'))>0:#如果存在新闻题目
                newsUrl = news.select('a')[0]['href']#获取新闻的链接
                newsDesc = news.select('.news-list-description')[0].text#获取摘要文本
                newsDict = anews(newsUrl)#通过链接获取题目时间点击数
                newsDict['description'] = newsDesc
                newsList.append(newsDict)#把每个新闻的信息放进字典扩展到列表里
        return newsList
    listUrl = 'http://news.gzcc.cn/html/xiaoyuanxinwen/'
    print(alist(listUrl))
    
    allnews = []
    for i in range(105,115):
        listUrl = 'http://news.gzcc.cn/html/xiaoyuanxinwen/{}.html'.format(i)
        allnews.extend(alist(listUrl))
    print(len(allnews))

allnews

结果如下：

![1480585-20190410005900838-1765713754.png][]

**4.设置合理的爬取间隔**

import time

import random

time.sleep(random.random()\*3)

具体代码如下：

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

import time
    import random
    for i in range(5):
        print(i)
        time.sleep(random.random()*3)#沉睡随机数的3倍秒数
    print(allnews)

爬取间隔

结果如下：

![1480585-20190410010300496-283029578.png][]

**5.用pandas做简单的数据处理并保存**

保存到csv或excel文件

newsdf.to\_csv(r'F:\\duym\\爬虫\\gzccnews.csv')

保存到数据库

import sqlite3  
with sqlite3.connect('gzccnewsdb.sqlite') as db:  
    newsdf.to\_sql('gzccnewsdb',db)

具体代码如下：

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

import pandas as pd
    s2 = pd.Series(anews(newsUrl))#一维数组对象
    print(s2)
    newsdf = pd.DataFrame(allnews)#表格型的数据结构
    print(newsdf)
    print(newsdf.sort_values(by=['newsDT'],ascending=False))#按更新时间降序排列
    print(newsdf.sort_index(by=['newsClick'],ascending=False))#按点击量降序排列
    
    newsdf.to_csv(r'gzccnews.csv')
    import sqlite3
    with sqlite3.connect('gzccnewsdb.sqlite') as db:
        newsdf.to_sql('gzccnewsdb',db)
    with sqlite3.connect('gzccnewsdb.sqlite') as db:
        df2 = pandas.read_sql_query('SELECT * FROM gzccnewsdb',con=db)
    print(df2[df2['newsClick']>385])

csv和数据库

结果如下：

csv

![1480585-20190410012734491-1096908483.png][]

数据库

![1480585-20190410025652582-1421964174.png][]

转载于:https://www.cnblogs.com/kunnkkk/p/10680775.html

[https_edu.cnblogs.com_campus_gzcc_GZCC-16SE2_homework_2941]: https://edu.cnblogs.com/campus/gzcc/GZCC-16SE2/homework/2941
[ContractedBlock.gif]: https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif
[ExpandedBlockStart.gif]: /images/20211221/19b562f188e946ac836f06a87c55a3f4.png
[1480585-20190410002346103-140253195.png]: /images/20211221/e0e2e526e91a40b98a9f2f9f4b2c37e5.png
[1480585-20190410004602995-1640085002.png]: /images/20211221/fcfe57fa820e495b8d034b1887b2045c.png
[1480585-20190410005900838-1765713754.png]: /images/20211221/bc718a222a7f48fa8b3dbe5f393f117d.png
[1480585-20190410010300496-283029578.png]: /images/20211221/a212a73fc2ce4b3784f63cb468c7f049.png
[1480585-20190410012734491-1096908483.png]: /images/20211221/8e1d66c1fd314aec82b645cb0a0feea5.png
[1480585-20190410025652582-1421964174.png]: /images/20211221/536f2a350441499c87dbc125ab7b0e7f.png