手把手教你爬取某果TV弹幕数据，并进行数据可视化分析~

亦凉 2022-09-11 06:25 445阅读 0赞

点击上方“**Python爬虫与数据挖掘**”，进行关注

回复“**书籍**”即可获赠Python从入门到进阶共10本电子书

今

日

鸡

汤

络纬秋啼金井阑，微霜凄凄簟色寒。

本期是对芒果TV视频评论的一次爬虫与数据分析。

![1eebb3e570fbadbecbd3910b722d07ad.png][]

爬虫方面：由于芒果的评论数据是封装在json里面，所以只需要找到json文件，对需要的数据进行提取保存即可。![3389f2c04c4693dd5978a3d4e37f3049.png][]

*  视频网址：https://www.mgtv.com/b/44793/11017269.html?fpa=se&lastp=so\_result
 *  评论json数据网址：https://comment.mgtv.com/v4/comment/getCommentList?page=1&subjectType=hunantv2014&subjectId=11017269
 *  注：只要替换subjectId的值，即可爬取其他视频的评论

![e5cd4b28c6056021c7fa7cc8f9f5603a.png][] 在这里插入图片描述

数据分析方面：涉及到了词云图，条形，折线，饼图，后三者是对评论时间的分析，然而芒果TV的评论时间是以时间戳的形式显示，所以要进行转换，再去统计出现次数。

**项目结构：**![c3209fbc41799b9efe73194c08175106.png][]一.  爬虫部分：1.爬虫代码：spiders.py

# coding=gbk
    import csv
    import os
    import sys
    import time
    
    import rdata as rdata
    import requests
    import json
    import pandas as pd
    
    
    # 封装数据的网站
    from python_helper.api.src.service.LogHelper import setting
    
    headers = {
            'cookie': 'cna=J7K2Fok5AXECARu7QWn6+cxu; isg=BCcnDiP-NfKV5bF-OctWuXuatl3xrPuOyBVJJfmQLrZn6ESqAX0y3jrhCuj2ANMG; l=eBSmWoPRQeT6Zn3iBO5whurza77O1CAf1sPzaNbMiIncC6BR1AvOCJxQLtyCvptRR8XcGLLB4nU7C5eTae7_7CDmndLHuI50MbkyCef..',
            'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
        }
    for i in range(1,99):
        #url = f'https://search.damai.cn/searchajax.html?keyword=&cty=&ctl=%E6%BC%94%E5%94%B1%E4%BC%9A&sctl=&tsg=0&st=&et=&order=1&pageSize=30&currPage={i}&tn='
        url = f'https://comment.mgtv.com/v4/comment/getCommentList?page={i}&subjectType=hunantv2014&subjectId=11017269'
        print(url)
    
        response = requests.get(url, headers=headers)
        json_text = json.loads(response.text)
    
        # print(json_text.keys())
        for t in range(1, 14):
            rdata1 = json_text['data']['list'][t]['content']
            rdata2 = int(json_text['data']['list'][t]['createTime'])
    
            # 转换为其他日期格式,如:"%Y-%m-%d %H:%M:%S"
            timeArray = time.localtime(rdata2)
            rdata2 = time.strftime("%Y-%m-%d %H:%M:%S", timeArray)
            #print(rdata2)
    
            print(rdata1,rdata2)
    
            f = open('百变大咖秀.txt', 'a+',encoding="utf-8")
            print(rdata1,file = f)
    
            f = open('时间.txt', 'a+', encoding="utf-8")
            print(rdata2, file=f)
    
    
            f.close()

2.将评论时间的txt文件读入csv文件 CD.py

# coding=gbk
    import csv
    csvFile = open("data.csv",'w',newline='',encoding='utf-8')
    writer = csv.writer(csvFile)
    csvRow = []
    
    f = open("时间.txt",'r',encoding='GB2312')
    for line in f:
        csvRow = line.split()
        writer.writerow(csvRow)
    
    f.close()
    csvFile.close()

![92c01a9cddfd9a0248d531b95caa5d75.png][]二. 数据分析 1.制作词云图 wc.py

import numpy as np
    import jieba
    from wordcloud import WordCloud
    from matplotlib import pyplot as plt
    from PIL import Image
    
    # 上面的包自己安装，不会的就百度
    
    f = open('../Spiders/百变大咖秀.txt', 'r', encoding='utf-8')  # 这是数据源，也就是想生成词云的数据
    txt = f.read()  # 读取文件
    f.close()  # 关闭文件，其实用with就好，但是懒得改了
    # 如果是文章的话，需要用到jieba分词，分完之后也可以自己处理下再生成词云
    words = jieba.lcut(txt)
    newtxt = ' '.join(words)
    img = Image.open(r'wc.jpg')  # 想要搞得形状
    img_array = np.array(img)
    
    # 相关配置，里面这个collocations配置可以避免重复
    wordcloud = WordCloud(
        background_color="white",
        width=1080,
        height=960,
        font_path="../文悦新青年.otf",
        max_words=150,
        scale=7,#清晰度
        max_font_size=100,
        mask=img_array,
        collocations=False).generate(txt)
    
    plt.imshow(wordcloud)
    plt.axis('off')
    plt.show()
    wordcloud.to_file('../Photo/result.png')

轮廓图：wc.jpg![1aef5b38eab4fc18e0fe9c35f947cf39.png][]词云图：result.png （注：停用词自己加，这里没有放）![06183fb88fe457b90e83250201ef6bb7.png][]2.可视化分析

（1）时间数据处理

py.py   (统计一天各个时间段内的评论数）

# coding=gbk
    import csv
    
    from pyecharts import options as opts
    from sympy.combinatorics import Subset
    from wordcloud import WordCloud
    
    with open('../Spiders/data.csv') as csvfile:
        reader = csv.reader(csvfile)
    
        data1 = [str(row[1])[0:2] for row in reader]
    
        print(data1)
    print(type(data1))
    
    
    #先变成集合得到seq中的所有元素，避免重复遍历
    set_seq = set(data1)
    rst = []
    for item in set_seq:
        rst.append((item,data1.count(item)))  #添加元素及出现个数
    rst.sort()
    print(type(rst))
    print(rst)
    
    with open("time2.csv", "w+", newline='', encoding='utf-8') as f:
        writer = csv.writer(f, delimiter=',')
        for i in rst:                # 对于每一行的，将这一行的每个元素分别写在对应的列中
            writer.writerow(i)
    
    with open('time2.csv') as csvfile:
         reader = csv.reader(csvfile)
         x = [str(row[0]) for row in reader]
         print(x)
    with open('time2.csv') as csvfile:
        reader = csv.reader(csvfile)
        y1 = [float(row[1]) for row in reader]
        print(y1)

处理结果（评论时间，评论数）![9ecbc942840066ca20e52f817f0bb7ff.png][]py1.py （统计最近评论数）

# coding=gbk
    import csv
    
    from pyecharts import options as opts
    from sympy.combinatorics import Subset
    from wordcloud import WordCloud
    
    with open('../Spiders/data.csv') as csvfile:
        reader = csv.reader(csvfile)
    
        data1 = [str(row[0]) for row in reader]
        #print(data1)
    print(type(data1))
    
    
    #先变成集合得到seq中的所有元素，避免重复遍历
    set_seq = set(data1)
    rst = []
    for item in set_seq:
        rst.append((item,data1.count(item)))  #添加元素及出现个数
    rst.sort()
    print(type(rst))
    print(rst)
    
    
    
    with open("time1.csv", "w+", newline='', encoding='utf-8') as f:
        writer = csv.writer(f, delimiter=',')
        for i in rst:                # 对于每一行的，将这一行的每个元素分别写在对应的列中
            writer.writerow(i)
    
    with open('time1.csv') as csvfile:
         reader = csv.reader(csvfile)
         x = [str(row[0]) for row in reader]
         print(x)
    with open('time1.csv') as csvfile:
        reader = csv.reader(csvfile)
        y1 = [float(row[1]) for row in reader]
    
        print(y1)

处理结果（评论时间，评论数）![a986b09eec0311dde20c6e9c8775fcee.png][]（2）制作最近评论数条形图与折线图 DrawBar.py

# encoding: utf-8
    import csv
    import pyecharts.options as opts
    from pyecharts.charts import Bar
    from pyecharts.globals import ThemeType
    
    
    class DrawBar(object):
    
        """绘制柱形图类"""
        def __init__(self):
            """创建柱状图实例，并设置宽高和风格"""
            self.bar = Bar(init_opts=opts.InitOpts(width='1500px', height='700px', theme=ThemeType.LIGHT))
    
        def add_x(self):
            """为图形添加X轴数据"""
            with open('time1.csv') as csvfile:
                reader = csv.reader(csvfile)
                x = [str(row[0]) for row in reader]
                print(x)
    
    
            self.bar.add_xaxis(
                xaxis_data=x,
    
            )
    
        def add_y(self):
            with open('time1.csv') as csvfile:
                reader = csv.reader(csvfile)
                y1 = [float(row[1]) for row in reader]
    
                print(y1)
    
    
    
            """为图形添加Y轴数据，可添加多条"""
            self.bar.add_yaxis(  # 第一个Y轴数据
                series_name="评论数",  # Y轴数据名称
                y_axis=y1,  # Y轴数据
                label_opts=opts.LabelOpts(is_show=False),  # 设置标签
                bar_max_width='70px',  # 设置柱子最大宽度
            )
    
    
        def set_global(self):
            """设置图形的全局属性"""
            #self.bar(width=2000,height=1000)
            self.bar.set_global_opts(
                title_opts=opts.TitleOpts(  # 设置标题
                    title='百变大咖秀近日评论统计',title_textstyle_opts=opts.TextStyleOpts(font_size=35)
    
                ),
                tooltip_opts=opts.TooltipOpts(  # 提示框配置项（鼠标移到图形上时显示的东西）
                    is_show=True,  # 是否显示提示框
                    trigger="axis",  # 触发类型（axis坐标轴触发，鼠标移到时会有一条垂直于X轴的实线跟随鼠标移动，并显示提示信息）
                    axis_pointer_type="cross"  # 指示器类型（cross将会生成两条分别垂直于X轴和Y轴的虚线，不启用trigger才会显示完全）
                ),
                toolbox_opts=opts.ToolboxOpts(),  # 工具箱配置项(什么都不填默认开启所有工具)
    
            )
    
        def draw(self):
            """绘制图形"""
    
            self.add_x()
            self.add_y()
            self.set_global()
            self.bar.render('../Html/DrawBar.html')  # 将图绘制到 test.html 文件内，可在浏览器打开
        def run(self):
            """执行函数"""
            self.draw()
    
    
    
    if __name__ == '__main__':
        app = DrawBar()
    
    app.run()

效果图：DrawBar.html![5c2f986645644e081c2cf3471abfe214.png][]![ddbbaedc7a9d3a6c9cbdcebead5ce5c1.png][]（3）制作每小时评论条形图与折线图 DrawBar2.py

# encoding: utf-8
    import csv
    import pyecharts.options as opts
    from pyecharts.charts import Bar
    from pyecharts.globals import ThemeType
    
    
    class DrawBar(object):
    
        """绘制柱形图类"""
        def __init__(self):
            """创建柱状图实例，并设置宽高和风格"""
            self.bar = Bar(init_opts=opts.InitOpts(width='1500px', height='700px', theme=ThemeType.MACARONS))
    
        def add_x(self):
            """为图形添加X轴数据"""
            str_name1 = '点'
    
            with open('time2.csv') as csvfile:
                reader = csv.reader(csvfile)
                x = [str(row[0] + str_name1) for row in reader]
                print(x)
    
    
            self.bar.add_xaxis(
                xaxis_data=x
            )
    
        def add_y(self):
            with open('time2.csv') as csvfile:
                reader = csv.reader(csvfile)
                y1 = [int(row[1]) for row in reader]
    
                print(y1)
    
    
    
            """为图形添加Y轴数据，可添加多条"""
            self.bar.add_yaxis(  # 第一个Y轴数据
                series_name="评论数",  # Y轴数据名称
                y_axis=y1,  # Y轴数据
                label_opts=opts.LabelOpts(is_show=False),  # 设置标签
                bar_max_width='50px',  # 设置柱子最大宽度
    
            )
    
    
        def set_global(self):
            """设置图形的全局属性"""
            #self.bar(width=2000,height=1000)
            self.bar.set_global_opts(
                title_opts=opts.TitleOpts(  # 设置标题
                    title='百变大咖秀各时间段评论统计',title_textstyle_opts=opts.TextStyleOpts(font_size=35)
    
                ),
                tooltip_opts=opts.TooltipOpts(  # 提示框配置项（鼠标移到图形上时显示的东西）
                    is_show=True,  # 是否显示提示框
                    trigger="axis",  # 触发类型（axis坐标轴触发，鼠标移到时会有一条垂直于X轴的实线跟随鼠标移动，并显示提示信息）
                    axis_pointer_type="cross"  # 指示器类型（cross将会生成两条分别垂直于X轴和Y轴的虚线，不启用trigger才会显示完全）
                ),
                toolbox_opts=opts.ToolboxOpts(),  # 工具箱配置项(什么都不填默认开启所有工具)
    
            )
    
        def draw(self):
            """绘制图形"""
    
            self.add_x()
            self.add_y()
            self.set_global()
            self.bar.render('../Html/DrawBar2.html')  # 将图绘制到 test.html 文件内，可在浏览器打开
        def run(self):
            """执行函数"""
            self.draw()
    
    
    
    if __name__ == '__main__':
        app = DrawBar()
    
    app.run()

效果图：DrawBar2.html![2c06e73ee520a99dbd3d2e03577e9425.png][]![dec8fa94d95ce0857d09b1b9e518559e.png][]（4）制作各类饼图

*  pie\_pyecharts.py

import csv
    
    from pyecharts import options as opts
    from pyecharts.charts import Pie
    from random import randint
    
    from pyecharts.globals import ThemeType
    
    with open('time1.csv') as csvfile:
        reader = csv.reader(csvfile)
        x = [str(row[0]) for row in reader]
        print(x)
    with open('time1.csv') as csvfile:
        reader = csv.reader(csvfile)
        y1 = [float(row[1]) for row in reader]
    
        print(y1)
    
    
    
    num = y1
    lab = x
    (
        Pie(init_opts=opts.InitOpts(width='1500px',height='500px',theme=ThemeType.LIGHT))#默认900，600
        .set_global_opts(
            title_opts=opts.TitleOpts(title="百变大咖秀近日评论统计",
                                                   title_textstyle_opts=opts.TextStyleOpts(font_size=27)),legend_opts=opts.LegendOpts(
    
                pos_top="8%",# 图例位置调整
                ),)
        .add(series_name='',center=[400, 300], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图
       #.add(series_name='',center=[750, 300],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图
        .add(series_name='', center=[1100, 300],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
    ).render('../Html/pie_pyecharts.html')

效果图![bed3f032f469b6010a811427ba051dec.png][]

*  pie\_pyecharts2.py

import csv
    
    from pyecharts import options as opts
    from pyecharts.charts import Pie
    from random import randint
    
    from pyecharts.globals import ThemeType
    
    str_name1 = '点'
    
    with open('time2.csv') as csvfile:
        reader = csv.reader(csvfile)
        x = [str(row[0]+str_name1) for row in reader]
        print(x)
    with open('time2.csv') as csvfile:
        reader = csv.reader(csvfile)
        y1 = [int(row[1]) for row in reader]
    
        print(y1)
    
    
    
    num = y1
    lab = x
    (
        Pie(init_opts=opts.InitOpts(width='1520px',height='520px',theme=ThemeType.LIGHT,))#默认900，600
         .set_global_opts(
            title_opts=opts.TitleOpts(title="百变大咖秀每小时评论统计"
                                      ,title_textstyle_opts=opts.TextStyleOpts(font_size=27)),
            legend_opts=opts.LegendOpts(
    
                pos_top="8%",# 图例位置调整
                ),
        )
        .add(series_name='',center=[250, 320], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图
        .add(series_name='',center=[790, 320],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图
        .add(series_name='', center=[1262, 320],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
    ).render('../Html/pie_pyecharts2.html')

效果图![0369fdb453c64fb65257c50ead09692c.png][]

*  pie\_pyecharts3.py 观看时间区间评论统计

# coding=gbk
    import csv
    
    from pyecharts import options as opts
    from pyecharts.globals import ThemeType
    from sympy.combinatorics import Subset
    from wordcloud import WordCloud
    
    with open('../Spiders/data.csv') as csvfile:
        reader = csv.reader(csvfile)
    
        data2 = [int(row[1].strip('')[0:2]) for row in reader]
    
    
        #print(data2)
    print(type(data2))
    
    #先变成集合得到seq中的所有元素，避免重复遍历
    set_seq = set(data2)
    list = []
    for item in set_seq:
        list.append((item,data2.count(item)))  #添加元素及出现个数
    list.sort()
    print(type(list))
    #print(list)
    
    
    with open("time2.csv", "w+", newline='', encoding='utf-8') as f:
        writer = csv.writer(f, delimiter=',')
        for i in list:                # 对于每一行的，将这一行的每个元素分别写在对应的列中
            writer.writerow(i)
    
    
    n = 4 #分成n组
    m = int(len(list)/n)
    list2 = []
    for i in range(0, len(list), m):
        list2.append(list[i:i+m])
    
    print("凌晨 : ",list2[0])
    print("上午 : ",list2[1])
    print("下午 : ",list2[2])
    print("晚上 : ",list2[3])
    
    with open('time2.csv') as csvfile:
        reader = csv.reader(csvfile)
        y1 = [int(row[1]) for row in reader]
    
        print(y1)
    
    n =6
    groups = [y1[i:i + n] for i in range(0, len(y1), n)]
    
    print(groups)
    
    x=['凌晨','上午','下午','晚上']
    y1=[]
    for y1 in groups:
        num_sum = 0
        for groups in y1:
            num_sum += groups
    
    print(x)
    print(y1)
    
    
    import csv
    
    from pyecharts import options as opts
    from pyecharts.charts import Pie
    from random import randint
    
    str_name1 = '点'
    
    num = y1
    lab = x
    (
        Pie(init_opts=opts.InitOpts(width='1500px',height='500px',theme=ThemeType.LIGHT))#默认900，600
            .set_global_opts(
            title_opts=opts.TitleOpts(title="百变大咖秀观看时间区间评论统计"
                                      , title_textstyle_opts=opts.TextStyleOpts(font_size=40)),
            legend_opts=opts.LegendOpts(
    
                pos_top="8%",  # 图例位置调整
            ),
        )
        .add(series_name='',center=[260, 300], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图
       .add(series_name='',center=[1230, 300],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图
        .add(series_name='', center=[750, 300],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
    ).render('../Html/pie_pyecharts3.html')

效果图![f3a5caf73d1a4936e1e878ab53bb6583.png][]

小伙伴们，快快用实践一下吧！如果在学习过程中，有遇到任何问题，欢迎加我好友，我拉你进Python学习交流群共同探讨学习。

![8403e690a13d812c5e41e2c36311382d.png][]

**---\--\---\---\---\--\---** End **---\--\---\--\---\---\--\-**

往期精彩文章推荐：

*  [手把手教你用Pyecharts库对淘宝数据进行可视化展示][Pyecharts]
 *  [手把手教你用Pandas库对淘宝原始数据进行数据处理和分词处理][Pandas]
 *  [盘点Flask与数据库的交互插件--Flask-Sqlalchemy][Flask_--Flask-Sqlalchemy]  
    
 *  [盘点那些年我们一起玩过的网络安全工具][Link 1]

![bf877176645b5a61c7193623d8a1262e.png][]

欢迎大家**点赞，****留言，****转发，转载，**感谢大家的相伴与支持

想加入Python学习群请在后台回复【**入群**】

万水千山总是情，点个【**在看**】行不行

**/今日留言主题/**

随便说一两句吧~~

[1eebb3e570fbadbecbd3910b722d07ad.png]: /images/20220828/b813cd3e561f4af4b3218193425c1dab.png
[3389f2c04c4693dd5978a3d4e37f3049.png]: /images/20220828/2e102c60e8994124b0e778fc238a2ecb.png
[e5cd4b28c6056021c7fa7cc8f9f5603a.png]: /images/20220828/472d014079bf45c5aa76c9c3584e5e1c.png
[c3209fbc41799b9efe73194c08175106.png]: /images/20220828/dc9a70ed634d498cb62c5422caaa24d0.png
[92c01a9cddfd9a0248d531b95caa5d75.png]: /images/20220828/a1c53f4cead7416f9c230f1bbd5d3879.png
[1aef5b38eab4fc18e0fe9c35f947cf39.png]: https://img-blog.csdnimg.cn/img_convert/1aef5b38eab4fc18e0fe9c35f947cf39.png
[06183fb88fe457b90e83250201ef6bb7.png]: /images/20220828/1b135973214b435184425b2d2529bd8d.png
[9ecbc942840066ca20e52f817f0bb7ff.png]: /images/20220828/ecebac1edf284eefbdf9b64e5cb7ce19.png
[a986b09eec0311dde20c6e9c8775fcee.png]: /images/20220828/49f2c975b5f741d5bada54fc5c497fc6.png
[5c2f986645644e081c2cf3471abfe214.png]: https://img-blog.csdnimg.cn/img_convert/5c2f986645644e081c2cf3471abfe214.png
[ddbbaedc7a9d3a6c9cbdcebead5ce5c1.png]: /images/20220828/5ecfdfd7face41c9ab07bfba767a4595.png
[2c06e73ee520a99dbd3d2e03577e9425.png]: https://img-blog.csdnimg.cn/img_convert/2c06e73ee520a99dbd3d2e03577e9425.png
[dec8fa94d95ce0857d09b1b9e518559e.png]: /images/20220828/3c1c60e19f4b48f08767eae2aacd8b01.png
[bed3f032f469b6010a811427ba051dec.png]: /images/20220828/d0ce6807fd9349edbedb09c0c848fec1.png
[0369fdb453c64fb65257c50ead09692c.png]: /images/20220828/afd1678cc4d740f883c03164b931d266.png
[f3a5caf73d1a4936e1e878ab53bb6583.png]: /images/20220828/52e5857482aa4b54a3955fa1918409cd.png
[8403e690a13d812c5e41e2c36311382d.png]: /images/20220828/aabd115737d34954bc42def95189dfac.png
[Pyecharts]: http://mp.weixin.qq.com/s?__biz=MzU3MzQxMjE2NA%3D%3D&chksm=fcc0b33ccbb73a2a1f4d56275c695dbe8e9ad08cbd54bffa1f20c4de6a434378c9e63967390a&idx=1&mid=2247496983&scene=21&sn=b3db9b4cf5f129b660448bb3d7beed8c#wechat_redirect
[Pandas]: http://mp.weixin.qq.com/s?__biz=MzU3MzQxMjE2NA%3D%3D&chksm=fcc0b32ecbb73a3899a12142429ce31e0758595bb6f46d69be7559f84d5814a48dfcf9a0b05d&idx=1&mid=2247496965&scene=21&sn=f34b655ead49dadd8abb24463e2e607f#wechat_redirect
[Flask_--Flask-Sqlalchemy]: http://mp.weixin.qq.com/s?__biz=MzU3MzQxMjE2NA%3D%3D&chksm=fcc0b2bdcbb73bab9267a53ac5bd3a6f4af4cc76c521f4c5ebef7aa1351039776291289f2b48&idx=1&mid=2247496854&scene=21&sn=60c64c22e97ce97756857b7dae96996e#wechat_redirect
[Link 1]: http://mp.weixin.qq.com/s?__biz=MzU3MzQxMjE2NA%3D%3D&chksm=fcc0b4b9cbb73daf05e46ffc74d8f3c5257db082f8592bc1d4dbd9a536491866367fe160c5ef&idx=1&mid=2247496338&scene=21&sn=cc109289ced145f50e1dfff0f67fc5f3#wechat_redirect
[bf877176645b5a61c7193623d8a1262e.png]: /images/20220828/0b39dc502c0d4b9baae43ec9efbfa843.png