8天Python从入门到精通第十五章 PySpark案例实战 15.4 数据计算

本是古典何须时尚 2024-03-27 15:02 53阅读 0赞

#### 8天Python从入门到精通 ####

[【黑马程序员python教程，8天python从入门到精通，学python看这套就够了】][python_8_python_python]

#### 文章目录 ####

*   *   *  8天Python从入门到精通
         *  第十五章 PySpark案例实战
         *   *  15.4 数据计算
             *   *  15.4.1 map方法
                 *  15.4.2 小结
                 *  15.4.3 flatMap方法
                 *  15.4.4 小结
                 *  15.4.5 reduceByKey方法
                 *  15.4.6 小结
                 *  15.4.7 练习案例1 - WordCount
                 *  15.4.8 filter方法
                 *  15.4.9 小结
                 *  15.4.10 distinct方法
                 *  15.4.11 小结
                 *  15.4.12 sortBy算子
                 *  15.4.13 小结
                 *  15.4.14 练习案例2

#### 第十五章 PySpark案例实战 ####

##### 15.4 数据计算 #####

###### 15.4.1 map方法 ######

PySpark的数据计算，都是基于RDD对象来进行的，那么如何进行呢？

自然是依赖，RDD对象内置丰富的：成员方法（算子）

map算子：是将RDD的数据一条条处理(处理的逻辑基于map算子中接收的处理函数),返回新的RDD

语法：

![在这里插入图片描述][8f015facf6d24beaafb9cf108b96bd57.png_pic_center]

演示：

![在这里插入图片描述][15cd1f4097314925b5fe2ccdee576507.png_pic_center]

![在这里插入图片描述][81bd868f00ba4423893c5b4199614b59.png_pic_center]

运行结果

![在这里插入图片描述][fa680ee39e7542e2b66089444c345f3b.png_pic_center]

###### 15.4.2 小结 ######

1.  map算子（成员方法）

接受一个处理函数，可用lambda表达式快速编写

对RDD内的元素逐个处理，并返回一个新的RDD

1.  链式调用

对于返回值是新RDD的算子，可以通过链式调用的方式多次调用算子。

###### 15.4.3 flatMap方法 ######

flatMap算子

功能：对rdd执行map操作，然后进行解除嵌套操作.

![在这里插入图片描述][be0a74c548e9422faf65e3a9603c3802.png_pic_center]

演示

![在这里插入图片描述][010c721f850f47ca89be819c1668d9e2.png_pic_center]

from pyspark import SparkConf, SparkContext
    import os
    
    os.environ['PYSPARK_PYTHON'] = "D:/Develop/Python/python3.10.9/python.exe"
    
    conf = SparkConf().setMaster("local[*]").setAppName("test_spark")
    sc = SparkContext(conf=conf)
    
    rdd = sc.parallelize(["itheima itcast 666", "itheima itheima itheima", "python itheima"])
    
    print(rdd.flatMap(lambda x: x.split(" ")).collect())
    
    sc.stop()

运行结果

![在这里插入图片描述][e90daf2d30224e6c9def3a50268ab81c.png_pic_center]

###### 15.4.4 小结 ######

1.  flatMap算子

*  计算逻辑和map一样
 *  可以比map多出，解除一层嵌套的功能

###### 15.4.5 reduceByKey方法 ######

reduceByKey算子

功能：针对KV型RDD,自动按照key分组，然后根据你提供的聚合逻辑，完成组内数据(value)的聚合操作

![在这里插入图片描述][9777f9725ca24d909e79ab7a810beec7.png_pic_center]

![在这里插入图片描述][bdab68c9e22146ec9afa21f6c7ce1319.png_pic_center]

![在这里插入图片描述][25e26a93a04d46e591199a3ec8186ae0.png_pic_center]

演示

![在这里插入图片描述][5b9cdc1fa52f4d1388cd74c2f5d9ef4f.png_pic_center]

from pyspark import SparkConf, SparkContext
    import os
    
    os.environ['PYSPARK_PYTHON'] = "D:/Develop/Python/python3.10.9/python.exe"
    
    conf = SparkConf().setMaster("local[*]").setAppName("test_spark")
    sc = SparkContext(conf=conf)
    
    rdd = sc.parallelize([('男', 99), ('男', 88), ('女', 99), ('女', 66)])
    print(rdd.reduceByKey(lambda a, b: a + b).collect())
    
    sc.stop()

运行结果

![在这里插入图片描述][4b9cb7e8752540f2881a2fd4afae7a1c.png_pic_center]

妙啊

###### 15.4.6 小结 ######

1.  reduceByKey算子

接受一个处理函数，对数据进行两两计算

![在这里插入图片描述][312eec6591b44db6b717e3714af31e9b.png_pic_center]

###### 15.4.7 练习案例1 - WordCount ######

使用学习到的内容，完成：

*  读取文件
 *  统计文件内，单词的出现数量

![在这里插入图片描述][d570dc67da74470998344425fe62cf85.png_pic_center]

itheima itheima itcast itheima
    spark python spark python itheima
    itheima itcast itcast itheima python
    python python spark pyspark pyspark
    itheima python pyspark itcast spark

![在这里插入图片描述][5c32dba1bdf349ea8de1ae1d1f5ba923.png_pic_center]

from pyspark import SparkConf, SparkContext
    import os
    
    os.environ['PYSPARK_PYTHON'] = "D:/Develop/Python/python3.10.9/python.exe"
    
    conf = SparkConf().setMaster("local[*]").setAppName("test_spark")
    sc = SparkContext(conf=conf)
    
    rdd = sc.textFile("hello.txt")
    word_rdd = rdd.flatMap(lambda x: x.split(" "))
    word_with_one_rdd = word_rdd.map(lambda word: (word, 1))
    result_rdd = word_with_one_rdd.reduceByKey(lambda a, b: a + b)
    print(result_rdd.collect())
    sc.stop()

运行结果

![在这里插入图片描述][76ffbd909f4c437c8e44d52ab58e4318.png_pic_center]

妙啊

###### 15.4.8 filter方法 ######

![在这里插入图片描述][4c8b145913b4413eab77d30bc2814292.png_pic_center]

![在这里插入图片描述][c355d7d603404c80858550bb14ba7ffe.png_pic_center]

运行结果

![在这里插入图片描述][a2fd5c3dbc424d8aab1a0d1ad0009b91.png_pic_center]

###### 15.4.9 小结 ######

1.  filter算子

*  接受一个处理函数，可用lambda快速编写
 *  函数对RDD数据逐个处理，得到True的保留至返回值的RDD中

###### 15.4.10 distinct方法 ######

![在这里插入图片描述][d8d4ee3aab2b4f8ab965b612dae22bbe.png_pic_center]

语法：

rdd.distinct() 无需传参

![在这里插入图片描述][492ea013ba824ff7954e6b7d43db0f96.png_pic_center]

运行结果

![在这里插入图片描述][b62d6f63eedd47a9b793e288eb1af0a2.png_pic_center]

###### 15.4.11 小结 ######

1.  distinct算子

完成对RDD内数据的去重操作

###### 15.4.12 sortBy算子 ######

功能：对RDD数据进行排序，基于你指定的排序依据。

![在这里插入图片描述][2408f4d5a49a498fbe3809926ced6f40.png_pic_center]

from pyspark import SparkConf, SparkContext
    import os
    
    os.environ['PYSPARK_PYTHON'] = "D:/Develop/Python/python3.10.9/python.exe"
    
    conf = SparkConf().setMaster("local[*]").setAppName("test_spark")
    sc = SparkContext(conf=conf)
    
    rdd = sc.textFile("hello.txt")
    word_rdd = rdd.flatMap(lambda x: x.split(" "))
    word_with_one_rdd = word_rdd.map(lambda word: (word, 1))
    
    result_rdd = word_with_one_rdd.reduceByKey(lambda a, b: a + b)
    
    final_rdd = result_rdd.sortBy(lambda x: x[1], ascending=False, numPartitions=1)
    print(final_rdd.collect())
    
    sc.stop()

运行结果

![在这里插入图片描述][1c0252880ccd431197ed514b5c4b19ee.png_pic_center]

###### 15.4.13 小结 ######

1.  sortBy算子

*  接收一个处理函数，可用lambda快速编写
 *  函数表示用来决定排序的依据
 *  可以控制升序或降序
 *  全局排序需要设置分区数为1

###### 15.4.14 练习案例2 ######

{"id":1,"timestamp":"2019-05-08T01:03.00Z","category":"平板电脑","areaName":"北京","money":"1450"}|{"id":2,"timestamp":"2019-05-08T01:01.00Z","category":"手机","areaName":"北京","money":"1450"}|{"id":3,"timestamp":"2019-05-08T01:03.00Z","category":"手机","areaName":"北京","money":"8412"}
    {"id":4,"timestamp":"2019-05-08T05:01.00Z","category":"电脑","areaName":"上海","money":"1513"}|{"id":5,"timestamp":"2019-05-08T01:03.00Z","category":"家电","areaName":"北京","money":"1550"}|{"id":6,"timestamp":"2019-05-08T01:01.00Z","category":"电脑","areaName":"杭州","money":"1550"}
    {"id":7,"timestamp":"2019-05-08T01:03.00Z","category":"电脑","areaName":"北京","money":"5611"}|{"id":8,"timestamp":"2019-05-08T03:01.00Z","category":"家电","areaName":"北京","money":"4410"}|{"id":9,"timestamp":"2019-05-08T01:03.00Z","category":"家具","areaName":"郑州","money":"1120"}
    {"id":10,"timestamp":"2019-05-08T01:01.00Z","category":"家具","areaName":"北京","money":"6661"}|{"id":11,"timestamp":"2019-05-08T05:03.00Z","category":"家具","areaName":"杭州","money":"1230"}|{"id":12,"timestamp":"2019-05-08T01:01.00Z","category":"书籍","areaName":"北京","money":"5550"}
    {"id":13,"timestamp":"2019-05-08T01:03.00Z","category":"书籍","areaName":"北京","money":"5550"}|{"id":14,"timestamp":"2019-05-08T01:01.00Z","category":"电脑","areaName":"北京","money":"1261"}|{"id":15,"timestamp":"2019-05-08T03:03.00Z","category":"电脑","areaName":"杭州","money":"6660"}
    {"id":16,"timestamp":"2019-05-08T01:01.00Z","category":"电脑","areaName":"天津","money":"6660"}|{"id":17,"timestamp":"2019-05-08T01:03.00Z","category":"书籍","areaName":"北京","money":"9000"}|{"id":18,"timestamp":"2019-05-08T05:01.00Z","category":"书籍","areaName":"北京","money":"1230"}
    {"id":19,"timestamp":"2019-05-08T01:03.00Z","category":"电脑","areaName":"杭州","money":"5551"}|{"id":20,"timestamp":"2019-05-08T01:01.00Z","category":"电脑","areaName":"北京","money":"2450"}
    {"id":21,"timestamp":"2019-05-08T01:03.00Z","category":"食品","areaName":"北京","money":"5520"}|{"id":22,"timestamp":"2019-05-08T01:01.00Z","category":"食品","areaName":"北京","money":"6650"}
    {"id":23,"timestamp":"2019-05-08T01:03.00Z","category":"服饰","areaName":"杭州","money":"1240"}|{"id":24,"timestamp":"2019-05-08T01:01.00Z","category":"食品","areaName":"天津","money":"5600"}
    {"id":25,"timestamp":"2019-05-08T01:03.00Z","category":"食品","areaName":"北京","money":"7801"}|{"id":26,"timestamp":"2019-05-08T01:01.00Z","category":"服饰","areaName":"北京","money":"9000"}
    {"id":27,"timestamp":"2019-05-08T01:03.00Z","category":"服饰","areaName":"杭州","money":"5600"}|{"id":28,"timestamp":"2019-05-08T01:01.00Z","category":"食品","areaName":"北京","money":"8000"}|{"id":29,"timestamp":"2019-05-08T02:03.00Z","category":"服饰","areaName":"杭州","money":"7000"}

需求，复制以上内容到文件中，使用Spark读取文件进行计算：

*  各个城市销售额排名，从大到小
 *  全部城市，有哪些商品类别在售卖
 *  北京市有哪些商品类别在售卖

![在这里插入图片描述][bce635c5f38c46d9a7b505940698d39b.png_pic_center]

from pyspark import SparkConf, SparkContext
    import os
    import json
    
    os.environ['PYSPARK_PYTHON'] = "D:/Develop/Python/python3.10.9/python.exe"
    
    conf = SparkConf().setMaster("local[*]").setAppName("test_spark")
    sc = SparkContext(conf=conf)
    
    file_rdd = sc.textFile("orders.txt")
    json_str_rdd = file_rdd.flatMap(lambda x: x.split("|"))
    dict_rdd = json_str_rdd.map(lambda x: json.loads(x))
    
    result1_rdd = dict_rdd.map(lambda x: (x['areaName'], int(x['money']))).reduceByKey(lambda a, b: a + b).sortBy(
        lambda x: x[1],
        ascending=False,
        numPartitions=1)
    
    print("各个城市销售额排名, 从大到小", result1_rdd.collect())
    
    result2_rdd = dict_rdd.map(lambda x: x['category']).distinct()
    print("全部城市有哪些商品类别在售卖", result2_rdd.collect())
    
    result3_rdd = dict_rdd.filter(lambda x: x['areaName'] == "北京").map(lambda x: x['category']).distinct()
    print("北京市有这些商品类别在售卖: ", result3_rdd.collect())
    sc.stop()

运行结果

![在这里插入图片描述][e6420461069948f3bc106093e905d236.png_pic_center]

[python_8_python_python]: https://www.bilibili.com/video/BV1qW4y1a7fU
[8f015facf6d24beaafb9cf108b96bd57.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/29dcbf1aa7064891b7c1cbf154043a93.png
[15cd1f4097314925b5fe2ccdee576507.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/217183ec65114eed80d0241cb2017ecc.png
[81bd868f00ba4423893c5b4199614b59.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/f45645d92db84594b13aec3814d7da66.png
[fa680ee39e7542e2b66089444c345f3b.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/76ee1a413578414186b404845b63dd35.png
[be0a74c548e9422faf65e3a9603c3802.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/fee2c295954a4b82889c6af3bbc5d606.png
[010c721f850f47ca89be819c1668d9e2.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/a899ae872c3d41469c6f7371250badbd.png
[e90daf2d30224e6c9def3a50268ab81c.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/0b29326be18846f0b066d9db297dbe6a.png
[9777f9725ca24d909e79ab7a810beec7.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/8afbc0362ccc41c7b795f422b16f342e.png
[bdab68c9e22146ec9afa21f6c7ce1319.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/72e75b4c6b43488699fecb9b2c7259de.png
[25e26a93a04d46e591199a3ec8186ae0.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/1979fce689874f14bc20a257e4709ae1.png
[5b9cdc1fa52f4d1388cd74c2f5d9ef4f.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/1abf0ab8edb248628127c7eefa336a1f.png
[4b9cb7e8752540f2881a2fd4afae7a1c.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/c958f757f313481a98b33928bce4eb9a.png
[312eec6591b44db6b717e3714af31e9b.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/b34bea023b084a64b70c525a61ad37aa.png
[d570dc67da74470998344425fe62cf85.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/d520027dfda44746a67e2b02f2d5e978.png
[5c32dba1bdf349ea8de1ae1d1f5ba923.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/1a6d0aea99154771afe764169cd710fb.png
[76ffbd909f4c437c8e44d52ab58e4318.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/6e72bb3c95f24dcc8b7849d23b9ca9c0.png
[4c8b145913b4413eab77d30bc2814292.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/6f49326a0cbe4d1ab89ebc89a59375ac.png
[c355d7d603404c80858550bb14ba7ffe.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/7420564b06b34d119e3d6246dc987066.png
[a2fd5c3dbc424d8aab1a0d1ad0009b91.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/4d4e6bdf01534b5aa0e329a549b53db4.png
[d8d4ee3aab2b4f8ab965b612dae22bbe.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/1c6f20803cd9474cb9acc572631e3f3b.png
[492ea013ba824ff7954e6b7d43db0f96.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/386f5088653243c48cb5a0e817fa0887.png
[b62d6f63eedd47a9b793e288eb1af0a2.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/434c28680eaf492d9354a25b9462734d.png
[2408f4d5a49a498fbe3809926ced6f40.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/1fd69dfab48f459caa2c38ea3b180fe4.png
[1c0252880ccd431197ed514b5c4b19ee.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/5e3018e6abf04c5b8b03bbd8c72ae9e6.png
[bce635c5f38c46d9a7b505940698d39b.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/f3876b9c7f5643bfb2d4871db8c3556a.png
[e6420461069948f3bc106093e905d236.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/c067caa33ced4ee4b92dace1bf444b9e.png