Spark学习（3）-Spark SQL（1）

Dear 丶 2022-03-14 08:44 199阅读 0赞

**目录：**

1.  Spark SQL发展历程
2.  Spark SQL性能
3.  DateFrame介绍
4.  Spark RDD SQL 简单使用

## ##

### 一.Spark SQL发展历程 ###

**1.hive 到shark 再到spark sql 的演变**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 3][]

![20190304120650808.png][]

**2.hive on spark（hive集成spark，计算效率提高，而不是仅仅依赖mapreduce）**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 4][]

![20190304121041216.png][]

**3.spark sql实现**

### ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 5][] ###

![20190304122655141.png][]

### 二.Spark SQL性能 ###

**shark 、shark （disk）、 spark sql 三者性能比较**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 7][]

## ##

### 三.DateFrame介绍 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 8][]

![20190304122625463.png][]  
![20190304173055271.png][]

[Spark中DataSet的基本使用][Spark_DataSet]

## ##

### 四.Spark RDD SQL简单使用 ###

**1.使用DataFrame**

1.  引入spark-sql\_2.10依赖
2.  构造SqlContext
    
        //1.创建一个SparkConf对象
            val conf=new SparkConf().setAppName("HelloSpark").setMaster("local[2]")
            //2.创建一个SparkConf对象
            val sc=new SparkContext(conf)
            //3.创建一个SQLContext对象
            val sqlContext=new SQLContext(sc)
3.  导入支持RDD到DataFrame的隐式转换
    
        import sqlContext.implicits._
4.  构造DataFrame
    
        val df:DataFrame=sqlContext.read.json("e://user.json")

**2.DataFrame简单操作**

*  显示df:DataFrame的内容：df.show()
 *  显示df:DataFrame中scheme的内容：df.printSchema()
 *  显示某一列信息：df.select("user").show()
 *  对列的值进行操作：df.select(df("user").show()，df("age")\*10).show()
 *  对值进行条件过滤：df.filter(df("age")>3).show()
 *  分组之后进行过数据统计：df.gorupBy("name").count().show()

**3.示例：****使用步骤**

1.  创建项目
2.  引入Spark 版本对应的Spark SQL 依赖（注意是在在Spark Core的基础之上）  
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 9][]  
    ![20190304123131143.png][]  
    ![20190304123207623.png][]
3.  在e盘创建一个user.json文件
    
        {"user":"aaa1","age":11}
        {"user":"aaa2","age":12}
        {"user":"aaa3","age":13}
        {"user":"aaa4","age":14}
        {"user":"aaa5","age":15}
        {"user":"aaa6","age":16}
        {"user":"aaa7","age":17}
        {"user":"aaa8","age":18}
        {"user":"aaa9","age":19}
    
     
4.  编写代码
    
        package test
        import org.apache.spark.SparkConf
        import org.apache.spark.SparkContext
        import org.apache.spark.SparkContext._
        import org.apache.spark.rdd.RDD
        import org.apache.spark.sql.SQLContext
        import org.apache.spark.sql.DataFrame
        object SparkFirst {
          def main(args: Array[String]): Unit = {
            //1.创建一个SparkConf对象
            val conf=new SparkConf().setAppName("HelloSpark").setMaster("local[2]")
            //2.创建一个SparkConf对象
            val sc=new SparkContext(conf)
            //3.创建一个SQLContext对象
            val sqlContext=new SQLContext(sc)
            
            //4.引入包，import的效果从开始延伸到语句块的结束
            import sqlContext.implicits._
            /*如果手里有一个RDD，需要通过spark sql的方式进行操作的话，那么就需要用到spark sql的函数。
            但是这些函数，默认情况下RDD是没有的。所以在scala中有一个很特殊的功能——隐式转换
            将本地文件内容转换成一个DataFrame*/
        
            //5.创建一个DataFrame
            val df:DataFrame=sqlContext.read.json("e://user.json")
            //6.使用spark sql函数
            df.show()
            
          }
        }
    
     
5.  运行结果
    
        +---+----+
        |age|user|
        +---+----+
        | 11|aaa1|
        | 12|aaa2|
        | 13|aaa3|
        | 14|aaa4|
        | 15|aaa5|
        | 16|aaa6|
        | 17|aaa7|
        | 18|aaa8|
        | 19|aaa9|
        +---+----+

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70]: /images/20220313/d0658dbb6ae34144bd40368503424c35.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 1]: /images/20220313/1e2251b956da40889ab6a28937d2f5bb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 2]: /images/20220313/fb2b0c4e7d7243809a18d87281fc8574.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 3]: /images/20220313/57de3009994c4c71ad5e0eb986fb7f3c.png
[20190304120650808.png]: /images/20220313/06f389bdcf8f40819a15addb8f5cc552.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 4]: /images/20220313/6d9e59fe94a149309649749215e11b83.png
[20190304121041216.png]: /images/20220313/319f3fb11f844e3fbaa2a211cc3bedaa.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 5]: /images/20220313/1ac957d926664940ad003d21e8edda81.png
[20190304122655141.png]: /images/20220313/160f3865f5d44e9ab09718772c1bb66e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 6]: /images/20220313/dd3402a818f349fba9a6c51655701767.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 7]: /images/20220313/426dcd6bafe347a8861067776cda5a40.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 8]: /images/20220313/6bf9e81d7824474bbae04bf84107f80f.png
[20190304122625463.png]: /images/20220313/ff895b15d9164aa384dce2103e905290.png
[20190304173055271.png]: /images/20220313/554988fa5a084fe3b8af26d96aa88905.png
[Spark_DataSet]: https://blog.csdn.net/weixin_42702831/article/details/82492421
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 9]: /images/20220313/b5fdbfbca975430ba4d5c262cff7919a.png
[20190304123131143.png]: /images/20220313/fd7f461ec2484cfaa80bde4dd385da6a.png
[20190304123207623.png]: /images/20220313/6c6940014a3543468be003f1b1563653.png