Spark RDD五大特性

落日映苍穹つ 2023-07-13 13:44 3阅读 0赞

学习rdd的特性最好是从官网和源码来进行学习，首先看下官网解释：[http://spark.apache.org/docs/latest/rdd-programming-guide.html\#resilient-distributed-datasets-rdds][http_spark.apache.org_docs_latest_rdd-programming-guide.html_resilient-distributed-datasets-rdds]

> # Resilient Distributed Datasets (RDDs) #
> 
>     Spark revolves around the concept of a *resilient distributed dataset* (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel. There are two ways to create RDDs: ***parallelizing* **an existing collection in your driver program, or **referencing a dataset** in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat.
> 
>    Spark围绕***弹性分布式数据集***（RDD）的概念展开，RDD是可并行操作的元素的容错集合。创建RDD的方法有两种：*并行化* 驱动程序中的现有集合，或引用外部存储系统（例如共享文件系统NFS，HDFS，HBase或可以提供Hadoop InputFormat的任何数据源）中的数据集。

看完官网解释，感觉解释的很笼统，RDD具体的具体特性不是很清晰，ok，接下来查看源码

>     /**
>      * A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable,
>      * partitioned collection of elements that can be operated on in parallel. This class contains the
>      * basic operations available on all RDDs, such as `map`, `filter`, and `persist`. In addition,
>      * [[org.apache.spark.rdd.PairRDDFunctions]] contains operations available only on RDDs of key-value
>      * pairs, such as `groupByKey` and `join`;
>      * [[org.apache.spark.rdd.DoubleRDDFunctions]] contains operations available only on RDDs of
>      * Doubles; and
>      * [[org.apache.spark.rdd.SequenceFileRDDFunctions]] contains operations available on RDDs that
>      * can be saved as SequenceFiles.
>      * All operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)])
>      * through implicit.
>      *
>      * Internally, each RDD is characterized by five main properties:
>      *
>      *  - A list of partitions
>      *  - A function for computing each split
>      *  - A list of dependencies on other RDDs
>      *  - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)
>      *  - Optionally, a list of preferred locations to compute each split on (e.g. block locations for
>      *    an HDFS file)
>      *
>      * All of the scheduling and execution in Spark is done based on these methods, allowing each RDD
>      * to implement its own way of computing itself. Indeed, users can implement custom RDDs (e.g. for
>      * reading data from a new storage system) by overriding these functions. Please refer to the
>      * <a href="http://people.csail.mit.edu/matei/papers/2012/nsdi_spark.pdf">Spark paper</a>
>      * for more details on RDD internals.
>      */
>     abstract class RDD[T: ClassTag](
>         @transient private var _sc: SparkContext,
>         @transient private var deps: Seq[Dependency[_]]
>       ) extends Serializable with Logging

源码翻译：

>  RDD(弹性分布式数据集)，是spark中一个基本的抽象类，代表着一个不可变的可分区的元素集合，并且可以并行操作。这个抽象类包含了所有RDD最基本的可用操作，比如\`map\`, \`filter\`, and \`persist\`算子。除此之外，还包含一些键值对的操作，如 \`groupByKey\` and \`join\`。

ok，接下来重点便是源码红色标注部分：

> **1、A list of partitions**
> 
> **       RDD有很多partition构成；**
> 
> **解析：**
> 
> **     在spark计算中，有多少个partition就会对应有多少个task来执行**
> 
>  
> 
> **2、A function for computing each split**
> 
> **      对RDD做计算，相当于对RDD的每个split或partition做计算**
> 
> **解析：**
> 
> **      partition == split **
> 
> **      例如： rdda.map(\_ \* 2)等价于rdda.map(x => x \* 2)，因为rdd有多个分片/分区，对rdd操作其实底层就是对每个分区/分片操作**
> 
>  
> 
> **3、A list of dependencies on other RDDs**
> 
> **      RDD之间存在一系列的依赖关系**
> 
> 解析：
> 
>      rdda => rddb => rddc   存在依赖关系，为窄依赖 
> 
>      rddx => rddy => rddc    存在依赖关系，为宽依赖（假如rddb和rddy通过联合操作），至于宽窄依赖，后面再说
> 
>      rddhdfs      则不存在依赖关系，为Null
> 
>  
> 
> **4、Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)**
> 
>        如果RDD里面存的数据是key-value形式，则可以传递一个自定义的Partitioner进行重新分区，比如可以按key的hash值分区
> 
> **5、Optionally, a list of preferred locations to compute each split on (e.g. block locations for \* an HDFS file)**
> 
>         对于每个计算分片有用最佳计算位置
> 
>

源码解读：

### RDD： ###

> abstract class RDD\[T: ClassTag\](  
>     @transient private var \_sc: SparkContext,  
>     @transient private var deps: Seq\[Dependency\[\_\]\]  
>   ) extends Serializable with Logging\{ ... \}
> 
> 1）抽象类：不能直接使用，具体是实现该抽象类的子类完成
> 
> 2）T：泛型，里面可以装任意类型的元素
> 
> 3）logging：日志记录，2.0版本后不自带，需要自己写一个
> 
> 4）序列化：在分布式计算框架里，序列化框架性能的好坏直接影响整个框架性能的优劣
> 
>

特性一、

>     /**
>      * Implemented by subclasses to return the set of partitions in this RDD. This method will only
>      * be called once, so it is safe to implement a time-consuming computation in it.
>      *
>      * The partitions in this array must satisfy the following property:
>      *   `rdd.partitions.zipWithIndex.forall { case (partition, index) => partition.index == index }`
>      */
>     protected def getPartitions: Array[Partition]
> 
> **通过子类来实现rdd中包含了分区的集合是哪些，这个方法仅仅会被调用一次，所以用来实现耗时计算是安全的。**

特性二、

>     /**
>      * :: DeveloperApi ::
>      * Implemented by subclasses to compute a given partition.
>      */
>     @DeveloperApi
>     def compute(split: Partition, context: TaskContext): Iterator[T]
> 
> **通过子类来实现计算给定的分区。**

特性三、

>     /**
>      * Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only
>      * be called once, so it is safe to implement a time-consuming computation in it.
>      */
>     protected def getDependencies: Seq[Dependency[_]] = deps
> 
> **通过子类来是实现此rdd的父RDD**

特性四、

>     /** Optionally overridden by subclasses to specify how they are partitioned. */
>     @transient val partitioner: Option[Partitioner] = None
> 
> ![20200312162339108.png][]

特性五、

>     /**
>      * Optionally overridden by subclasses to specify placement preferences.
>      */
>     protected def getPreferredLocations(split: Partition): Seq[String] = Nil
> 
> **Nil:空List **

\--------------------------

用人品去感动别人，用行动去带动别人，用阳光去照耀别人，用坚持去赢得别人，要求自己每天都去做与目标有关的事情，哪怕每天只进步一点点，坚持下来你就是最优秀卓越的！欢迎大家加入大数据qq交流群：725967421    一起交流，一起进步！！  
\--------------------------

[http_spark.apache.org_docs_latest_rdd-programming-guide.html_resilient-distributed-datasets-rdds]: http://spark.apache.org/docs/latest/rdd-programming-guide.html#resilient-distributed-datasets-rdds
[20200312162339108.png]: https://img-blog.csdnimg.cn/20200312162339108.png