Spark性能优化涉及的相关参数配置(中文版wiki)

ゝ一世哀愁。 2022-05-31 01:53 184阅读 0赞

** Spark性能优化涉及的相关参数配置(中文版wiki)**

# Spark性能相关参数配置 #

随着Spark的逐渐成熟完善, 越来越多的可配置参数被添加到Spark中来, 在Spark的官方文档http://spark.apache.org/docs/latest/configuration.html 中提供了这些可配置参数中相当大一部分的说明.

但是文档的更新总是落后于代码的开发的, 还有一些配置参数没有来得及被添加到这个文档中, 最重要的是在这个文档中,对于许多的参数也只能简单的介绍它所代表的内容的字面含义, 如果没有一定的实践基础或者对其背后原理的理解, 往往无法真正理解该如何针对具体应用场合进行合理配置。

本文试图通过阐述这其中部分参数的工作原理和配置思路, 和大家一起探讨一下如何根据实际场合对Spark进行配置优化。需要注意的是，理论上，没有绝对正确的配置（否则也就不需要对应的配置参数了，Spark框架内部直接写死就好了），所以请结合自己的实际情况，辩证的看下面的内容。

由于本文主要针对和性能相关的一些配置参数进行阐述，所以基本不会覆盖其它和性能没有太多关系的配置参数。

*  [*Shuffle 相关*][Shuffle]
 *  [*Storage相关配置参数*][Storage]
 *  [*压缩和序列化相关*][Link 1]
 *  *[schedule调度相关][schedule]*

## 章节 ##

*  [Shuffle 相关][Shuffle]
    
     *  [spark.shuffle.manager][]
     *  [spark.shuffle.sort.bypassMergeThreshold][]
     *  [spark.shuffle.consolidateFiles][]
     *  [spark.shuffle.spill][]
     *  [spark.shuffle.memoryFraction / spark.shuffle.safetyFraction][spark.shuffle.memoryFraction _ spark.shuffle.safetyFraction]
     *  [spark.shuffle.spill.compress / spark.shuffle.compress][spark.shuffle.spill.compress _ spark.shuffle.compress]
 *  [Storage相关配置参数][Storage]
    
     *  [spark.local.dir][]
     *  [spark.executor.memory][]
     *  [spark.storage.memoryFraction][]
     *  [spark.streaming.blockInterval][]
 *  [压缩和序列化相关][Link 1]
    
     *  [spark.serializer][]
     *  [spark.rdd.compress][]
     *  [spark.broadcast.compress][]
     *  [spark.io.compression.codec][]
 *  [schedule调度相关][schedule]
    
     *  [spark.cores.max][]
     *  [spark.task.cpus][]
     *  [spark.scheduler.mode][]
     *  [spark.locality.wait][]
     *  [spark.speculation][]

参考：http://spark-config.readthedocs.io/en/latest/

[Shuffle]: http://spark-config.readthedocs.io/en/latest/shuffle.html
[Storage]: http://spark-config.readthedocs.io/en/latest/storage.html
[Link 1]: http://spark-config.readthedocs.io/en/latest/compress.html
[schedule]: http://spark-config.readthedocs.io/en/latest/scheduler.html
[spark.shuffle.manager]: http://spark-config.readthedocs.io/en/latest/shuffle.html#spark-shuffle-manager
[spark.shuffle.sort.bypassMergeThreshold]: http://spark-config.readthedocs.io/en/latest/shuffle.html#spark-shuffle-sort-bypassmergethreshold
[spark.shuffle.consolidateFiles]: http://spark-config.readthedocs.io/en/latest/shuffle.html#spark-shuffle-consolidatefiles
[spark.shuffle.spill]: http://spark-config.readthedocs.io/en/latest/shuffle.html#spark-shuffle-spill
[spark.shuffle.memoryFraction _ spark.shuffle.safetyFraction]: http://spark-config.readthedocs.io/en/latest/shuffle.html#spark-shuffle-memoryfraction-spark-shuffle-safetyfraction
[spark.shuffle.spill.compress _ spark.shuffle.compress]: http://spark-config.readthedocs.io/en/latest/shuffle.html#spark-shuffle-spill-compress-spark-shuffle-compress
[spark.local.dir]: http://spark-config.readthedocs.io/en/latest/storage.html#spark-local-dir
[spark.executor.memory]: http://spark-config.readthedocs.io/en/latest/storage.html#spark-executor-memory
[spark.storage.memoryFraction]: http://spark-config.readthedocs.io/en/latest/storage.html#spark-storage-memoryfraction
[spark.streaming.blockInterval]: http://spark-config.readthedocs.io/en/latest/storage.html#spark-streaming-blockinterval
[spark.serializer]: http://spark-config.readthedocs.io/en/latest/compress.html#spark-serializer
[spark.rdd.compress]: http://spark-config.readthedocs.io/en/latest/compress.html#spark-rdd-compress
[spark.broadcast.compress]: http://spark-config.readthedocs.io/en/latest/compress.html#spark-broadcast-compress
[spark.io.compression.codec]: http://spark-config.readthedocs.io/en/latest/compress.html#spark-io-compression-codec
[spark.cores.max]: http://spark-config.readthedocs.io/en/latest/scheduler.html#spark-cores-max
[spark.task.cpus]: http://spark-config.readthedocs.io/en/latest/scheduler.html#spark-task-cpus
[spark.scheduler.mode]: http://spark-config.readthedocs.io/en/latest/scheduler.html#spark-scheduler-mode
[spark.locality.wait]: http://spark-config.readthedocs.io/en/latest/scheduler.html#spark-locality-wait
[spark.speculation]: http://spark-config.readthedocs.io/en/latest/scheduler.html#spark-speculation