Spark监控篇

亦凉 2023-07-19 12:05 45阅读 0赞

[Spark监控官网地址][Spark]

一、每个Spark程序都有的4040端口Web 界面进行监控  
Every SparkContext launches a web UI, by default on port 4040, that displays useful information about the application. This includes:

*  A list of scheduler stages and tasks
 *  A summary of RDD sizes and memory usage
 *  Environmental information.
 *  Information about the running executors

![format_png][]

image.png

这种方式有一个很严重的弊端就，就是这个4040 Web UI只在程序运行期间可以看，程序运行完毕后就看不了。所以就有了我们的第二种监控，spark-history-server。

二、通过配置spark-history-server相关参数来监控Application  
这种方式会创建一个默认的18080 web interface界面查看完成和未完成的Spark Application信息。说先会将Spark Application信息写入到**spark.eventLog.dir**目录下，然后spark-history-server回去这个目录把日志信息渲染到Web UI供开发人员查看。  
1、配置Spark Application信息写入的的路径

[hadoop@hadoop000 logs]$ cd $SPARK_HOME/conf
    ---------------------------------------------------------------
    vim spark-defaults.conf
    ---------------------------------------------------------------
     spark.eventLog.enabled           true
     spark.eventLog.dir               hdfs://hadoop000:8020/spark_eventLog

2、Spark History Server Configuration Option在Spark-env.sh中完成

vim spark-env.sh
    ---------------------------------------------------------------
    SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://hadoop000:8020/spark_eventLog"

其他相关参数:

<table> 
 <thead> 
  <tr> 
   <th>Property Name</th> 
   <th>Default</th> 
   <th>Meaning</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>spark.history.fs.update.interval</td> 
   <td>10s</td> 
   <td>文件系统历史提供程序在日志目录中检查新日志或更新日志的周期。较短的间隔可以更快地检测新应用程序，但代价是需要更多的服务器负载重新读取更新的应用程序。一旦更新完成，已完成和未完成的应用程序的清单将反映更改</td> 
  </tr> 
  <tr> 
   <td>spark.history.retainedApplications</td> 
   <td>50</td> 
   <td>在缓存中保留UI数据的应用程序数量。如果超过这个上限，那么最老的应用程序将从缓存中删除。如果应用程序不在缓存中，则必须从磁盘加载它(如果是从UI访问它)</td> 
  </tr> 
  <tr> 
   <td>spark.history.ui.port</td> 
   <td>18080</td> 
   <td>Web 默认端口</td> 
  </tr> 
  <tr> 
   <td>spark.history.fs.cleaner.enabled</td> 
   <td>false</td> 
   <td>是否周期性的删除storage中的event log(生产必定是true)</td> 
  </tr> 
  <tr> 
   <td>spark.history.fs.cleaner.interval</td> 
   <td>1d</td> 
   <td>多久删除一次</td> 
  </tr> 
  <tr> 
   <td>spark.history.fs.cleaner.maxAge</td> 
   <td>7d</td> 
   <td>每次删除多久的event log，配合上一个参数就是每天删除前七天的数据</td> 
  </tr> 
 </tbody> 
</table>

3、HDFS创建存放event log 路径

hadoop fs -mkdir spark_eventLog

4、启动historyserver

./start-history-server.sh

![format_png 1][]

image.png

每个Application都会生成一个日志文件

[hadoop@hadoop000 sbin]$ hadoop fs -ls /spark_eventLog
    Found 1 items
    -rwxrwx---   1 hadoop supergroup      38056 2019-05-19 13:33 /spark_eventLog/local-1558243986730

三、REST API监控  
如果公司有自己的监控平台，想获取到Spark Application日志信息到平台展示可以通过REST API

1、获取所有的Application  
[http://hadoop000:18080/api/v1/applications][http_hadoop000_18080_api_v1_applications]

![format_png 2][]

image.png

2、获取正在跑或者完成的application  
[http://hadoop000:18080/api/v1/applications/?status=running][http_hadoop000_18080_api_v1_applications_status_running]  
[http://hadoop000:18080/api/v1/applications/?status=completed][http_hadoop000_18080_api_v1_applications_status_completed]

3、只获取1条application  
[http://hadoop000:18080/api/v1/applications/?limit=1][http_hadoop000_18080_api_v1_applications_limit_1]  
还可以获取到指定时间内的application，具体请参考官网

4、获取某一具体的application  
[http://hadoop000:18080/api/v1/applications/local-1558243986730][http_hadoop000_18080_api_v1_applications_local-1558243986730]

四、基于ES的监控  
[参考美团文章][Link 1]

[Spark]: https://links.jianshu.com/go?to=http%3A%2F%2Fspark.apache.org%2Fdocs%2Flatest%2Fmonitoring.html
[format_png]: /images/20230528/8da4081d45874383bd49f766fd3a0c81.png
[format_png 1]: /images/20230528/38b8fc6fa6364d1192f2d5bb3b729e71.png
[http_hadoop000_18080_api_v1_applications]: https://links.jianshu.com/go?to=http%3A%2F%2Fhadoop000%3A18080%2Fapi%2Fv1%2Fapplications
[format_png 2]: /images/20230528/8504eba9af784494959726f784c24c0a.png
[http_hadoop000_18080_api_v1_applications_status_running]: https://links.jianshu.com/go?to=http%3A%2F%2Fhadoop000%3A18080%2Fapi%2Fv1%2Fapplications%2F%3Fstatus%3Drunning
[http_hadoop000_18080_api_v1_applications_status_completed]: https://links.jianshu.com/go?to=http%3A%2F%2Fhadoop000%3A18080%2Fapi%2Fv1%2Fapplications%2F%3Fstatus%3Dcompleted
[http_hadoop000_18080_api_v1_applications_limit_1]: https://links.jianshu.com/go?to=http%3A%2F%2Fhadoop000%3A18080%2Fapi%2Fv1%2Fapplications%2F%3Flimit%3D1
[http_hadoop000_18080_api_v1_applications_local-1558243986730]: https://links.jianshu.com/go?to=http%3A%2F%2Fhadoop000%3A18080%2Fapi%2Fv1%2Fapplications%2Flocal-1558243986730
[Link 1]: https://links.jianshu.com/go?to=https%3A%2F%2Ftech.meituan.com%2F2016%2F11%2F04%2Fspark-streaming-es.html