hue并发重用同一个hiveserver连接报错：hive on spark任务运行Connection to remote Spark driver was los问题

逃离我推掉我的手 2024-03-23 00:30 9阅读 0赞

1.2 hive on spark任务运行过程中报错Connection to remote Spark driver was los问题  
报错信息如下：

org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Spark job failed during runtime. Please check stacktrace for the root cause.

Caused by: java.lang.IllegalStateException: Connection to remote Spark driver was lost

分析：

1、hue上执行sql17点54分报错，异常为连接不上远端driver，连接关掉

2022-02-18 17:54:53,038 | ERROR | HiveServer2-Background-Pool: Thread-5893035 | Failed to monitor Job\[-1\] with exception ‘java.lang.IllegalStateException(Connection to remote Spark driver was lost)’ Last known state = STARTED  
java.lang.IllegalStateException: Connection to remote Spark driver was lost

2． 找到该spark作业对应的appid和sparksession id

2022-02-18 17:49:03,592 INFO SparkContext：Submitted application：Hive on Spark （sessionId=a0197a4e-2920-46f2-abc5-e2d0d94150be  
在17:54 driver发出shutdown命令，hiveserver连接不上spark driver

3． 从hiveserver日志中发现sparkSession由以下线程关闭。关闭主要原因是hue连接中切换了引擎，从spark切换成MR。

2022-02-18 17:54:55,620 | INFO | HiveServer2-Background-Pool: Thread-5893035 | Trying to close SparkSession a0197a4e-2920-46f2-abc5-e2d0d94150be

2022-02-18 17:54:55,624 | INFO | HiveServer2-Background-Pool: Thread-5893035 | SparkSession a0197a4e-2920-46f2-abc5-e2d0d94150be is successfully closed

2022-02-18 17:54:55,633 | WARN | HiveServer2-Background-Pool: Thread-5893035 | Hive-on-MR is deprecated in hive2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases

再次测试：

1、开启hue窗口执行hive on spark作业

2、spark作业启动后，在开一个hue窗口执行mr引擎作业，两个作业session一样，都运行在同一个连接，mr任务运行后spark任务中断。

规避：使用同一引擎或者在beeline中执行

hive on spark 原理: 一般情况下一个连接同时只处理一个sql，例如beeline（对应一个hiveserver连接）。当运行hive on spark，同一个连接只会启动一个sparkSession，同一个连接的任务使用同一个sparkSession。

问题原因：hue并发重用同一个hiveserver连接，也就是hue一个连接可以同时处理多个sql。在hue上如果两个sql运行的引擎不一样，例如sql1 是hive on spark任务，sql2是mr任务。sql1起spark作业，sql2后起mr作业。引擎从spark切换成mr, 在hive中会将sparkSession关闭，sparkSession关闭导致正在运行任务失败。由于两个sql对应同一个连接，所以sql1的任务会失败。