Exception in thread “main“ java.lang.NoSuchFieldError: HIVE_STATS_JDBC

Exception in thread “main“ java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT

男娘i 2023-09-24 16:36 71阅读 0赞

问题

Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT

详细问题

springboot集成spark，创建sparkSession对象后，对于sparkSession对象添加hive支持，控制台报错

Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
    at org.apache.spark.sql.hive.HiveUtils$.formatTimeVarsForHiveClient(HiveUtils.scala:204)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:285)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141)
    at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:759)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:749)
    at org.apache.spark.sql.execution.command.ShowTablesCommand$$anonfun$16.apply(tables.scala:745)
    at org.apache.spark.sql.execution.command.ShowTablesCommand$$anonfun$16.apply(tables.scala:745)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.command.ShowTablesCommand.run(tables.scala:745)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
    at com.haut.edu.epidemicstatisticsbackend2.controller.Test_Hive.main(Test_Hive.java:107)

解决方案

若pom文件中存在该依赖，进行删除

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>2.1.0</version>
</dependency>

原因

上述依赖与

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.11</artifactId>
    <version>2.4.4</version>
</dependency>

产生冲突，笔者希望spark-hive_2.11依赖使spark数据处理与hive数据仓库建立连接，但是编译器使用
hive-jdbc使spark数据处理与hive数据仓库建立连接，由于笔者后端无需与hive数据库直接通信，因此删除hive-jdbc即可保证使用spark-hive_2.11依赖使spark数据处理与hive数据仓库建立连接，另外，笔者认为spark hive java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT一文与Spark运行CDH6.3.2碰到的由于hive驱动引起的问题解决方法所提到的解决办法大概是可以解决该问题，但由于笔者理解能力与编程能力都相对较差，对于上述两文的解决方案未能实现验证

参考文献

java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
spark hive java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
原创不易
转载请标明出处
如果对你有所帮助别忘啦点赞支持哈
在这里插入图片描述