spark streaming 读kafka写入hbase报错

àì夳堔傛蜴生んèń 2022-04-12 03:51 529阅读 0赞

使用spark streaming消费kafka topic系消息， 再写入到hbase中， 使用spark submit 他job时，报了一些错误， 此处归纳一下：

1, io.netty.handler.codec.EncoderException: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.(Ljava/io/File;JJ)V\`

问题： 使用的是 spark-2.1.0-bin-hadoop2.7， scala-2.11.8  
解决： 使用更高版本的spark : spark-2.4.0-bin-hadoop2.7

2, 使用spark-2.4.0, 又继续报错

Exception in thread "main" java.lang.AbstractMethodError
            at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
            at org.apache.spark.streaming.kafka010.KafkaUtils$.initializeLogIfNecessary(KafkaUtils.scala:40)
            at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
            at org.apache.spark.streaming.kafka010.KafkaUtils$.log(KafkaUtils.scala:40)
            at org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66)
            at org.apache.spark.streaming.kafka010.KafkaUtils$.logWarning(KafkaUtils.scala:40)
            at org.apache.spark.streaming.kafka010.KafkaUtils$.fixKafkaParams(KafkaUtils.scala:208)
            at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.<init>(DirectKafkaInputDStream.scala:66)
            at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:150)
            at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:127)
            at a.SparkafkaStreaming$.readKafkaAndWriteHbase(SparkafkaStreaming.scala:55)
            at a.SparkafkaStreaming$.main(SparkafkaStreaming.scala:21)
            at a.SparkafkaStreaming.main(SparkafkaStreaming.scala)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
            at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
            at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
            at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
            at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
            at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)

问题： spark-sql-kafka的版本是2.2，而spark的版本是2.4，  
解决： 修改spark-sql-kafka的版本为2.4

3, spark 写入hbase报错

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
            at a.SparkafkaStreaming$.sparkWriteHbase(SparkafkaStreaming.scala:93)
            at a.SparkafkaStreaming$$anonfun$readKafkaAndWriteHbase$2$$anonfun$apply$1.apply(SparkafkaStreaming.scala:76)
            at a.SparkafkaStreaming$$anonfun$readKafkaAndWriteHbase$2$$anonfun$apply$1.apply(SparkafkaStreaming.scala:74)
            at scala.collection.Iterator$class.foreach(Iterator.scala:891)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)

问题： [spark-env.sh][] 没有关联hbase配置  
解决：  
export HIVE\_CONF\_DIR=/soft/hive/conf  
export SPARK\_CLASSPATH=$SPARK\_CLASSPATH:/soft/hbase/lib/\*

4, spark 连接hadoop报错

java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPassword(Ljava/lang/String;)[C
            at org.apache.spark.SSLOptions$$anonfun$8.apply(SSLOptions.scala:188)
            at org.apache.spark.SSLOptions$$anonfun$8.apply(SSLOptions.scala:188)
            at scala.Option.orElse(Option.scala:289)
            at org.apache.spark.SSLOptions$.parse(SSLOptions.scala:188)
            at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:117)
            at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359)
            at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:359)
            at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)
            at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)
            at scala.Option.map(Option.scala:146)
            at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:366)
            at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
            at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
            at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决： 把spark版本改为 spark2.2.1

[spark-env.sh]: http://spark-env.sh