spark on yarn报错java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.Client

今天药忘吃喽~ 2022-06-04 08:07 280阅读 0赞

环境：

*  **hadoop2.7.4**
 *  **spark2.1.0**

--------------------

配置完spark-historyserver和yarn-timelineserver后，启动的时候没有报错，但是在spark用  
./spark-submit –class org.apache.spark.examples.SparkPi  
–master yarn  
–num-executors 3  
–driver-memory 1g  
–executor-cores 1  
/opt/spark-2.1.0-bin-hadoop2.7/examples/jars/spark-examples\_2.11-2.1.0.jar 20  
命令提交application的时候，报了如下错误

[root@node-2 bin]# ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --num-executors 3 --driver-memory 1g --executor-cores 1 /opt/spark-2.1.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.0.jar 20
    17/12/08 02:01:37 INFO SparkContext: Running Spark version 2.1.0
    17/12/08 02:01:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/12/08 02:01:38 INFO SecurityManager: Changing view acls to: root
    17/12/08 02:01:38 INFO SecurityManager: Changing modify acls to: root
    17/12/08 02:01:38 INFO SecurityManager: Changing view acls groups to: 
    17/12/08 02:01:38 INFO SecurityManager: Changing modify acls groups to: 
    17/12/08 02:01:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
    17/12/08 02:01:38 INFO Utils: Successfully started service 'sparkDriver' on port 40685.
    17/12/08 02:01:39 INFO SparkEnv: Registering MapOutputTracker
    17/12/08 02:01:39 INFO SparkEnv: Registering BlockManagerMaster
    17/12/08 02:01:39 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
    17/12/08 02:01:39 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
    17/12/08 02:01:39 INFO DiskBlockManager: Created local directory at /tmp/spark/blockmgr-5041ec61-5aa3-48ec-86ec-692db6efcce3
    17/12/08 02:01:39 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
    17/12/08 02:01:39 INFO SparkEnv: Registering OutputCommitCoordinator
    17/12/08 02:01:39 INFO Utils: Successfully started service 'SparkUI' on port 4040.
    17/12/08 02:01:39 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.82:4040
    17/12/08 02:01:39 INFO SparkContext: Added JAR file:/opt/spark-2.1.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.0.jar at spark://192.168.1.82:40685/jars/spark-examples_2.11-2.1.0.jar with timestamp 1512698499980
    Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
        at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 24 more
    17/12/08 02:01:43 INFO DiskBlockManager: Shutdown hook called
    17/12/08 02:01:43 INFO ShutdownHookManager: Shutdown hook called
    17/12/08 02:01:43 INFO ShutdownHookManager: Deleting directory /tmp/spark/spark-c9cb41bd-2a3e-4519-92a8-dd3cf232458f
    17/12/08 02:01:43 INFO ShutdownHookManager: Deleting directory /tmp/spark/spark-c9cb41bd-2a3e-4519-92a8-dd3cf232458f/userFiles-0d69f3fd-fd19-4b50-ac77-b52dcc686c78

意思是找不到com.sun.jersey.api.client.config.ClientConfig这个类．查找答案说是缺少jersey-core．jar包  
我cd到SPARK\_HOME/jars路径下，发现存在的jersey包为以下  
![这里写图片描述][SouthEast]  
确实没有这个jar包，而且jersey版本为2.22.2，然后我去官方API中查证，发现jersey2.22.2版本中并没有jersey-core这个jar包，更没有com.sun.jersey.api.client.config.ClientConfig这个类，我又去查看jersey1.9的API,成功发现了缺失的类，看来是jersey在以后的版本中去掉了这个类．

[SouthEast]: /images/20220604/f89091b51dcf49ad80fb89da4eb079e2.png