spark - Caused by: java.net.UnknownHostException: XXX

我会带着你远行 2022-09-02 12:50 91阅读 0赞

## spark Caused by: java.net.UnknownHostException: XXX ##

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
           at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
           at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:283)
           at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:272)
           at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
    Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
           at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
           at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
           at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
           at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$3(CoarseGrainedExecutorBackend.scala:303)
           at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
           at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
           at scala.collection.immutable.Range.foreach(Range.scala:158)
           at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
           at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$1(CoarseGrainedExecutorBackend.scala:301)
           at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
           at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
           at java.security.AccessController.doPrivileged(Native Method)
           at javax.security.auth.Subject.doAs(Subject.java:422)
           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
           ... 4 more
    Caused by: java.io.IOException: Failed to connect to XXX
           at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:253)
           at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
           at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
           at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
           at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
    
     - [ ] Caused by: java.net.UnknownHostException: XXX
    
           at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
           at java.net.InetAddress.getAllByName(InetAddress.java:1193)
           at java.net.InetAddress.getAllByName(InetAddress.java:1127)
           at java.net.InetAddress.getByName(InetAddress.java:1077)
           at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156)
           at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153)
           at java.security.AccessController.doPrivileged(Native Method)
           at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153)
           at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41)
           at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61)
           at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
           at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
           at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
           at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
           at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:200)
           at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46)
           at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180)
           at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166)
           at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
           at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
           at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
           at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
           at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
           at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
           at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
           at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:984)
           at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:504)
           at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417)
           at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474)
           at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
           at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
           at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
           at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
           at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
           at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
           ... 1 more

## 解决方案 ##

我的报错原因：在client模式下忘记设置`spark.driver.host`以及`spark.driver.port`的属性值了。