window下Sparksql 无法获取文件权限问题

快来打我* 2022-12-21 11:59 194阅读 0赞

### Caused by: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F D:\\tmp\\hive ###

Caused by: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F D:\tmp\hive
    	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:762)
    	at org.apache.hadoop.util.Shell.execCommand(Shell.java:859)
    	at org.apache.hadoop.util.Shell.execCommand(Shell.java:842)
    	at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
    	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:587)
    	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:562)
    	at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
    	at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
    	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    	at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:183)
    	at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:117)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    	at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:271)
    	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
    	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
    	at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
    	at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
    	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
    	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
    	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
    	at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    	at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    	at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141)
    	at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136)
    	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
    	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)
    	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)
    	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)
    	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:258)
    	at org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)
    	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    	at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)
    	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
    	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
    	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
    	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)
    	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
    	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
    	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
    	at com.offcn.spark.Driver2$.main(Driver2.scala:10)
    	at com.offcn.spark.Driver2.main(Driver2.scala)
    
    	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:625)
    	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:562)
    	at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
    	at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
    	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    	... 41 more

### 解决方法 ###

这个问题主要是因为 Spark初始化Hive上下文所需的winutils.exe丢失文件引起的，而该 文件又依赖于 Hadoop，后者需要Windows上有hadoop才可以;

winutils.exe下载链接：  
[https://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe][https_github.com_steveloughran_winutils_raw_master_hadoop-2.6.0_bin_winutils.exe]

Hadoop下载链接:  
[http://spark.apache.org/downloads.html][http_spark.apache.org_downloads.html]  
选择spark版本,这里我选的是2.4.4

winutils.exe和hadoop都下载完后,将winutils.exe放入下载的hadoop\\bin目录下

Hadoop存储位置需和你 报错信息中的 \\tmp\\hive的位置要一致

上述报错中,我报错位置是 null ls -F D:\\tmp\\hive 所以我将hadoop放到了D:\\下

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhbmdzaHVtaW5qYXZh_size_16_color_FFFFFF_t_70][]

上述步骤都完成后,在我的电脑属性中配置环境变量HADOOP\_HOME指定到我们haodoop的根目录下  
**然后在Path下配置%HADOOP\_HOME%\\bin**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhbmdzaHVtaW5qYXZh_size_16_color_FFFFFF_t_70 1][]

**环境变量配置完后,win+r打开cmd窗口**

执行完后,去我们存放hadoop的根目录下查看是否,生成目录tmp/derby.log

当成功后,退出spark 切换盘符到存放hadoop的根目录下,重新打开cmd

输入%HADOOP\_HOME%\\bin\\winuitls.exe chmod 777 \\tmp\\hive 修改文件夹权限

**此时,如果你的ideal正在运行,请关闭它(最好这样做,不关闭有可能前面步骤你都做对了,但是还是会报之前的错)**

**需要idea 重新加载环境配置文件**

重启ideal 运行你的程序,一般就没问题了

[https_github.com_steveloughran_winutils_raw_master_hadoop-2.6.0_bin_winutils.exe]: https://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe
[http_spark.apache.org_downloads.html]: http://spark.apache.org/downloads.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhbmdzaHVtaW5qYXZh_size_16_color_FFFFFF_t_70]: /images/20221120/1ed2a7a6d3ac4d90bc78a726433cfb60.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhbmdzaHVtaW5qYXZh_size_16_color_FFFFFF_t_70 1]: /images/20221120/7372a11e179043bbaa241061afcb58cf.png