Development hadoop program on Windows

向右看齐 2022-04-25 04:04 150阅读 0赞

This post is about the development of hadoop program on windows. The detail configuration describes as follows:

**Step 1: copy the hadoop plugin to the eclipse's plugin directory.**

**Step 2:restart the ecplise IDE.**

**Step 3:configure the mapreduce plugin in eclipse IDE.**

![20130525103314323][]

**Step 4:mapping the hostname to the ip address in C:\\WINDOWS\\system32\\drivers\\etc\\hosts.**

\# e.g.

xxx.xxx.xxx.xxx master

**Step 5: you will encounter the errors, when running the programs on the hadoop.**

13/05/25 10:40:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    13/05/25 10:40:28 ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator1090172040\.staging to 0700
    Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator1090172040\.staging to 0700
    	at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
    	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
    	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
    	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
    	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
    	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:396)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
    	at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
    	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
    	at WordCount.main(WordCount.java:69)

Reason: The job starts with the local JobRunner and the jobClient isn't initialized with the configure files on eclipse IDE.

Solution: add the below codes after Configuration conf = new Configuration();

conf.set("mapred.job.tracker", "master's ip address:9001");

[20130525103314323]: /images/20220210/e7b05cd7fa8646b7a0ef9c2a175a61ac.png