SequenceFile doesn't work with GzipCodec without native-hadoop code的解决办法

偏执的太偏执、 2022-10-22 09:21 5阅读 0赞

前几天将hive的版本由 0.8.1升级到 0.11.0 ，新版本新增了很多内置函数，执行效率比之前也有了一定的提升，  
但是有新的问题产生，问题如下:    
create table tab1  
   (  
    id int,  
   name string  
   );  
    
   create table tab2  
   (  
    id int  
   );  
     
   select a.name,count(\*)  
   from tab1 a left outer join tab2 b on a.id = b.id  
   group by a.name;  
  
执行上面的语句的时候  
会报错,错误信息：  
Diagnostic Messages for this Task:  
java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!  
        at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:313)  
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:479)  
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)  
        at org.apache.hadoop.mapred.Child$4.run(Child.java:266)  
        at java.security.AccessController.doPrivileged(Native Method)  
        at javax.security.auth.Subject.doAs(Subject.java:396)  
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)  
        at org.apache.hadoop.mapred.Child.main(Child.java:260)  
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!  
        at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:240)  
        at org.apache.hadoop.hive.ql.exec.FileSink  
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask  
  
通过对比发现，有个配置参数的默认值是不同的。

hive.exec.compress.intermediate=false   0.8.1

hive.exec.compress.intermediate=ture   0.11.0

将这个参数修改为false之后 这个问题就修复了。