hive 0.12 安装配置-蒲公英云

基本安装过程参考hive官方的安装指导：

https://cwiki.apache.org/confluence/display/Hive/GettingStarted\#GettingStarted-InstallationandConfiguration

但有些细节官方指导并没有写，需要注意：

1. 生成配置文件

进入hive下的conf目录，将

hive-env.sh.template、hive-exec-log4j.properties.template、hive-log4j.properties.template重命名，去掉”.template”

将hive-default.xml.template重命名为hive-site.xml

（hive-env.sh.template文件中存在一个bug，第2000行，auth,应该改成auth，否则启动时会报错

[Fatal Error] hive-site.xml16: The element type “value” must be terminated by the matching end-tag ““.

14/01/21 16:10:39 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: The element type “value” must be terminated by the matching end-tag “.）

2. 修改hive-env.sh文件

配置HADOOP_HOME，将其设为hadoop所在目录，配置HIVE_CONF_DIR为hive配置文件所在目录，HIVE_AUX_JARS_PATH配置hive用到的jar包所在的路径主要是在hive、hbase集成时会用到

配置完成，就可以到hive的bin目录下，执行./hive，如果报错：

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

还需要找到hadoop配置所在路径，修改hadoop-env.sh中的export HADOOP_CLASSPATH项，

export HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.94.9.jar:$HBASE_HOME/hbase-0.94.8-test.jar:${HBASE_HOME}/lib/zookeeper-3.4.5.jar:${HBASE_HOME}/lib/guava-11.0.2.jar

改成

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.94.9.jar:$HBASE_HOME/hbase-0.94.8-test.jar:${HBASE_HOME}/lib/zookeeper-3.4.5.jar:${HBASE_HOME}/lib/guava-11.0.2.jar

再执行bin/hive，就可以看到hive> 了

但是，还没完，这时候看一下hive的配置文件/tmp/root/hive.log(hive配置文件的路径在conf/hive-log4j.properties下配置)，提示

2014-01-21 12:13:11,556 WARN common.LogUtils (LogUtils.java:logConfigLocation(142)) - hive-site.xml not found on CLASSPATH

说明hive-site.xml没有被找到，这也需要修改hadoop的hadoop-env.sh，再将HADOOP_CLASSPATH改为：

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.94.9.jar:$HBASE_HOME/hbase-0.94.8-test.jar:${HBASE_HOME}/lib/zookeeper-3.4.5.jar:${HBASE_HOME}/lib/guava-11.0.2.jar:/opt/apache/hive-0.12.0-bin/conf/:/opt/apache/hive-0.12.0-bin/lib/:$HBASE_HOME/conf

即再加入hive配置文件、jar包所在路径、hbase配置文件所在路径（如果需要整合hbase的话），重新执行bin/hive，日志中就没有hive-site.xml not found on CLASSPATH的提示了。

参考 http://genius-bai.iteye.com/blog/643691

 http://stackoverflow.com/questions/14353394/hive-site-xml-not-found-on-classpath

3. 修改hive-site.xml文件

如果使用mysql存储元数据的话，需要修改hive-site.xml文件，主要是针对mysql相关的配置，内容如下：

javax.jdo.option.ConnectionURL

jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true

JDBC connect string for a JDBC metastore

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

Driver class name for a JDBC metastore

javax.jdo.option.ConnectionUserName

hive

username to use against metastore database

javax.jdo.option.ConnectionPassword

111111

password to use against metastore database

其中javax.jdo.option.ConnectionUserName、javax.jdo.option.ConnectionPassword是连接mysql时的用户名、密码。

4. mysql配置及遇到的问题

mysql的安装过程不再累述，第一次运行前需要执行

mysql_install_db

之后，运行mysqld_safe &，启动mysqld

再运行

mysql ，进入mysql命令行，执行

mysql> CREATE USER hive IDENTIFIED BY “111111”; 创建hive用户，密码为111111

mysql> GRANT ALL PRIVILEGES ON *.* TO hive@localhost IDENTIFIED BY “111111” WITH GRANT OPTION; 为hive用过赋予最大权限

其中hive、111111与hive-site.xml配置文件的javax.jdo.option.ConnectionUserName、javax.jdo.option.ConnectionPassword对应，

在退回到shell的命令行下，执行

mysql -uhive -p111111

后，能够进入到mysql>，就数目hive用户创建成功

参考 http://blog.csdn.net/luyee2010/article/details/8466146

 http://xiaolangit.blog.51cto.com/3343422/1269258
 http://www.eygle.com/digest/2008/01/mysql\_create\_user.html
 http://www.cnblogs.com/analyzer/articles/1045072.html

配置完mysql的用户名密码，还要安装mysql-connector-java

执行yum install mysql-connector-java命令，然后find / -name “mysql-connector-java*“,就能找到

/usr/share/java/mysql-connector-java-5.1.17.jar，

然后将其拷贝到hive的lib目录既可

参考 http://blog.sina.com.cn/s/blog\_81e6c30b0101a3jm.html

问题1

如果执行

hive> show tables;

出现如下提示

FAILED: Error in metadata: java.lang.RuntimeException:

Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

八成是以上mysql相关的配置、用户名、密码错误

问题2

如果执行

hive> CREATE TABLE pokes (foo INT, bar STRING);

FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory

NestedThrowables:

java.lang.reflect.InvocationTargetException

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

主要可能是mysql-connector-java没有装好

问题3

如果执行

hive> CREATE TABLE pokes (foo INT, bar STRING);

2014-01-21 12:16:24,357 ERROR ql.Driver (SessionState.java:printError(419)) -

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.EncodingUtils.setBit(BIZ)B

2014-01-21 12:16:24,357 ERROR ql.Driver (SessionState.java:printError(419)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.EncodingUtils.setBit(BIZ)B

执行locate *thrift*.jar后，可以看到如下结果：

/opt/apache/hadoop-1.0.4/lib/libthrift-0.8.0.jar

/opt/apache/hbase-0.94.8/lib/libthrift-0.8.0.jar

而hive自带的是

/opt/apache/hive-0.12.0-bin/lib/libthrift-0.9.0.jar

libthrift的版本冲突造成的问题，解决方法就是将hive的libthrift覆盖hadoop和hbase下相应libthrift，再启动hive，建表，问题消失

问题4

如果执行

hive>show databases;

报错：

ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

日志中出现如下内容：

14/06/29 10:40:20 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1074)
at org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2198)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:328)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1212)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1072)
… 19 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
… 24 more
Caused by: MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:5638)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:5622)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
at $Proxy10.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:403)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:286)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:121)
… 29 more

则修改hive-site.xml文件中的hive.metastore.schema.verification，将其置为false，重启后，问题消失

参考https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

查找问题的时候，可以执行 ./hive -hiveconf hive.root.logger=DEBUG,console ，能够看到详细的调试信息，方便定位问题。比如当用户名密码配置有问题是，执行show databases;就会打印错误信息，其中就会带有“java.sql.SQLException: Access denied for user ‘hive’@’localhost’ (using password: YES)”