大数据Hive系列之Hive数据仓库部署

柔光的暖阳◎ 2022-05-23 08:00 343阅读 0赞

# 一、部署准备 #

*  Hive依赖于Hadoop分布式集群，Hadoop分布式集群部署参考文章：[Apache Hadoop 分布式集群部署][Apache Hadoop]
 *  Hive依赖于MySQL数据库，MySQL数据库部署参考文章：[CentOS系统二进制安装MySQL][CentOS_MySQL]
 *  Hive软件包下载：[http://archive.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz][http_archive.apache.org_dist_hive_hive-2.1.1_apache-hive-2.1.1-bin.tar.gz]

# **二、部署Hive** #

【主机slave62上执行】

### 1.  创建hive工作目录，上传解压hive软件 ###

mkdir -p /apps/svr/hive/  
    tar -zxvf ~/apache-hive-2.1.1-bin.tar.gz -C /apps/svr/hive/

### 2.  部署 ###

cd /apps/svr/hive/apache-hive-2.1.1-bin/conf/  
    cp hive-env.sh.template hive-env.sh  
    cp hive-log4j2.properties.template hive-log4j2.properties

**2.1.  配置hive-env.sh**

vim hive-env.sh

export JAVA_HOME=/apps/svr/java/jdk1.8.0_172
    export HADOOP_HOME=/apps/svr/hadoop/hadoop-2.7.3
    export HIVE_CONF_DIR=/apps/svr/hive/apache-hive-2.1.1-bin/conf

**2.2.  配置log4j2**

mkdir -p /apps/logs/hive  
    vim hive-log4j2.properties

property.hive.log.dir = /apps/logs/hive/${sys:user.name}

**2.3.  配置hive-site.xml**

vim hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    
    <configuration>
    	
    	<property>
    		<name>javax.jdo.option.ConnectionURL</name>
    		<value>jdbc:mysql://localhost:3306/hive?characterEncoding=UTF-8&useSSL=false</value>
    	</property>
    	<property>
    		<name>javax.jdo.option.ConnectionDriverName</name>
    		<value>com.mysql.jdbc.Driver</value>
    	</property>
    	<property>
    		<name>javax.jdo.option.ConnectionUserName</name>
    		<value>hive</value>
    	</property>
    	<property>
    		<name>javax.jdo.option.ConnectionPassword</name>
    		<value>hive!@#123</value>
    	</property>
    	
    	 
    	<property>
    		<name>hive.metastore.warehouse.dir</name>
    		<value>hdfs://master60:9000/user/hive/warehouse</value>
    	</property>
    	<property>
    		<name>hive.exec.scratchdir</name>
    		<value>hdfs://master60:9000/tmp/hive/hive-${user.name}</value>
    	</property>
    	
    	 
    	<property>
    		<name>hive.exec.local.scratchdir</name>
    		<value>/tmp/hive/iotmp</value>
    	</property>
    	
    	
    	<property>
    		<name>hive.downloaded.resources.dir</name>
    		<value>/tmp/hive/iotmp</value>
    	</property>
    	
    	 
    	<property>
    		<name>hive.querylog.location</name>
    		<value>/tmp/${user.name}</value>
    	</property>
    	 
    	<property>
    		<name>hive.server2.logging.operation.log.location</name>
    		<value>/tmp/${user.name}/operation_logs</value>
    	</property>
    
    	
    	<property>
    		<name>hive.cli.print.header</name>
    		<value>true</value>
    	</property>
    	<property>
    		<name>hive.cli.print.current.db</name>
    		<value>true</value>
    	</property>
    </configuration>

**2.4.  上传并拷贝相关jar包**

cd /apps/svr/hive/apache-hive-2.1.1-bin/  
    cp ~/jar/mysql-connector-java-5.1.46.jar ./lib/  
    cp $HADOOP\_HOME/share/hadoop/common/hadoop-common-2.7.3.jar ./lib/

**2.5.  创建Hive仓库目录**

【主机master60上执行】

hdfs dfs -mkdir /tmp  
    hdfs dfs -mkdir -p /user/hive/warehouse  
    hdfs dfs -chmod g+w /tmp  
    hdfs dfs -chmod g+w /user/hive/warehouse

**2.6.  创建Hive元数据储存数据库**

create database hive default charset utf8 collate utf8_general_ci;
    grant all on hive.* to 'hive'@'%' identified by 'hive!@#123';
    flush privileges;

### 3.  配置hive环境变量 ###

**3.1.  配置.bash\_profile**

vim ~/.bash\_profile

# HIVE_HOME
    export HIVE_HOME=/apps/svr/hive/apache-hive-2.1.1-bin
    export HCAT_HOME=$HIVE_HOME/hcatalog/
    export PATH=$PATH:$HIVE_HOME/bin

**3.2.  立即生效**

source ~/.bash\_profile

### 4.  启动测试hive ###

**4.1.  初始化metastore库**

schematool -dbType mysql -initSchema

**4.2.  启动测试**

hive -e 'show tables'

[Apache Hadoop]: https://blog.csdn.net/volitationlong/article/details/80285123
[CentOS_MySQL]: https://blog.csdn.net/volitationLong/article/details/78362695
[http_archive.apache.org_dist_hive_hive-2.1.1_apache-hive-2.1.1-bin.tar.gz]: http://archive.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz