Linux搭建sqoop数据迁移工具及简单的数据迁移测试

桃扇骨 2021-12-12 13:34 689阅读 0赞

### 1.准备环境及工具 ###

*  JDK 1.8 已搭建
 *  Hadoop 3.1.1 已搭建
 *  zookeeper 3.5.5 [下载][Link 1]
 *  hbase 2.2.0 [下载][Link 2]
 *  hive 3.1.1 已搭建
 *  sqoop-1.4.7 [下载][Link 3]
 *  ojdbc8(该版本的驱动支持jdk1.8) [下载][Link 4]

### 2.搭建zookeeper 3.5.5单机模式 ###

2.1 解压zookeeper

tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz

2.2 在zookeeper 3.5.5目录下新建data，logs两个文件夹

cd /home/fangw/bigData/zookeeper-3.5.5
    mkdir data
    mkdir logs

2.3 将zoo\_sample.cfg重命名为zoo.cfg

cd /home/fangw/bigData/zookeeper-3.5.5/conf
    cp zoo_sample.cfg zoo.cfg

2.4 修改zoo.cfg文件，主要修改以下几个地方

vim zoo.cfg

dataDir=/home/fangw/bigData/zookeeper-3.5.5/data
    dataLogDir=/home/fangw/bigData/zookeeper-3.5.5/data/logs
    clientPort=2181

2.5 配置完以后，启动zookeeper服务

cd /home/fangw/bigData/zookeeper-3.5.5/bin
    ./zkServer.sh start

2.6 启动完成后，查看服务状态

./zkServer.sh status
    ps -ef | grep zookeeper

### 3.搭建hbase-2.2.0 ###

3.1 解压hbase-2.2.0-bin.tar.gz

tar -zxvf hbase-2.2.0-bin.tar.gz

3.2 配置hbase-env.sh，主要修改如下配置

vim /home/fangw/bigData/hbase-2.2.0/conf/hbase-env.sh

#指定HBase是否使用HBase本身自带的Zookeeper
    export HBASE_MANAGES_ZK=false
    #指定Jdk目录
    export JAVA_HOME=/home/fangw/jdk1.8.0_191
    #指定Hadoop配置文件目录
    export HBASE_CLASSPATH=/home/fangw/bigData/hadoop-3.1.1

3.3 新建文件夹

cd /home/fangw/bigData/hbase-2.2.0
    mkdir data
    mkdir tmp
    mkdir zoo

3.4 配置hbase-site.xml，配置内容如下

<configuration>
      <property>  
        <name>hbase.rootdir</name>  
        <value>/home/fangw/bigData/hbase-2.2.0/data</value>  
      </property>  
      <property>  
        <name>hbase.tmp.dir</name>  
        <value>/home/fangw/bigData/hbase-2.2.0/tmp</value>  
      </property>  
      <property>  
        <name>hbase.zookeeper.quorum</name>  
        <value>10.0.72.13</value>  
      </property>  
      <property>
         <name>hbase.zookeeper.property.clientPort</name>
         <value>2182</value>                                                                                                                                             
      </property>
      <property>  
        <name>hbase.zookeeper.property.dataDir</name>  
        <value>/home/fangw/bigData/hbase-2.2.0/zoo</value>  
      </property>  
      <property>  
        <name>hbase.cluster.distributed</name>  
        <value>false</value>  
      </property>
    </configuration>

3.5 启动Hbase

start-hbase.sh

### 4.搭建Sqoop-1.4.7 ###

4.1 解压sqoop-1.4.7.bin\_\_hadoop-2.6.0.tar.gz

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

4.2 使用WinSCP将sqoop-env-template.sh重命名为sqoop-env.sh，在sqoop-env.sh文件中添加如下内容

export HADOOP_COMMON_HOME=/home/fangw/bigData/hadoop-3.1.1
    export HADOOP_MAPRED_HOME=/home/fangw/bigData/hadoop-3.1.1
    export HIVE_HOME=/home/fangw/bigData/hive-3.1.1
    export HIVE_CONF_DIR=${HIVE_HOME}/conf

4.3 将准备好的ojdbc8.jar包通过WinSCP复制到sqoop的lib目录下

4.4 验证sqoop是否可用

sqoop-version

4.5 配置以上所有的环境变量

vim /etc/profile

export JAVA_HOME=/home/fangw/jdk1.8.0_191
    export HADOOP_HOME=/home/fangw/bigData/hadoop-3.1.1
    export HBASE_HOME=/home/fangw/bigData/hbase-2.2.0
    export HIVE_HOME=/home/fangw/bigData/hive-3.1.1
    export SQOOP_HOME=/home/fangw/bigData/sqoop-1.4.7
    export FLUME_HOME=/home/fangw/bigData/flume-1.9.0
    export ZOOKEEPER_HOME=/home/fangw/bigData/zookeeper-3.5.5
    export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$FLUME_HOME/bin
    export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
    export JRE_HOME=/home/fangw/jdk1.8.0_191/jre
    export SPARK_HOME=/home/fangw/bigData/spark-2.4.3
    export PATH=$PATH:$SPARK_HOME/bin
    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
    export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
    export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*

source /etc/profile

### 5.搭建Sqoop过程中遇到的问题 ###

5.1 错误一

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
    
    Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
    <property>
      <name>yarn.app.mapreduce.am.env</name>
      <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
    </property>
    <property>
      <name>mapreduce.map.env</name>
      <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
    </property>
    <property>
      <name>mapreduce.reduce.env</name>
      <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
    </property>

解决办法：找到/home/fangw/bigData/hadoop-3.1.1/etc/hadoop/mapred-site.xml，加上如下配置

<property>
         <name>yarn.app.mapreduce.am.env</name>
         <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
       </property>
       <property>
         <name>mapreduce.map.env</name>
         <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
       </property>
       <property>
         <name>mapreduce.reduce.env</name>
         <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
       </property>

5.2 错误二

2019-07-11 10:52:00,517 INFO hive.HiveImport: Loading uploaded data into Hive
    2019-07-11 10:52:00,519 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
    2019-07-11 10:52:00,519 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

解决办法：添加环境变量HADOOP\_CLASSPATH

vim /etc/profile

加上

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*

source /etc/profile

### 6.Sqoop数据迁移测试：Oracle表迁移到Hive ###

6.1 列出oracle中的数据库

sqoop list-databases --connect jdbc:oracle:thin:@*******:1521:fangweidbt --username **** --password ****

结果1：

6.2 列出oracle的数据表

sqoop list-tables --connect jdbc:oracle:thin:@*******:1521:fangweidbt --username ****** --password ****

结果2：

6.3 oracle数据表导入Hive

sqoop import --connect jdbc:oracle:thin:@*********:1521:fangweidbt --username **** --password **** --table FANGWEI.SYS_LOG -m 1 --hive-import --hive-database fangweim --hive-table SYS_LOG

结果：

Warning: /home/fangw/bigData/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
    Please set $HCAT_HOME to the root of your HCatalog installation.
    Warning: /home/fangw/bigData/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
    Please set $ACCUMULO_HOME to the root of your Accumulo installation.
    Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/home/fangw/bigData/hadoop-3.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/home/fangw/bigData/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
    2019-07-11 15:26:00,172 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
    2019-07-11 15:26:00,192 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
    2019-07-11 15:26:00,192 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
    2019-07-11 15:26:00,192 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
    2019-07-11 15:26:00,254 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
    2019-07-11 15:26:00,272 INFO manager.SqlManager: Using default fetchSize of 1000
    2019-07-11 15:26:00,275 INFO tool.CodeGenTool: Beginning code generation
    2019-07-11 15:26:00,840 INFO manager.OracleManager: Time zone has been set to GMT
    2019-07-11 15:26:00,863 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM FANGWEI.SYS_LOG t WHERE 1=0
    2019-07-11 15:26:00,898 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/fangw/bigData/hadoop-3.1.1
    Note: /tmp/sqoop-fangw/compile/e674a2983dfb84a00bea16999a5ff9d8/FANGWEI_SYS_LOG.java uses or overrides a deprecated API.
    Note: Recompile with -Xlint:deprecation for details.
    2019-07-11 15:26:01,822 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-fangw/compile/e674a2983dfb84a00bea16999a5ff9d8/FANGWEI.SYS_LOG.jar
    2019-07-11 15:26:01,845 INFO manager.OracleManager: Time zone has been set to GMT
    2019-07-11 15:26:01,862 INFO manager.OracleManager: Time zone has been set to GMT
    2019-07-11 15:26:01,869 INFO mapreduce.ImportJobBase: Beginning import of FANGWEI.SYS_LOG
    2019-07-11 15:26:01,869 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
    2019-07-11 15:26:01,976 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
    2019-07-11 15:26:01,983 INFO manager.OracleManager: Time zone has been set to GMT
    2019-07-11 15:26:02,563 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    2019-07-11 15:26:02,610 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
    2019-07-11 15:26:02,809 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/fangw/.staging/job_1562813433780_0002
    2019-07-11 15:26:05,704 INFO db.DBInputFormat: Using read commited transaction isolation
    2019-07-11 15:26:05,725 INFO mapreduce.JobSubmitter: number of splits:1
    2019-07-11 15:26:05,747 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
    2019-07-11 15:26:06,202 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1562813433780_0002
    2019-07-11 15:26:06,204 INFO mapreduce.JobSubmitter: Executing with tokens: []
    2019-07-11 15:26:06,324 INFO conf.Configuration: resource-types.xml not found
    2019-07-11 15:26:06,324 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
    2019-07-11 15:26:06,361 INFO impl.YarnClientImpl: Submitted application application_1562813433780_0002
    2019-07-11 15:26:06,384 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1562813433780_0002/
    2019-07-11 15:26:06,384 INFO mapreduce.Job: Running job: job_1562813433780_0002
    2019-07-11 15:26:10,440 INFO mapreduce.Job: Job job_1562813433780_0002 running in uber mode : false
    2019-07-11 15:26:10,441 INFO mapreduce.Job:  map 0% reduce 0%
    2019-07-11 15:26:17,482 INFO mapreduce.Job:  map 100% reduce 0%
    2019-07-11 15:26:18,489 INFO mapreduce.Job: Job job_1562813433780_0002 completed successfully
    2019-07-11 15:26:18,549 INFO mapreduce.Job: Counters: 32
    	File System Counters
    		FILE: Number of bytes read=0
    		FILE: Number of bytes written=224052
    		FILE: Number of read operations=0
    		FILE: Number of large read operations=0
    		FILE: Number of write operations=0
    		HDFS: Number of bytes read=87
    		HDFS: Number of bytes written=16019599
    		HDFS: Number of read operations=6
    		HDFS: Number of large read operations=0
    		HDFS: Number of write operations=2
    	Job Counters 
    		Launched map tasks=1
    		Other local map tasks=1
    		Total time spent by all maps in occupied slots (ms)=4925
    		Total time spent by all reduces in occupied slots (ms)=0
    		Total time spent by all map tasks (ms)=4925
    		Total vcore-milliseconds taken by all map tasks=4925
    		Total megabyte-milliseconds taken by all map tasks=5043200
    	Map-Reduce Framework
    		Map input records=43991
    		Map output records=43991
    		Input split bytes=87
    		Spilled Records=0
    		Failed Shuffles=0
    		Merged Map outputs=0
    		GC time elapsed (ms)=86
    		CPU time spent (ms)=4020
    		Physical memory (bytes) snapshot=399536128
    		Virtual memory (bytes) snapshot=2838581248
    		Total committed heap usage (bytes)=429391872
    		Peak Map Physical memory (bytes)=399536128
    		Peak Map Virtual memory (bytes)=2838581248
    	File Input Format Counters 
    		Bytes Read=0
    	File Output Format Counters 
    		Bytes Written=16019599
    2019-07-11 15:26:18,552 INFO mapreduce.ImportJobBase: Transferred 15.2775 MB in 15.9838 seconds (978.7508 KB/sec)
    2019-07-11 15:26:18,554 INFO mapreduce.ImportJobBase: Retrieved 43991 records.
    2019-07-11 15:26:18,554 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table FANGWEI.SYS_LOG
    2019-07-11 15:26:18,570 INFO manager.OracleManager: Time zone has been set to GMT
    2019-07-11 15:26:18,570 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM FANGWEI.SYS_LOG t WHERE 1=0
    2019-07-11 15:26:18,582 WARN hive.TableDefWriter: Column CREATE_DATE had to be cast to a less precise type in Hive
    2019-07-11 15:26:18,584 INFO hive.HiveImport: Loading uploaded data into Hive
    2019-07-11 15:26:18,605 INFO conf.HiveConf: Found configuration file file:/home/fangw/bigData/hive-3.1.1/conf/hive-site.xml
    2019-07-11 07:26:20,422 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")
    	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    	at java.lang.SecurityManager.checkPermission(SecurityManager.java:585)
    	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848)
    	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322)
    	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    	at org.apache.logging.log4j.core.jmx.Server.register(Server.java:389)
    	at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167)
    	at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:140)
    	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:556)
    	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:261)
    	at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:87)
    	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:240)
    	at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:158)
    	at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:131)
    	at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:101)
    	at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:188)
    	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:173)
    	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:106)
    	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:98)
    	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
    	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
    	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:331)
    	at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
    	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
    	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
    
    Hive Session ID = 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:20,475 INFO SessionState: Hive Session ID = 3bbe26fe-3f5b-485e-a329-27131bed762a
    
    Logging initialized using configuration in jar:file:/home/fangw/bigData/hive-3.1.1/lib/hive-common-3.1.1.jar!/hive-log4j2.properties Async: true
    2019-07-11 15:26:20,539 INFO SessionState: 
    Logging initialized using configuration in jar:file:/home/fangw/bigData/hive-3.1.1/lib/hive-common-3.1.1.jar!/hive-log4j2.properties Async: true
    2019-07-11 15:26:20,564 INFO session.SessionState: Created HDFS directory: /tmp/hive/fangw/3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:20,596 INFO session.SessionState: Created local directory: /tmp/fangw/3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:20,597 INFO session.SessionState: Created HDFS directory: /tmp/hive/fangw/3bbe26fe-3f5b-485e-a329-27131bed762a/_tmp_space.db
    2019-07-11 15:26:20,600 INFO conf.HiveConf: Using the default value passed in for log id: 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:20,600 INFO session.SessionState: Updating thread name to 3bbe26fe-3f5b-485e-a329-27131bed762a main
    2019-07-11 15:26:22,321 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:22,356 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:22,360 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:22,362 INFO conf.MetastoreConf: Found configuration file file:/home/fangw/bigData/hive-3.1.1/conf/hive-site.xml
    2019-07-11 15:26:22,363 INFO conf.MetastoreConf: Unable to find config file hivemetastore-site.xml
    2019-07-11 15:26:22,363 INFO conf.MetastoreConf: Found configuration file null
    2019-07-11 15:26:22,364 INFO conf.MetastoreConf: Unable to find config file metastore-site.xml
    2019-07-11 15:26:22,364 INFO conf.MetastoreConf: Found configuration file null
    2019-07-11 15:26:22,596 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
    2019-07-11 15:26:22,846 INFO hikari.HikariDataSource: HikariPool-1 - Starting...
    2019-07-11 15:26:22,911 INFO hikari.HikariDataSource: HikariPool-1 - Start completed.
    2019-07-11 15:26:23,010 INFO hikari.HikariDataSource: HikariPool-2 - Starting...
    2019-07-11 15:26:23,070 INFO hikari.HikariDataSource: HikariPool-2 - Start completed.
    2019-07-11 15:26:23,197 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
    2019-07-11 15:26:23,381 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:23,383 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:23,673 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:23,673 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:23,674 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:23,674 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:23,674 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:23,674 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:25,193 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:25,193 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:25,193 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:25,194 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:25,194 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:25,194 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored
    2019-07-11 15:26:27,537 INFO metastore.HiveMetaStore: Added admin role in metastore
    2019-07-11 15:26:27,542 INFO metastore.HiveMetaStore: Added public role in metastore
    2019-07-11 15:26:27,585 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
    2019-07-11 15:26:27,965 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:27,989 INFO metastore.HiveMetaStore: 0: get_all_functions
    2019-07-11 15:26:27,992 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_all_functions	
    Hive Session ID = 571f954f-914e-458b-bd72-b3c42269a2f0
    2019-07-11 15:26:28,046 INFO SessionState: Hive Session ID = 571f954f-914e-458b-bd72-b3c42269a2f0
    2019-07-11 15:26:28,047 INFO conf.HiveConf: Using the default value passed in for log id: 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:28,053 INFO session.SessionState: Created HDFS directory: /tmp/hive/fangw/571f954f-914e-458b-bd72-b3c42269a2f0
    2019-07-11 15:26:28,054 INFO session.SessionState: Created local directory: /tmp/fangw/571f954f-914e-458b-bd72-b3c42269a2f0
    2019-07-11 15:26:28,056 INFO session.SessionState: Created HDFS directory: /tmp/hive/fangw/571f954f-914e-458b-bd72-b3c42269a2f0/_tmp_space.db
    2019-07-11 15:26:28,057 INFO metastore.HiveMetaStore: 1: get_databases: @hive#
    2019-07-11 15:26:28,057 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_databases: @hive#	
    2019-07-11 15:26:28,058 INFO metastore.HiveMetaStore: 1: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:28,059 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:28,066 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:28,067 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:28,079 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW
    2019-07-11 15:26:28,079 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW	
    2019-07-11 15:26:28,092 INFO metastore.HiveMetaStore: 1: get_multi_table : db=default tbls=
    2019-07-11 15:26:28,092 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_multi_table : db=default tbls=	
    2019-07-11 15:26:28,093 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#fangweim pat=.*,type=MATERIALIZED_VIEW
    2019-07-11 15:26:28,093 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_tables_by_type: db=@hive#fangweim pat=.*,type=MATERIALIZED_VIEW	
    2019-07-11 15:26:28,097 INFO metastore.HiveMetaStore: 1: get_multi_table : db=fangweim tbls=
    2019-07-11 15:26:28,097 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_multi_table : db=fangweim tbls=	
    2019-07-11 15:26:28,097 INFO metadata.HiveMaterializedViewsRegistry: Materialized views registry has been initialized
    2019-07-11 15:26:28,202 INFO ql.Driver: Compiling command(queryId=fangw_20190711072628_de0d76e7-1e88-4bc0-8129-e4618f90685d): CREATE TABLE IF NOT EXISTS `fangweim`.`SYS_LOG` ( `ID` STRING, `TYPE` STRING, `TITLE` STRING, `CREATE_BY` STRING, `CREATE_DATE` STRING, `REMOTE_ADDR` STRING, `USER_AGENT` STRING, `REQUEST_URI` STRING, `METHOD` STRING, `PARAMS` STRING, `EXCEPTION` STRING) COMMENT 'Imported by sqoop on 2019/07/11 07:26:18' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE
    2019-07-11 15:26:29,399 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
    2019-07-11 15:26:29,406 INFO parse.CalcitePlanner: Starting Semantic Analysis
    2019-07-11 15:26:29,437 INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=3bbe26fe-3f5b-485e-a329-27131bed762a, clientType=HIVECLI]
    2019-07-11 15:26:29,443 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
    2019-07-11 15:26:29,444 INFO metastore.HiveMetaStoreClient: Mestastore configuration metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
    2019-07-11 15:26:29,446 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore...
    2019-07-11 15:26:29,446 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Cleaning up thread local RawStore...	
    2019-07-11 15:26:29,446 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
    2019-07-11 15:26:29,446 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Done cleaning up thread local RawStore	
    2019-07-11 15:26:29,448 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:29,448 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:29,448 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:29,456 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:29,456 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:29,456 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:29,477 INFO parse.CalcitePlanner: Creating table fangweim.SYS_LOG position=27
    2019-07-11 15:26:29,512 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:29,512 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:29,516 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:29,516 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:29,517 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:29,521 INFO metastore.HiveMetaStore: 0: get_table : tbl=hive.fangweim.SYS_LOG
    2019-07-11 15:26:29,521 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_table : tbl=hive.fangweim.SYS_LOG	
    2019-07-11 15:26:29,591 INFO metastore.HiveMetaStore: 0: get_database: @hive#fangweim
    2019-07-11 15:26:29,591 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_database: @hive#fangweim	
    2019-07-11 15:26:29,641 INFO ql.Driver: Semantic Analysis Completed (retrial = false)
    2019-07-11 15:26:29,659 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
    2019-07-11 15:26:29,670 INFO ql.Driver: Completed compiling command(queryId=fangw_20190711072628_de0d76e7-1e88-4bc0-8129-e4618f90685d); Time taken: 1.542 seconds
    2019-07-11 15:26:29,670 INFO reexec.ReExecDriver: Execution #1 of query
    2019-07-11 15:26:29,672 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
    2019-07-11 15:26:29,672 INFO ql.Driver: Executing command(queryId=fangw_20190711072628_de0d76e7-1e88-4bc0-8129-e4618f90685d): CREATE TABLE IF NOT EXISTS `fangweim`.`SYS_LOG` ( `ID` STRING, `TYPE` STRING, `TITLE` STRING, `CREATE_BY` STRING, `CREATE_DATE` STRING, `REMOTE_ADDR` STRING, `USER_AGENT` STRING, `REQUEST_URI` STRING, `METHOD` STRING, `PARAMS` STRING, `EXCEPTION` STRING) COMMENT 'Imported by sqoop on 2019/07/11 07:26:18' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE
    2019-07-11 15:26:29,696 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
    2019-07-11 15:26:29,698 INFO metastore.HiveMetaStoreClient: Mestastore configuration metastore.filter.hook changed from org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl
    2019-07-11 15:26:29,698 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore...
    2019-07-11 15:26:29,698 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Cleaning up thread local RawStore...	
    2019-07-11 15:26:29,698 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
    2019-07-11 15:26:29,698 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Done cleaning up thread local RawStore	
    2019-07-11 15:26:29,878 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:29,878 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:29,878 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:29,883 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:29,883 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:29,883 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:29,884 INFO metastore.HiveMetaStore: 0: create_table: Table(tableName:SYS_LOG, dbName:fangweim, owner:fangw, createTime:1562829989, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id, type:string, comment:null), FieldSchema(name:type, type:string, comment:null), FieldSchema(name:title, type:string, comment:null), FieldSchema(name:create_by, type:string, comment:null), FieldSchema(name:create_date, type:string, comment:null), FieldSchema(name:remote_addr, type:string, comment:null), FieldSchema(name:user_agent, type:string, comment:null), FieldSchema(name:request_uri, type:string, comment:null), FieldSchema(name:method, type:string, comment:null), FieldSchema(name:params, type:string, comment:null), FieldSchema(name:exception, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{line.delim=
    , field.delim=, serialization.format=}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, numRows=0, rawDataSize=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"create_by":"true","create_date":"true","exception":"true","id":"true","method":"true","params":"true","remote_addr":"true","request_uri":"true","title":"true","type":"true","user_agent":"true"}}, numFiles=0, bucketing_version=2, comment=Imported by sqoop on 2019/07/11 07:26:18}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{fangw=[PrivilegeGrantInfo(privilege:INSERT, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:SELECT, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:UPDATE, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:DELETE, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true)]}, groupPrivileges:null, rolePrivileges:null), temporary:false, catName:hive, ownerType:USER)
    2019-07-11 15:26:29,884 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=create_table: Table(tableName:SYS_LOG, dbName:fangweim, owner:fangw, createTime:1562829989, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id, type:string, comment:null), FieldSchema(name:type, type:string, comment:null), FieldSchema(name:title, type:string, comment:null), FieldSchema(name:create_by, type:string, comment:null), FieldSchema(name:create_date, type:string, comment:null), FieldSchema(name:remote_addr, type:string, comment:null), FieldSchema(name:user_agent, type:string, comment:null), FieldSchema(name:request_uri, type:string, comment:null), FieldSchema(name:method, type:string, comment:null), FieldSchema(name:params, type:string, comment:null), FieldSchema(name:exception, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{line.delim=
    , field.delim=, serialization.format=}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, numRows=0, rawDataSize=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"create_by":"true","create_date":"true","exception":"true","id":"true","method":"true","params":"true","remote_addr":"true","request_uri":"true","title":"true","type":"true","user_agent":"true"}}, numFiles=0, bucketing_version=2, comment=Imported by sqoop on 2019/07/11 07:26:18}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{fangw=[PrivilegeGrantInfo(privilege:INSERT, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:SELECT, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:UPDATE, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:DELETE, createTime:-1, grantor:fangw, grantorType:USER, grantOption:true)]}, groupPrivileges:null, rolePrivileges:null), temporary:false, catName:hive, ownerType:USER)	
    2019-07-11 15:26:29,899 INFO utils.FileUtils: Creating directory if it doesn't exist: hdfs://fangweima-weixin02:9000/user/hive/warehouse/fangweim.db/sys_log
    2019-07-11 15:26:30,449 INFO ql.Driver: Completed executing command(queryId=fangw_20190711072628_de0d76e7-1e88-4bc0-8129-e4618f90685d); Time taken: 0.777 seconds
    OK
    2019-07-11 15:26:30,449 INFO ql.Driver: OK
    2019-07-11 15:26:30,449 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
    Time taken: 2.324 seconds
    2019-07-11 15:26:30,451 INFO CliDriver: Time taken: 2.324 seconds
    2019-07-11 15:26:30,451 INFO conf.HiveConf: Using the default value passed in for log id: 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:30,451 INFO session.SessionState: Resetting thread name to  main
    2019-07-11 15:26:30,451 INFO conf.HiveConf: Using the default value passed in for log id: 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:30,451 INFO session.SessionState: Updating thread name to 3bbe26fe-3f5b-485e-a329-27131bed762a main
    2019-07-11 15:26:30,453 INFO ql.Driver: Compiling command(queryId=fangw_20190711072630_441fe37c-1cdf-4178-bbe4-28d39f80917e): 
    LOAD DATA INPATH 'hdfs://fangweima-weixin02:9000/user/fangw/FANGWEI.SYS_LOG' INTO TABLE `fangweim`.`SYS_LOG`
    2019-07-11 15:26:30,454 INFO metastore.HiveMetaStoreClient: Mestastore configuration metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
    2019-07-11 15:26:30,455 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore...
    2019-07-11 15:26:30,455 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Cleaning up thread local RawStore...	
    2019-07-11 15:26:30,455 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
    2019-07-11 15:26:30,455 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Done cleaning up thread local RawStore	
    2019-07-11 15:26:30,455 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
    2019-07-11 15:26:30,462 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:30,462 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:30,462 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:30,466 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:30,466 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:30,466 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:30,466 INFO metastore.HiveMetaStore: 0: get_table : tbl=hive.fangweim.SYS_LOG
    2019-07-11 15:26:30,466 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_table : tbl=hive.fangweim.SYS_LOG	
    2019-07-11 15:26:30,756 INFO ql.Driver: Semantic Analysis Completed (retrial = false)
    2019-07-11 15:26:30,757 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
    2019-07-11 15:26:30,757 INFO ql.Driver: Completed compiling command(queryId=fangw_20190711072630_441fe37c-1cdf-4178-bbe4-28d39f80917e); Time taken: 0.304 seconds
    2019-07-11 15:26:30,757 INFO reexec.ReExecDriver: Execution #1 of query
    2019-07-11 15:26:30,757 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
    2019-07-11 15:26:30,757 INFO ql.Driver: Executing command(queryId=fangw_20190711072630_441fe37c-1cdf-4178-bbe4-28d39f80917e): 
    LOAD DATA INPATH 'hdfs://fangweima-weixin02:9000/user/fangw/FANGWEI.SYS_LOG' INTO TABLE `fangweim`.`SYS_LOG`
    2019-07-11 15:26:30,757 INFO ql.Driver: Starting task [Stage-0:MOVE] in serial mode
    2019-07-11 15:26:30,757 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore...
    2019-07-11 15:26:30,757 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Cleaning up thread local RawStore...	
    2019-07-11 15:26:30,757 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
    2019-07-11 15:26:30,757 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Done cleaning up thread local RawStore	
    Loading data to table fangweim.sys_log
    2019-07-11 15:26:30,758 INFO exec.Task: Loading data to table fangweim.sys_log from hdfs://fangweima-weixin02:9000/user/fangw/FANGWEI.SYS_LOG
    2019-07-11 15:26:30,759 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:30,759 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:30,759 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:30,763 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:30,763 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:30,764 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:30,764 INFO metastore.HiveMetaStore: 0: get_table : tbl=hive.fangweim.sys_log
    2019-07-11 15:26:30,764 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_table : tbl=hive.fangweim.sys_log	
    2019-07-11 15:26:30,835 INFO metastore.HiveMetaStore: 0: get_table : tbl=hive.fangweim.sys_log
    2019-07-11 15:26:30,835 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_table : tbl=hive.fangweim.sys_log	
    2019-07-11 15:26:30,922 INFO metastore.HiveMetaStore: 0: alter_table: hive.fangweim.sys_log newtbl=sys_log
    2019-07-11 15:26:30,922 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=alter_table: hive.fangweim.sys_log newtbl=sys_log	
    2019-07-11 15:26:31,079 INFO ql.Driver: Starting task [Stage-1:STATS] in serial mode
    2019-07-11 15:26:31,079 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore...
    2019-07-11 15:26:31,079 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Cleaning up thread local RawStore...	
    2019-07-11 15:26:31,079 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
    2019-07-11 15:26:31,079 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Done cleaning up thread local RawStore	
    2019-07-11 15:26:31,079 INFO stats.BasicStatsTask: Executing stats task
    2019-07-11 15:26:31,081 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
    2019-07-11 15:26:31,081 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
    2019-07-11 15:26:31,081 INFO metastore.ObjectStore: ObjectStore, initialize called
    2019-07-11 15:26:31,086 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is ORACLE
    2019-07-11 15:26:31,086 INFO metastore.ObjectStore: Initialized ObjectStore
    2019-07-11 15:26:31,087 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=fangw (auth:SIMPLE) retries=1 delay=1 lifetime=0
    2019-07-11 15:26:31,087 INFO metastore.HiveMetaStore: 0: get_table : tbl=hive.fangweim.sys_log
    2019-07-11 15:26:31,087 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=get_table : tbl=hive.fangweim.sys_log	
    2019-07-11 15:26:31,142 INFO metastore.HiveMetaStore: 0: alter_table: hive.fangweim.sys_log newtbl=sys_log
    2019-07-11 15:26:31,142 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=alter_table: hive.fangweim.sys_log newtbl=sys_log	
    2019-07-11 15:26:31,273 INFO stats.BasicStatsTask: Table fangweim.sys_log stats: [numFiles=1, numRows=0, totalSize=16019599, rawDataSize=0]
    2019-07-11 15:26:31,273 INFO ql.Driver: Completed executing command(queryId=fangw_20190711072630_441fe37c-1cdf-4178-bbe4-28d39f80917e); Time taken: 0.516 seconds
    OK
    2019-07-11 15:26:31,273 INFO ql.Driver: OK
    2019-07-11 15:26:31,273 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
    Time taken: 0.821 seconds
    2019-07-11 15:26:31,273 INFO CliDriver: Time taken: 0.821 seconds
    2019-07-11 15:26:31,273 INFO conf.HiveConf: Using the default value passed in for log id: 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:31,273 INFO session.SessionState: Resetting thread name to  main
    2019-07-11 15:26:31,274 INFO conf.HiveConf: Using the default value passed in for log id: 3bbe26fe-3f5b-485e-a329-27131bed762a
    2019-07-11 15:26:31,280 INFO session.SessionState: Deleted directory: /tmp/hive/fangw/3bbe26fe-3f5b-485e-a329-27131bed762a on fs with scheme hdfs
    2019-07-11 15:26:31,280 INFO session.SessionState: Deleted directory: /tmp/fangw/3bbe26fe-3f5b-485e-a329-27131bed762a on fs with scheme file
    2019-07-11 15:26:31,291 INFO metastore.HiveMetaStore: 0: Cleaning up thread local RawStore...
    2019-07-11 15:26:31,291 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Cleaning up thread local RawStore...	
    2019-07-11 15:26:31,291 INFO metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
    2019-07-11 15:26:31,291 INFO HiveMetaStore.audit: ugi=fangw	ip=unknown-ip-addr	cmd=Done cleaning up thread local RawStore	
    2019-07-11 15:26:31,291 INFO hive.HiveImport: Hive import complete.
    2019-07-11 15:26:31,292 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.

使用Hive命令查询结果

hive> show tables;
    OK
    sys_log
    Time taken: 0.03 seconds, Fetched: 1 row(s)

数据迁移成功！

[Link 1]: https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.5/
[Link 2]: http://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.2.0/hbase-2.2.0-bin.tar.gz
[Link 3]: https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.7/
[Link 4]: https://download.csdn.net/download/linhaiyun_ytdx/11302365