HBase数据的导入导出

小咪咪 2022-08-05 00:57 1190阅读 0赞

HBase数据的导入导出

1、导出：  
hbase org.apache.hadoop.hbase.mapreduce.Driver export 表名 导出存放路径  
  
其中数据文件位置可为本地文件目录，也可以分布式文件系统hdfs的路径。  
当其为前者时，直接指定即可，也可以加前缀file:///  
而当其为后者时，必须明确指明hdfs的路径，例如hdfs://192.168.1.200:9000/path  
  
  
2、导入：  
hbase org.apache.hadoop.hbase.mapreduce.Driver import 表名 要导入的文件路径  
同上，其中数据文件位置可为本地文件目录，也可以分布式文件系统hdfs的路径。  
另外，该接口类还提供了一些其它的方法，例如表与表之间的数据拷贝，导入tsv文件等，可回车键查看  
  
  
  
Import操作必须是使用Export出的数据才可以，要不然会报错 not a SequenceFile。  
  
  
  
  
3、本文用的示例：  
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export waln\_log /usr/local/waln\_log  
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver import waln\_log1 hdfs://192.168.1.200:9000/usr/local/waln\_log/part-m-00000  
  
  
3.1、首先查看HBase的表waln\_log数据，一会就把这个表数据导出来：  
hbase(main):009:0> scan 'waln\_log'  
ROW                                COLUMN+CELL  
 row1                              column=cf:age, timestamp=1432740300560, value=22  
 row1                              column=cf:city, timestamp=1432740308281, value=shanghai  
 row1                              column=cf:name, timestamp=1432740263412, value=zhangsan  
 row2                              column=cf:name, timestamp=1432740296373, value=lisi  
2 row(s) in 12.0670 seconds  
  
  
hbase(main):010:0> desc 'waln\_log'  
Table waln\_log is ENABLED  
COLUMN FAMILIES DESCRIPTION  
\{NAME => 'cf', DATA\_BLOCK\_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION\_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',  
MIN\_VERSIONS => '0', TTL => 'FOREVER', KEEP\_DELETED\_CELLS => 'FALSE', BLOCKSIZE => '65536', IN\_MEMORY => 'false', BLOCKCACHE => 'true'  
\}  
1 row(s) in 1.1220 seconds  
  
  
hbase(main):011:0>  
  
  
  
  
3.2、开始执行：  
bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export waln\_log /usr/local/waln\_log  
  
  
  
执行完之后，生成的HDFS目录：  
\[root@baozi hbase-0.99.2\]\# hdfs dfs -ls /usr/local/waln\_log  
Found 2 items  
\-rw-r--r--   1 root supergroup          0 2015-05-27 23:34 /usr/local/waln\_log/\_SUCCESS  
\-rw-r--r--   1 root supergroup        289 2015-05-27 23:33 /usr/local/waln\_log/part-m-00000  
  
  
  
  
执行部分过程，会跑一个MR：  
2015-05-27 23:27:56,261 INFO  \[main\] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib/native  
2015-05-27 23:27:56,261 INFO  \[main\] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp  
2015-05-27 23:27:56,261 INFO  \[main\] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>  
2015-05-27 23:27:56,262 INFO  \[main\] zookeeper.ZooKeeper: Client environment:os.name=Linux  
2015-05-27 23:27:56,262 INFO  \[main\] zookeeper.ZooKeeper: Client environment:os.arch=amd64  
2015-05-27 23:27:56,262 INFO  \[main\] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86\_64  
2015-05-27 23:27:56,262 INFO  \[main\] zookeeper.ZooKeeper: Client environment:user.name=root  
2015-05-27 23:27:56,262 INFO  \[main\] zookeeper.ZooKeeper: Client environment:user.home=/root  
2015-05-27 23:27:56,262 INFO  \[main\] zookeeper.ZooKeeper: Client environment:user.dir=/usr/local/hbase-0.99.2  
2015-05-27 23:27:56,264 INFO  \[main\] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.1.200:2181 sessionTimeout=90000 watcher=hconnection-0x5629409d, quorum=192.168.1.200:2181, baseZNode=/hbase  
2015-05-27 23:27:56,542 INFO  \[main-SendThread(192.168.1.200:2181)\] zookeeper.ClientCnxn: Opening socket connection to server 192.168.1.200/192.168.1.200:2181. Will not attempt to authenticate using SASL (unknown error)  
2015-05-27 23:27:56,544 INFO  \[main-SendThread(192.168.1.200:2181)\] zookeeper.ClientCnxn: Socket connection established to 192.168.1.200/192.168.1.200:2181, initiating session  
2015-05-27 23:27:56,989 INFO  \[main-SendThread(192.168.1.200:2181)\] zookeeper.ClientCnxn: Session establishment complete on server 192.168.1.200/192.168.1.200:2181, sessionid = 0x14d95f3ad100012, negotiated timeout = 90000  
2015-05-27 23:27:57,415 INFO  \[main\] util.RegionSizeCalculator: Calculating region sizes for table "waln\_log".  
2015-05-27 23:28:13,737 INFO  \[main\] mapreduce.JobSubmitter: number of splits:1  
2015-05-27 23:28:13,920 INFO  \[main\] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum  
2015-05-27 23:28:15,913 INFO  \[main\] mapreduce.JobSubmitter: Submitting tokens for job: job\_1432735450462\_0004  
2015-05-27 23:28:24,928 INFO  \[main\] impl.YarnClientImpl: Submitted application application\_1432735450462\_0004  
2015-05-27 23:28:25,448 INFO  \[main\] mapreduce.Job: The url to track the job: http://baozi:8088/proxy/application\_1432735450462\_0004/  
2015-05-27 23:28:25,449 INFO  \[main\] mapreduce.Job: Running job: job\_1432735450462\_0004  
2015-05-27 23:31:26,790 INFO  \[main\] mapreduce.Job: Job job\_1432735450462\_0004 running in uber mode : false  
2015-05-27 23:31:32,444 INFO  \[main\] mapreduce.Job:  map 0% reduce 0%  
2015-05-27 23:34:02,361 INFO  \[main\] mapreduce.Job:  map 100% reduce 0%  
2015-05-27 23:34:53,248 INFO  \[main\] mapreduce.Job: Job job\_1432735450462\_0004 completed successfully  
2015-05-27 23:35:22,330 INFO  \[main\] mapreduce.Job: Counters: 41  
        File System Counters  
                FILE: Number of bytes read=0  
                FILE: Number of bytes written=133771  
                FILE: Number of read operations=0  
                FILE: Number of large read operations=0  
                FILE: Number of write operations=0  
                HDFS: Number of bytes read=65  
                HDFS: Number of bytes written=289  
                HDFS: Number of read operations=4  
                HDFS: Number of large read operations=0  
                HDFS: Number of write operations=2  
        Job Counters  
                Launched map tasks=1  
                Data-local map tasks=1  
                Total time spent by all maps in occupied slots (ms)=169690  
                Total time spent by all reduces in occupied slots (ms)=0  
                Total time spent by all map tasks (ms)=169690  
                Total vcore-seconds taken by all map tasks=169690  
                Total megabyte-seconds taken by all map tasks=173762560  
        Map-Reduce Framework  
                Map input records=2  
                Map output records=2  
                Input split bytes=65  
                Spilled Records=0  
                Failed Shuffles=0  
                Merged Map outputs=0  
                GC time elapsed (ms)=2879  
                CPU time spent (ms)=3350  
                Physical memory (bytes) snapshot=85295104  
                Virtual memory (bytes) snapshot=848232448  
                Total committed heap usage (bytes)=15859712  
        HBase Counters  
                BYTES\_IN\_REMOTE\_RESULTS=0  
                BYTES\_IN\_RESULTS=157  
                MILLIS\_BETWEEN\_NEXTS=13240  
                NOT\_SERVING\_REGION\_EXCEPTION=0  
                NUM\_SCANNER\_RESTARTS=0  
                NUM\_SCAN\_RESULTS\_STALE=0  
                REGIONS\_SCANNED=1  
                REMOTE\_RPC\_CALLS=0  
                REMOTE\_RPC\_RETRIES=0  
                RPC\_CALLS=3  
                RPC\_RETRIES=0  
        File Input Format Counters  
                Bytes Read=0  
        File Output Format Counters  
                Bytes Written=289  
2015-05-27 23:35:25,049 INFO  \[main\] ipc.Client: Retrying connect to server: baozi/192.168.1.200:36055. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)  
2015-05-27 23:35:26,181 INFO  \[main\] ipc.Client: Retrying connect to server: baozi/192.168.1.200:36055. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)  
2015-05-27 23:35:27,182 INFO  \[main\] ipc.Client: Retrying connect to server: baozi/192.168.1.200:36055. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)  
2015-05-27 23:35:30,592 INFO  \[main\] mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
3.3、为导入数据创建一个表：  
hbase(main):011:0> create 'waln\_log1','cf'  
0 row(s) in 30.2070 seconds  
  
  
=> Hbase::Table - waln\_log1  
hbase(main):012:0> scan 'waln\_log'  
ROW                                COLUMN+CELL  
 row1                              column=cf:age, timestamp=1432740300560, value=22  
 row1                              column=cf:city, timestamp=1432740308281, value=shanghai  
 row1                              column=cf:name, timestamp=1432740263412, value=zhangsan  
 row2                              column=cf:name, timestamp=1432740296373, value=lisi  
2 row(s) in 1.3960 seconds  
  
  
hbase(main):013:0> scan 'waln\_log1'  
ROW                                COLUMN+CELL  
0 row(s) in 0.2750 seconds  
  
  
hbase(main):014:0> desc 'waln\_log1'  
Table waln\_log1 is ENABLED  
COLUMN FAMILIES DESCRIPTION  
\{NAME => 'cf', DATA\_BLOCK\_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION\_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',  
MIN\_VERSIONS => '0', TTL => 'FOREVER', KEEP\_DELETED\_CELLS => 'FALSE', BLOCKSIZE => '65536', IN\_MEMORY => 'false', BLOCKCACHE => 'true'  
\}  
1 row(s) in 0.1500 seconds  
  
  
hbase(main):015:0>  
  
  
3.4、执行导入命令：  
  
  
\[root@baozi hbase-0.99.2\]\# bin/hbase org.apache.hadoop.hbase.mapreduce.Driver import waln\_log1 hdfs://192.168.1.200:9000/usr/local/waln\_log/part-m-00000  
  
  
  
  
会跑MR，部分过程：  
2015-05-27 23:46:27,426 INFO  \[main\] mapreduce.TableOutputFormat: Created table instance for waln\_log1  
2015-05-27 23:46:43,436 INFO  \[main\] input.FileInputFormat: Total input paths to process : 1  
2015-05-27 23:46:45,387 INFO  \[main\] mapreduce.JobSubmitter: number of splits:1  
2015-05-27 23:46:47,327 INFO  \[main\] mapreduce.JobSubmitter: Submitting tokens for job: job\_1432735450462\_0005  
2015-05-27 23:46:52,916 INFO  \[main\] impl.YarnClientImpl: Submitted application application\_1432735450462\_0005  
2015-05-27 23:46:53,233 INFO  \[main\] mapreduce.Job: The url to track the job: http://baozi:8088/proxy/application\_1432735450462\_0005/  
2015-05-27 23:46:53,251 INFO  \[main\] mapreduce.Job: Running job: job\_1432735450462\_0005  
2015-05-27 23:48:52,937 INFO  \[main\] mapreduce.Job: Job job\_1432735450462\_0005 running in uber mode : false  
2015-05-27 23:48:54,258 INFO  \[main\] mapreduce.Job:  map 0% reduce 0%  
2015-05-27 23:51:32,098 INFO  \[main\] mapreduce.Job:  map 100% reduce 0%  
2015-05-27 23:52:50,001 INFO  \[main\] mapreduce.Job: Job job\_1432735450462\_0005 completed successfully  
2015-05-27 23:52:54,965 INFO  \[main\] mapreduce.Job: Counters: 30  
        File System Counters  
                FILE: Number of bytes read=0  
                FILE: Number of bytes written=133322  
                FILE: Number of read operations=0  
                FILE: Number of large read operations=0  
                FILE: Number of write operations=0  
                HDFS: Number of bytes read=411  
                HDFS: Number of bytes written=0  
                HDFS: Number of read operations=3  
                HDFS: Number of large read operations=0  
                HDFS: Number of write operations=0  
        Job Counters  
                Launched map tasks=1  
                Data-local map tasks=1  
                Total time spent by all maps in occupied slots (ms)=217016  
                Total time spent by all reduces in occupied slots (ms)=0  
                Total time spent by all map tasks (ms)=217016  
                Total vcore-seconds taken by all map tasks=217016  
                Total megabyte-seconds taken by all map tasks=222224384  
        Map-Reduce Framework  
                Map input records=2  
                Map output records=2  
                Input split bytes=122  
                Spilled Records=0  
                Failed Shuffles=0  
                Merged Map outputs=0  
                GC time elapsed (ms)=2050  
                CPU time spent (ms)=2140  
                Physical memory (bytes) snapshot=80756736  
                Virtual memory (bytes) snapshot=845209600  
                Total committed heap usage (bytes)=15859712  
        File Input Format Counters  
                Bytes Read=289  
        File Output Format Counters  
                Bytes Written=0  
  
  
  
  
  
  
数据导入成功：  
hbase(main):015:0> scan 'waln\_log1'  
ROW                                COLUMN+CELL  
 row1                              column=cf:age, timestamp=1432740300560, value=22  
 row1                              column=cf:city, timestamp=1432740308281, value=shanghai  
 row1                              column=cf:name, timestamp=1432740263412, value=zhangsan  
 row2                              column=cf:name, timestamp=1432740296373, value=lisi  
2 row(s) in 2.5040 seconds  
  
  
hbase(main):016:0>  
  
  
  
  
  
  
3.5、创建的表结构与导出数据的表结构不同，会报错：  
hbase(main):017:0> create 'waln\_log2','cf1','cf2'  
0 row(s) in 25.7630 seconds  
  
  
=> Hbase::Table - waln\_log2  
hbase(main):018:0> desc 'waln\_log2'  
Table waln\_log2 is ENABLED  
COLUMN FAMILIES DESCRIPTION  
\{NAME => 'cf1', DATA\_BLOCK\_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION\_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',  
 MIN\_VERSIONS => '0', TTL => 'FOREVER', KEEP\_DELETED\_CELLS => 'FALSE', BLOCKSIZE => '65536', IN\_MEMORY => 'false', BLOCKCACHE => 'true  
'\}  
\{NAME => 'cf2', DATA\_BLOCK\_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION\_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',  
 MIN\_VERSIONS => '0', TTL => 'FOREVER', KEEP\_DELETED\_CELLS => 'FALSE', BLOCKSIZE => '65536', IN\_MEMORY => 'false', BLOCKCACHE => 'true  
'\}  
2 row(s) in 0.2650 seconds  
  
  
hbase(main):019:0> scan 'waln\_log2'  
ROW                                COLUMN+CELL  
0 row(s) in 0.1870 seconds  
  
  
hbase(main):020:0>  
  
  
报错信息：  
2015-05-28 00:02:31,224 INFO  \[main\] mapreduce.Job:  map 0% reduce 0%  
2015-05-28 00:08:20,162 INFO  \[main\] mapreduce.Job:  map 100% reduce 0%  
2015-05-28 00:08:55,603 INFO  \[main\] mapreduce.Job:  map 0% reduce 0%  
2015-05-28 00:09:30,247 INFO  \[main\] mapreduce.Job: Task Id : attempt\_1432735450462\_0006\_m\_000000\_0, Status : FAILED

Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family cf does not exist in region waln\_log2,,1432742282179.6cd6a2a4d5ae585bd425ffbce92783c4. in table 'waln\_log2', \{NAME => 'cf1', DATA\_BLOCK\_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION\_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN\_VERSIONS => '0', KEEP\_DELETED\_CELLS => 'FALSE', BLOCKSIZE => '65536', IN\_MEMORY => 'false', BLOCKCACHE => 'true'\}, \{NAME => 'cf2', DATA\_BLOCK\_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION\_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN\_VERSIONS => '0', KEEP\_DELETED\_CELLS => 'FALSE', BLOCKSIZE => '65536', IN\_MEMORY => 'false', BLOCKCACHE => 'true'\}