HDFS的DistCp使用

àì夳堔傛蜴生んèń 2022-01-26 09:23 455阅读 0赞

**用法:**

hadoop distcp OPTIONS [source_path...] <target_path>

<table> 
 <thead> 
  <tr> 
   <th>参数</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>-append</td> 
   <td>重用目标文件中的现有数据，并在可能的情况下添加新数据</td> 
  </tr> 
  <tr> 
   <td>-atomic</td> 
   <td>提交所有更改或不提交更改</td> 
  </tr> 
  <tr> 
   <td>-bandwidth </td> 
   <td>以MB为单位指定每个映射的带宽</td> 
  </tr> 
  <tr> 
   <td>-blocksperchunk </td> 
   <td>如果将该值设置为正值，则包含比该值更多块的文件将被分割为多个块，以便并行传输，并在目标上重新组装。默认情况下，值为0，文件将被完整地传输，而不会被分割。此切换仅适用于源文件系统实现getBlockLocations方法和目标文件系统mplements concat方法时</td> 
  </tr> 
  <tr> 
   <td>-delete</td> 
   <td>从目标中删除源中丢失的文件</td> 
  </tr> 
  <tr> 
   <td>-diff </td> 
   <td>使用snapshot diff报告来识别源和目标之间的差异</td> 
  </tr> 
  <tr> 
   <td>-f </td> 
   <td>需要复制的文件列表</td> 
  </tr> 
  <tr> 
   <td>-filelimit </td> 
   <td>(弃用)将复制的文件数量限制为 &lt;= n</td> 
  </tr> 
  <tr> 
   <td>-filters </td> 
   <td>从复制的文件列表中排除</td> 
  </tr> 
  <tr> 
   <td>-i</td> 
   <td>忽略复制过程中的失败</td> 
  </tr> 
  <tr> 
   <td>-log </td> 
   <td>保存distcp执行日志的路径</td> 
  </tr> 
  <tr> 
   <td>-m </td> 
   <td>map 最大数量</td> 
  </tr> 
  <tr> 
   <td>-mapredSslConf </td> 
   <td>ssl配置文件的配置，必须在类路径中使用hftps://</td> 
  </tr> 
  <tr> 
   <td>-numListstatusThreads </td> 
   <td>用于构建文件清单的线程数(最多40个)</td> 
  </tr> 
  <tr> 
   <td>-overwrite</td> 
   <td>选择无条件地覆盖目标文件，即使它们存在。</td> 
  </tr> 
  <tr> 
   <td>-p </td> 
   <td>保存状态(rbugpcaxt)(复制、块大小、用户、组、权限、校验和类型、ACL、XATTR、时间戳)</td> 
  </tr> 
  <tr> 
   <td>-rdiff </td> 
   <td>使用目标快照差异报告进行识别目标变更</td> 
  </tr> 
  <tr> 
   <td>-sizelimit </td> 
   <td>(弃用)限制复制的文件数量 &lt;= n字节</td> 
  </tr> 
  <tr> 
   <td>-skipcrccheck</td> 
   <td>是否跳过源和目标路径之间的CRC检查。</td> 
  </tr> 
  <tr> 
   <td>-strategy </td> 
   <td>复制要使用的策略。默认情况下是根据文件大小划分工作</td> 
  </tr> 
  <tr> 
   <td>-tmp </td> 
   <td>用于原子提交的中间工作路径</td> 
  </tr> 
  <tr> 
   <td>-update</td> 
   <td>更新目标，只复制丢失的文件或目录</td> 
  </tr> 
 </tbody> 
</table>

[root@cdh01:~]# hadoop distcp hdfs://192.168.1.11:8020/tmp/hbase/test/ /tmp/hbase/test/
    19/05/29 10:01:49 INFO tools.OptionsParser: parseChunkSize: blocksperchunk false
    19/05/29 10:01:50 INFO tools.DistCp: Input Options: DistCpOptions{ atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://192.168.1.11:8020/tmp/hbase/test], targetPath=/tmp/hbase/test, targetPathExists=false, filtersFile='null', blocksPerChunk=0}
    19/05/29 10:01:50 INFO client.RMProxy: Connecting to ResourceManager at cdh01/192.168.1.101:8032
    19/05/29 10:01:51 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 20; dirCnt = 1
    19/05/29 10:01:51 INFO tools.SimpleCopyListing: Build file listing completed.
    19/05/29 10:01:51 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
    19/05/29 10:01:51 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
    19/05/29 10:01:51 INFO tools.DistCp: Number of paths in the copy list: 20
    19/05/29 10:01:51 INFO tools.DistCp: Number of paths in the copy list: 20
    19/05/29 10:01:51 INFO client.RMProxy: Connecting to ResourceManager at cdh01/192.168.1.101:8032
    19/05/29 10:01:51 INFO mapreduce.JobSubmitter: number of splits:11
    19/05/29 10:01:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559091338325_0002
    19/05/29 10:01:52 INFO impl.YarnClientImpl: Submitted application application_1559091338325_0002
    19/05/29 10:01:52 INFO mapreduce.Job: The url to track the job: http://cdh01:8088/proxy/application_1559091338325_0002/
    19/05/29 10:01:52 INFO tools.DistCp: DistCp job-id: job_1559091338325_0002
    19/05/29 10:01:52 INFO mapreduce.Job: Running job: job_1559091338325_0002
    19/05/29 10:01:58 INFO mapreduce.Job: Job job_1559091338325_0002 running in uber mode : false
    19/05/29 10:01:58 INFO mapreduce.Job:  map 0% reduce 0%
    19/05/29 10:02:03 INFO mapreduce.Job:  map 9% reduce 0%
    19/05/29 10:02:04 INFO mapreduce.Job:  map 18% reduce 0%
    19/05/29 10:02:07 INFO mapreduce.Job:  map 27% reduce 0%
    19/05/29 10:02:08 INFO mapreduce.Job:  map 36% reduce 0%
    19/05/29 10:02:09 INFO mapreduce.Job:  map 45% reduce 0%
    19/05/29 10:02:14 INFO mapreduce.Job:  map 55% reduce 0%
    19/05/29 10:02:16 INFO mapreduce.Job:  map 64% reduce 0%
    19/05/29 10:02:18 INFO mapreduce.Job:  map 82% reduce 0%
    19/05/29 10:02:20 INFO mapreduce.Job:  map 91% reduce 0%
    19/05/29 10:02:23 INFO mapreduce.Job:  map 100% reduce 0%
    19/05/29 10:04:33 INFO mapreduce.Job: Job job_1559091338325_0002 completed successfully
    19/05/29 10:04:34 INFO mapreduce.Job: Counters: 33
    	File System Counters
    		FILE: Number of bytes read=0
    		FILE: Number of bytes written=1638715
    		FILE: Number of read operations=0
    		FILE: Number of large read operations=0
    		FILE: Number of write operations=0
    		HDFS: Number of bytes read=16936022957
    		HDFS: Number of bytes written=16936015418
    		HDFS: Number of read operations=234
    		HDFS: Number of large read operations=0
    		HDFS: Number of write operations=61
    	Job Counters 
    		Launched map tasks=11
    		Other local map tasks=11
    		Total time spent by all maps in occupied slots (ms)=679786
    		Total time spent by all reduces in occupied slots (ms)=0
    		Total time spent by all map tasks (ms)=679786
    		Total vcore-milliseconds taken by all map tasks=679786
    		Total megabyte-milliseconds taken by all map tasks=1044151296
    	Map-Reduce Framework
    		Map input records=20
    		Map output records=0
    		Input split bytes=1265
    		Spilled Records=0
    		Failed Shuffles=0
    		Merged Map outputs=0
    		GC time elapsed (ms)=1804
    		CPU time spent (ms)=133510
    		Physical memory (bytes) snapshot=3378626560
    		Virtual memory (bytes) snapshot=33501384704
    		Total committed heap usage (bytes)=2952790016
    	File Input Format Counters 
    		Bytes Read=6274
    	File Output Format Counters 
    		Bytes Written=0
    	DistCp Counters
    		Bytes Copied=16936015418
    		Bytes Expected=16936015418
    		Files Copied=20