四十、分析镜像文件、编辑日志文件的生成

Love The Way You Lie 2022-05-11 12:44 123阅读 0赞

#                                                          分析镜像文件、编辑日志文件的生成 #

我们使用一个文件上传来分析镜像文件、编辑日志文件中都记录了什么内容。

### 1、初始化集群 ###

初始化：

[admin@hadoop14 hadoop-2.7.2]$ xcall rm -rf /tmp/hadoop*
    -------------localhost----------
    ----------hadoop14---------
    ----------hadoop15---------
    ----------hadoop16---------
    [admin@hadoop14 hadoop-2.7.2]$ xcall rm -rf /opt/module/hadoop-2.7.2/logs/ /opt/module/hadoop-2.7.2/data/
    -------------localhost----------
    ----------hadoop14---------
    ----------hadoop15---------
    ----------hadoop16---------
    [admin@hadoop14 hadoop-2.7.2]$  bin/hdfs namenode -format

启动：

[admin@hadoop14 hadoop-2.7.2]$ sbin/start-dfs.sh
    
    [admin@hadoop15 hadoop-2.7.2]$ sbin/start-yarn.sh

### 2、查看编辑日志 ###

编辑日志文件记录的是文件系统（hdfs）的**操作步骤**。

[admin@hadoop14 hadoop-2.7.2]$ cd /opt/module/hadoop-2.7.2/data/tmp/dfs/name/current
    
    上传文件：
    [admin@hadoop14 hadoop-2.7.2]$ hadoop hdfs -put VERSION /
    
    查看编辑日志文件：
    [admin@hadoop14 current]$ hdfs oev -p XML -i edits_inprogress_0000000000000000001 -o edit.xml
    [admin@hadoop14 current]$ ll
    总用量 1044
    -rw-rw-r--. 1 admin admin 1048576 10月 20 14:55 edits_inprogress_0000000000000000001
    -rw-rw-r--. 1 admin admin    2655 10月 20 15:00 edit.xml
    -rw-rw-r--. 1 admin admin     352 10月 20 14:47 fsimage_0000000000000000000
    -rw-rw-r--. 1 admin admin      62 10月 20 14:47 fsimage_0000000000000000000.md5
    -rw-rw-r--. 1 admin admin       2 10月 20 14:49 seen_txid
    -rw-rw-r--. 1 admin admin     204 10月 20 14:47 VERSION
    [admin@hadoop14 current]$ cat edit.xml 
    <?xml version="1.0" encoding="UTF-8"?>
    <EDITS>
      <EDITS_VERSION>-63</EDITS_VERSION>
      <RECORD>
        <OPCODE>OP_START_LOG_SEGMENT</OPCODE>
        <DATA>
          <TXID>1</TXID>
        </DATA>
      </RECORD>
      <RECORD>
        <OPCODE>OP_ADD</OPCODE>
        <DATA>
          <TXID>2</TXID>
          <LENGTH>0</LENGTH>
          <INODEID>16386</INODEID>
          <PATH>/VERSION._COPYING_</PATH>
          <REPLICATION>3</REPLICATION>
          <MTIME>1540018511447</MTIME>
          <ATIME>1540018511447</ATIME>
          <BLOCKSIZE>134217728</BLOCKSIZE>
          <CLIENT_NAME>DFSClient_NONMAPREDUCE_201991339_1</CLIENT_NAME>
          <CLIENT_MACHINE>192.168.1.114</CLIENT_MACHINE>
          <OVERWRITE>true</OVERWRITE>
          <PERMISSION_STATUS>
            <USERNAME>admin</USERNAME>
            <GROUPNAME>supergroup</GROUPNAME>
            <MODE>420</MODE>
          </PERMISSION_STATUS>
          <RPC_CLIENTID>38625766-ebba-405c-9c66-352cc9e1e908</RPC_CLIENTID>
          <RPC_CALLID>3</RPC_CALLID>
        </DATA>
      </RECORD>
      <RECORD>
        <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE>
        <DATA>
          <TXID>3</TXID>
          <BLOCK_ID>1073741825</BLOCK_ID>
        </DATA>
      </RECORD>
      <RECORD>
        <OPCODE>OP_SET_GENSTAMP_V2</OPCODE>
        <DATA>
          <TXID>4</TXID>
          <GENSTAMPV2>1001</GENSTAMPV2>
        </DATA>
      </RECORD>
      <RECORD>
        <OPCODE>OP_ADD_BLOCK</OPCODE>
        <DATA>
          <TXID>5</TXID>
          <PATH>/VERSION._COPYING_</PATH>
          <BLOCK>
            <BLOCK_ID>1073741825</BLOCK_ID>
            <NUM_BYTES>0</NUM_BYTES>
            <GENSTAMP>1001</GENSTAMP>
          </BLOCK>
          <RPC_CLIENTID></RPC_CLIENTID>
          <RPC_CALLID>-2</RPC_CALLID>
        </DATA>
      </RECORD>
      <RECORD>
        <OPCODE>OP_CLOSE</OPCODE>
        <DATA>
          <TXID>6</TXID>
          <LENGTH>0</LENGTH>
          <INODEID>0</INODEID>
          <PATH>/VERSION._COPYING_</PATH>
          <REPLICATION>3</REPLICATION>
          <MTIME>1540018514438</MTIME>
          <ATIME>1540018511447</ATIME>
          <BLOCKSIZE>134217728</BLOCKSIZE>
          <CLIENT_NAME></CLIENT_NAME>
          <CLIENT_MACHINE></CLIENT_MACHINE>
          <OVERWRITE>false</OVERWRITE>
          <BLOCK>
            <BLOCK_ID>1073741825</BLOCK_ID>
            <NUM_BYTES>229</NUM_BYTES>
            <GENSTAMP>1001</GENSTAMP>
          </BLOCK>
          <PERMISSION_STATUS>
            <USERNAME>admin</USERNAME>
            <GROUPNAME>supergroup</GROUPNAME>
            <MODE>420</MODE>
          </PERMISSION_STATUS>
        </DATA>
      </RECORD>
      <RECORD>
        <OPCODE>OP_RENAME_OLD</OPCODE>
        <DATA>
          <TXID>7</TXID>
          <LENGTH>0</LENGTH>
          <SRC>/VERSION._COPYING_</SRC>
          <DST>/VERSION</DST>
          <TIMESTAMP>1540018514452</TIMESTAMP>
          <RPC_CLIENTID>38625766-ebba-405c-9c66-352cc9e1e908</RPC_CLIENTID>
          <RPC_CALLID>8</RPC_CALLID>
        </DATA>
      </RECORD>
    </EDITS>

### 3、查看镜像文件 ###

[admin@hadoop14 current]$ hdfs oiv -p XML -i fsimage_0000000000000000000 -o image.xml
    [admin@hadoop14 current]$ cat image.xml 
    <?xml version="1.0"?>
    <fsimage>
    	<NameSection>
    		<genstampV1>1000</genstampV1>
    		<genstampV2>1000</genstampV2>
    		<genstampV1Limit>0</genstampV1Limit>
    		<lastAllocatedBlockId>1073741824</lastAllocatedBlockId>
    		<txid>0</txid>
    	</NameSection>
    	<INodeSection>
    		<lastInodeId>16385</lastInodeId>
    		<inode>
    			<id>16385</id>
    			<type>DIRECTORY</type>
    			<name/>
    			<mtime>0</mtime>
    			<permission>admin:supergroup:rwxr-xr-x</permission>
    			<nsquota>9223372036854775807</nsquota>
    			<dsquota>-1</dsquota>
    		</inode>
    	</INodeSection>
    	<INodeReferenceSection/>
    	<SnapshotSection>
    		<snapshotCounter>0</snapshotCounter>
    	</SnapshotSection>
    	<INodeDirectorySection/>
    	<FileUnderConstructionSection/>
    	<SnapshotDiffSection>
    		<diff>
    			<inodeid>16385</inodeid>
    		</diff>
    	</SnapshotDiffSection>
    	<SecretManagerSection>
    		<currentId>0</currentId>
    		<tokenSequenceNumber>0</tokenSequenceNumber>
    	</SecretManagerSection>
    	<CacheManagerSection>
    		<nextDirectiveId>1</nextDirectiveId>
    	</CacheManagerSection>
    </fsimage>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25ld2JpZV85MDc0ODY4NTI_size_16_color_FFFFFF_t_70_pic_center][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25ld2JpZV85MDc0ODY4NTI_size_16_color_FFFFFF_t_70_pic_center]: /images/20220505/8ad842d87eff4c82b8858b87b5711d4c.png