【hudi】数据湖客户端运维工具Hudi-Cli实战

ゞ浴缸里的玫瑰 2024-03-03 08:48 47阅读 0赞

### 数据湖客户端运维工具Hudi-Cli实战 ###

#### help ####

hudi:student_mysql_cdc_hudi_fl->help
    AVAILABLE COMMANDS
    
    Archived Commits Command
           trigger archival: trigger archival
           show archived commits: Read commits from archived files and show details
           show archived commit stats: Read commits from archived files and show details
    
    Bootstrap Command
           bootstrap run: Run a bootstrap action for current Hudi table
           bootstrap index showmapping: Show bootstrap index mapping
           bootstrap index showpartitions: Show bootstrap indexed partitions
    
    Built-In Commands
           help: Display help about available commands
           stacktrace: Display the full stacktrace of the last error.
           clear: Clear the shell screen.
           quit, exit: Exit the shell.
           history: Display or save the history of previously run commands
           version: Show version info
           script: Read and execute commands from a file.
    
    Cleans Command
           cleans show: Show the cleans
           clean showpartitions: Show partition level details of a clean
           cleans run: run clean
    
    Clustering Command
           clustering run: Run Clustering
           clustering scheduleAndExecute: Run Clustering. Make a cluster plan first and execute that plan immediately
           clustering schedule: Schedule Clustering
    
    Commits Command
           commits compare: Compare commits with another Hoodie table
           commits sync: Sync commits with another Hoodie table
           commit showpartitions: Show partition level details of a commit
           commits show: Show the commits
           commits showarchived: Show the archived commits
           commit showfiles: Show file level details of a commit
           commit show_write_stats: Show write stats of a commit
    
    Compaction Command
           compaction run: Run Compaction for given instant time
           compaction scheduleAndExecute: Schedule compaction plan and execute this plan
           compaction showarchived: Shows compaction details for a specific compaction instant
           compaction repair: Renames the files to make them consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially.
           compaction schedule: Schedule Compaction
           compaction show: Shows compaction details for a specific compaction instant
           compaction unscheduleFileId: UnSchedule Compaction for a fileId
           compaction validate: Validate Compaction
           compaction unschedule: Unschedule Compaction
           compactions show all: Shows all compactions that are in active timeline
           compactions showarchived: Shows compaction details for specified time window
    
    Diff Command
           diff partition: Check how file differs across range of commits. It is meant to be used only for partitioned tables.
           diff file: Check how file differs across range of commits
    
    Export Command
           export instants: Export Instants and their metadata from the Timeline
    
    File System View Command
           show fsview all: Show entire file-system view
           show fsview latest: Show latest file-system view
    
    HDFS Parquet Import Command
           hdfsparquetimport: Imports Parquet table to a hoodie table
    
    Hoodie Log File Command
           show logfile records: Read records from log files
           show logfile metadata: Read commit metadata from log files
    
    Hoodie Sync Validate Command
           sync validate: Validate the sync by counting the number of records
    
    Kerberos Authentication Command
           kerberos kdestroy: Destroy Kerberos authentication
           kerberos kinit: Perform Kerberos authentication
    
    Markers Command
           marker delete: Delete the marker
    
    Metadata Command
           metadata stats: Print stats about the metadata
           metadata list-files: Print a list of all files in a partition from the metadata
           metadata list-partitions: List all partitions from metadata
           metadata validate-files: Validate all files in all partitions from the metadata
           metadata delete: Remove the Metadata Table
           metadata create: Create the Metadata Table if it does not exist
           metadata init: Update the metadata table from commits since the creation
           metadata set: Set options for Metadata Table
    
    Repairs Command
           repair deduplicate: De-duplicate a partition path contains duplicates & produce repaired files to replace with
           rename partition: Rename partition. Usage: rename partition --oldPartition <oldPartition> --newPartition <newPartition>
           repair overwrite-hoodie-props: Overwrite hoodie.properties with provided file. Risky operation. Proceed with caution!
           repair migrate-partition-meta: Migrate all partition meta file currently stored in text format to be stored in base file format. See HoodieTableConfig#PARTITION_METAFILE_USE_DATA_FORMAT.
           repair addpartitionmeta: Add partition metadata to a table, if not present
           repair deprecated partition: Repair deprecated partition ("default"). Re-writes data from the deprecated partition into __HIVE_DEFAULT_PARTITION__
           repair show empty commit metadata: show failed commits
           repair corrupted clean files: repair corrupted clean files
    
    Rollbacks Command
           show rollback: Show details of a rollback instant
           commit rollback: Rollback a commit
           show rollbacks: List all rollback instants
    
    Savepoints Command
           savepoint rollback: Savepoint a commit
           savepoints show: Show the savepoints
           savepoint create: Savepoint a commit
           savepoint delete: Delete the savepoint
    
    Spark Env Command
           set: Set spark launcher env to cli
           show env: Show spark launcher env by key
           show envs all: Show spark launcher envs
    
    Stats Command
           stats filesizes: File Sizes. Display summary stats on sizes of files
           stats wa: Write Amplification. Ratio of how many records were upserted to how many records were actually written
    
    Table Command
           table update-configs: Update the table configs with configs with provided file.
           table recover-configs: Recover table configs, from update/delete that failed midway.
           refresh, metadata refresh, commits refresh, cleans refresh, savepoints refresh: Refresh table metadata
           create: Create a hoodie table if not present
           table delete-configs: Delete the supplied table configs from the table.
           fetch table schema: Fetches latest table schema
           connect: Connect to a hoodie table
           desc: Describe Hoodie Table properties
    
    Temp View Command
           temp_query, temp query: query against created temp view
           temps_show, temps show: Show all views name
           temp_delete, temp delete: Delete view name
    
    Timeline Command
           metadata timeline show incomplete: List all incomplete instants in active timeline of metadata table
           metadata timeline show active: List all instants in active timeline of metadata table
           timeline show incomplete: List all incomplete instants in active timeline
           timeline show active: List all instants in active timeline
    
    Upgrade Or Downgrade Command
           downgrade table: Downgrades a table
           upgrade table: Upgrades a table
    
    Utils Command
           utils loadClass: Load a class

#### kerberos ####

kerberos kinit --principal xxx@XXXXX.COM --keytab /xxx/kerberos/xxx.keytab

![在这里插入图片描述][4f4c0037a75b4f14b6e40230dce8d67e.png]  
先看下样例表的表结构：  
分区表哦！

-- FLink SQL建表语句
    create table student_mysql_cdc_hudi_fl(
      `_hoodie_commit_time` string comment 'hoodie commit time',
      `_hoodie_commit_seqno` string comment 'hoodie commit seqno',
      `_hoodie_record_key` string comment 'hoodie record key',
      `_hoodie_partition_path` string comment 'hoodie partition path',
      `_hoodie_file_name` string comment 'hoodie file name',
      `s_id` bigint not null comment '主键',
      `s_name` string not null comment '姓名',
      `s_age` int comment '年龄',
      `s_sex` string comment '性别',
      `s_part` string not null comment '分区字段',
      `create_time` timestamp(6) not null comment '创建时间',
      `dl_ts` timestamp(6) not null,
      `dl_s_sex` string not null,
      PRIMARY KEY(s_id) NOT ENFORCED
    )PARTITIONED BY (`dl_s_sex`) with ( 
    ,'connector' = 'hudi'
    ,'hive_sync.table' = 'student_mysql_cdc_hudi'
    ,'hoodie.datasource.write.drop.partition.columns' = 'true'
    ,'hoodie.datasource.write.hive_style_partitioning' = 'true'
    ,'hoodie.datasource.write.partitionpath.field' = 'dl_s_sex'
    ,'hoodie.datasource.write.precombine.field' = 'dl_ts'
    ,'path' = 'hdfs://xxx/hudi_db.db/student_mysql_cdc_hudi'
    ,'precombine.field' = 'dl_ts'
    ,'primaryKey' = 's_id'
    )

#### table ####

##### connect #####

connect --path /xxx/hudi_db.db/student_mysql_cdc_hudi

![在这里插入图片描述][595c396cf17940daa7884af28334bfda.png]

##### desc #####

desc

![在这里插入图片描述][b6ced542c2bb4fc98e3a783ae416e28c.png]

##### refresh #####

refresh

![在这里插入图片描述][88b4ff12098d4676b24e3ccf9479c3da.png]

##### fetch table schema #####

fetch table schema

![在这里插入图片描述][481dedfd6b90419c91ebdb591160a733.png]

"type" : "record",
      "name" : "student_mysql_cdc_hudi_fl_record",
      "namespace" : "hoodie.student_mysql_cdc_hudi_fl",
      "fields" : [ {
        
        "name" : "_hoodie_commit_time",
        "type" : [ "null", "string" ],
        "doc" : "",
        "default" : null
      }, {
        
        "name" : "_hoodie_commit_seqno",
        "type" : [ "null", "string" ],
        "doc" : "",
        "default" : null
      }, {
        
        "name" : "_hoodie_record_key",
        "type" : [ "null", "string" ],
        "doc" : "",
        "default" : null
      }, {
        
        "name" : "_hoodie_partition_path",
        "type" : [ "null", "string" ],
        "doc" : "",
        "default" : null
      }, {
        
        "name" : "_hoodie_file_name",
        "type" : [ "null", "string" ],
        "doc" : "",
        "default" : null
      }, {
        
        "name" : "_hoodie_operation",
        "type" : [ "null", "string" ],
        "doc" : "",
        "default" : null
      }, {
        
        "name" : "s_id",
        "type" : "long"
      }, {
        
        "name" : "s_name",
        "type" : "string"
      }, {
        
        "name" : "s_age",
        "type" : [ "null", "int" ],
        "default" : null
      }, {
        
        "name" : "s_sex",
        "type" : [ "null", "string" ],
        "default" : null
      }, {
        
        "name" : "s_part",
        "type" : "string"
      }, {
        
        "name" : "create_time",
        "type" : {
        
          "type" : "long",
          "logicalType" : "timestamp-micros"
        }
      }, {
        
        "name" : "dl_ts",
        "type" : {
        
          "type" : "long",
          "logicalType" : "timestamp-micros"
        }
      }, {
        
        "name" : "dl_s_sex",
        "type" : "string"
      } ]
    }

#### commit ####

##### commits show #####

commits show --sortBy "Total Bytes Written" --desc true --limit 10

![在这里插入图片描述][b9f9f3330d104277b84b4683c7bce104.png]

##### commits showarchived #####

commits showarchived

![在这里插入图片描述][825eca15506e45799c9aa6de4af509eb.png]

##### commit showfiles #####

commit showfiles --commit 20230915164442583

![在这里插入图片描述][3364e105644e4d0883f63c80d43b3848.png]

commit showfiles --commit 20230915164442583 --sortBy "Partition Path"

![在这里插入图片描述][099ac0ce544849dcba0723d91bb844a1.png]

##### commit showpartitions #####

commit showpartitions --commit 20230915164442583

![在这里插入图片描述][6eed1ea604dd40b69cd91cc7966717ea.png]

commit showpartitions --commit 20230915164442583 --sortBy "Total Bytes Written" --desc true --limit 10

![在这里插入图片描述][48413ab1afa14250854bb3d42cf5bfe5.png]

##### commit show\_write\_stats #####

commit show_write_stats --commit 20230915164442583

![在这里插入图片描述][58e6356cbc1e408899c2d44b020e5f44.png]

#### File System View ####

##### show fsview all #####

show fsview all

![在这里插入图片描述][ff2233075f5244b4b65946c54871b813.png]

##### show fsview latest #####

show fsview latest --partitionPath dl_s_sex=female

![在这里插入图片描述][4ec70835f3734066901075a29d68f950.png]

#### Log File ####

##### show logfile records #####

# 注意10 是需要取数据记录条数
    show logfile records 10 /xxx/hudi_db.db/student_mysql_cdc_hudi/dl_s_sex=female/.bf4b06b4-e897-42df-8a3c-a3a2f737d367_20230915163856302.log.1_0-1-0

![在这里插入图片描述][3d421bcf1752471ca37194f8b189b34c.png]  
数据是json格式的：

{
        
      "_hoodie_commit_time": "20230915163856302",
      "_hoodie_commit_seqno": "20230915163856302_0_83",
      "_hoodie_record_key": "88",
      "_hoodie_partition_path": "dl_s_sex=female",
      "_hoodie_file_name": "bf4b06b4-e897-42df-8a3c-a3a2f737d367",
      "_hoodie_operation": "I",
      "s_id": 88,
      "s_name": "傅亮",
      "s_age": 4,
      "s_sex": "female",
      "s_part": "2017/11/20",
      "create_time": 790128367000000,
      "dl_ts": -28800000000,
      "dl_s_sex": "female"
    }

##### show logfile metadata #####

show logfile metadata /xxx/xxx/hive/hudi_db.db/student_mysql_cdc_hudi/dl_s_sex=female/dl_create_time_yyyy=1971/dl_create_time_mm=03/.dadac2dd-7e5e-46c3-9b27-f1f03e04a90c_20230915151426134.log.1_0

图片中还有`FooterMetadata`列没显示全  
![在这里插入图片描述][16982a5aeb604a7db45f2189657e87ca.png]

{
        
      "SCHEMA": "{\"type\":\"record\",\"name\":\"student_mysql_cdc_hudi_fl_record\",\"namespace\":\"hoodie.student_mysql_cdc_hudi_fl\",\"fields\":[{\"name\":\"_hoodie_commit_time\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_commit_seqno\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_record_key\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_partition_path\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_file_name\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_operation\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"s_id\",\"type\":\"long\"},{\"name\":\"s_name\",\"type\":\"string\"},{\"name\":\"s_age\",\"type\":[\"null\",\"int\"],\"default\":null},{\"name\":\"s_sex\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"s_part\",\"type\":\"string\"},{\"name\":\"create_time\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}},{\"name\":\"dl_ts\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}},{\"name\":\"dl_s_sex\",\"type\":\"string\"}]}",
      "INSTANT_TIME": "20230915164442583"
    }

#### differ ####

##### diff partition #####

diff partition dl_s_sex=female

![在这里插入图片描述][12c525878c974cfd888d9be363be8a2d.png]

###### differ file ######

# 需要提供FileID。就是log文件的部分
    # 如log文件：.bf4b06b4-e897-42df-8a3c-a3a2f737d367_20230915163856302.log.1_0-1-0
    diff file bf4b06b4-e897-42df-8a3c-a3a2f737d367

![在这里插入图片描述][ef55792cbeec4a2db808bce56888bb8f.png]![在这里插入图片描述][27642a119ac841fca8ee7066f9fe737b.png]

#### rollbacks ####

##### show rollbacks #####

show rollbacks

![在这里插入图片描述][f17e24b9b8dc4bf5823f54a3c49b514f.png]

#### stats ####

##### stats filesizes #####

stats filesizes --partitionPath dl_s_sex=female --sortBy "95th" --desc true --limit 3

![在这里插入图片描述][507075c6c3384776b867272c72b282c3.png]

##### stats wa #####

stats wa

![在这里插入图片描述][223b1c1c17ef40bf979e22ea9cb1401b.png]

#### compaction ####

##### compactions show all #####

compactions show all

![待续！！！][f8d4ac6303284e82b96f38b1eddf6c60.png]

##### compactions showarchived #####

compactions showarchived

![在这里插入图片描述][b1ebaa5db9b24ee08486cc902d301d9f.png]

##### compaction showarchived #####

compaction showarchived 20230915200042501

![在这里插入图片描述][09d3c5f055994274a0e1892a2cc80a8a.png]

##### compaction show #####

compaction show 20230915174042680

![在这里插入图片描述][355ba10fb2d84ab2b19eb7c7e88cc4ce.png]

参考文章：  
[Apache Hudi数据湖hudi-cli客户端使用][Apache Hudi_hudi-cli]

[4f4c0037a75b4f14b6e40230dce8d67e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/748e904abafa4950b71ee902a1b3cdd6.png
[595c396cf17940daa7884af28334bfda.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/6e6748df498648768609ed3255bd1d4d.png
[b6ced542c2bb4fc98e3a783ae416e28c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/8f4618ac538a42d9ba54598c24af920e.png
[88b4ff12098d4676b24e3ccf9479c3da.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/2f6362fda9f3420e82ec0e4727761d4f.png
[481dedfd6b90419c91ebdb591160a733.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/279392d81e034becbecfbe71744a96bd.png
[b9f9f3330d104277b84b4683c7bce104.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/5dd378432c80457b9a9329cc312d0cfa.png
[825eca15506e45799c9aa6de4af509eb.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/066115cd7b314279ba9efb5fbd665f45.png
[3364e105644e4d0883f63c80d43b3848.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/bf8557009e0b472bbd73fb746a13495b.png
[099ac0ce544849dcba0723d91bb844a1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/61a3a993a8704b8aac726c1cf93f1ab4.png
[6eed1ea604dd40b69cd91cc7966717ea.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/fbe03a1d9ba24930a0ed63d513bf9291.png
[48413ab1afa14250854bb3d42cf5bfe5.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/de122650a33a4c058d6b491b277b95f7.png
[58e6356cbc1e408899c2d44b020e5f44.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/bf9c65614292424fa40705739d164417.png
[ff2233075f5244b4b65946c54871b813.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/7c07849dcafb4152bc1e33b541e1ffc0.png
[4ec70835f3734066901075a29d68f950.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/f9b990163af24b25b5d1816caacca482.png
[3d421bcf1752471ca37194f8b189b34c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/5ab1fa23efe3482f87ec33069bc22705.png
[16982a5aeb604a7db45f2189657e87ca.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/f8efa837acbf4ea0966d00720e4aaf31.png
[12c525878c974cfd888d9be363be8a2d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/c2547d7712cb45bb824aa3beaa0908a9.png
[ef55792cbeec4a2db808bce56888bb8f.png]: https://img-blog.csdnimg.cn/ef55792cbeec4a2db808bce56888bb8f.png
[27642a119ac841fca8ee7066f9fe737b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/30284b0d9bac4aea9ad0d45e9c3ca1e1.png
[f17e24b9b8dc4bf5823f54a3c49b514f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/8d8232d43b2e4fc782a8176688b8c713.png
[507075c6c3384776b867272c72b282c3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/00943690d85b4f0dbbe41f07b7a032ce.png
[223b1c1c17ef40bf979e22ea9cb1401b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/fd33aefc952948a5a4351dbe9b7ffbb5.png
[f8d4ac6303284e82b96f38b1eddf6c60.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/0b8b940aa2aa436ba4076a0aa2652ac7.png
[b1ebaa5db9b24ee08486cc902d301d9f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/e0e0a5a9843f4250bbb4da7b8d63369e.png
[09d3c5f055994274a0e1892a2cc80a8a.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/41bcf1660075404995941c7429f2b071.png
[355ba10fb2d84ab2b19eb7c7e88cc4ce.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/03/8a32c575a08f45e98b4beade29b85f80.png
[Apache Hudi_hudi-cli]: http://t.csdn.cn/rwqQD