修炼k8s+flink+hdfs+dlink（七：flinkcdc）

一时失言乱红尘 2024-02-18 10:31 33阅读 0赞

## 一 ：flinkcdc官网链接。 ##

> https://ververica.github.io/flink-cdc-connectors/release-2.1/content/about.html

## 二：在flink中添加jar包。 ##

在flink lib目录下增加你所需要的包。

> https://kdocs.cn/join/gv467qi?f=101  
> 邀请你加入共享群「工作使用重要工具」一起进行文档协作  
> ![在这里插入图片描述][e72ad8482eee4e79abd3722f1418c2ef.png]

## 三：申请资源。 ##

cd /opt/app/flink/flink-1.13.6/bin
    yarn-session.sh  -s 1 -jm 1024m -tm 1024m -nm test_cdc -d
    yarn application -list

![在这里插入图片描述][be65fd399e794c8ca6399c88adad314d.png]

## 四：配置dlink。 ##

![在这里插入图片描述][949e0caa2e5e48988ff23d243daefb15.png]  
![在这里插入图片描述][d93bc826cc3f4b189e529fe57bc51121.png]

## 五：编写代码。 ##

set execution.checkpointing.interval=30s;
    set state.checkpoints.dir=hdfs://cluster/flink/mysql_es1;
    set state.savepoints.dir=hdfs://cluster/flink/mysql_es1;
    
    
    
    CREATE TABLE `mobile_device`( 
    `id` int, 
    `user_id` string, 
    `app_id` string, 
    `group_id` string, 
    `app_version` string, 
    `sdk_version` string, 
    `mobile` string, 
    `brand` string, 
    `model_identifier` string, 
    `model` string, 
    `system` string, 
    `device_id` string, 
    `secret_key` string, 
    `pkd` string, 
    `access_token` string, 
    `refresh_token`string, 
    `iot_device_secret` string, 
    `status` int, 
    `expiration_timeout` timestamp, 
    `login_type` int, 
    `create_time` timestamp, 
    `update_time` timestamp, 
    PRIMARY KEY (id) NOT ENFORCED
    ) WITH (
     'connector' = 'mysql-cdc',
     'hostname' = 'xx.xx.xx.xx',
     'port' = '3306',
     'username' = 'xxxx',
     'password' = 'xxxxx',
     'database-name' = 'xx',
     'table-name' = 'xxxx'
    );
    
    
    select * from mobile_device;

注意：大小写会影响数据的读取。

![在这里插入图片描述][a09c5310f5e8473a901d8bc1d05f08c5.png]

## 六：查看具体的任务。 ##

![在这里插入图片描述][491ef137a9a245c3b41e4fa919608640.png]  
![在这里插入图片描述][f353f886f4e945658a93204938eb4454.png]  
![在这里插入图片描述][a02fe4c11d954e78babda8784427c0a3.png]

[e72ad8482eee4e79abd3722f1418c2ef.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/6890997f0cfe49e9a471d3098e2c4793.png
[be65fd399e794c8ca6399c88adad314d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/5d5c3d4a33de405aa169525069a1b199.png
[949e0caa2e5e48988ff23d243daefb15.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/9831223be4ae4851a0d227fb6ff0f910.png
[d93bc826cc3f4b189e529fe57bc51121.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/9d8834f13b7a4c5cbe59028791601c5d.png
[a09c5310f5e8473a901d8bc1d05f08c5.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/1ac7d488a39a4e72aa7892c693e72156.png
[491ef137a9a245c3b41e4fa919608640.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/894418868e1a401ba702cd0ae3d12c36.png
[f353f886f4e945658a93204938eb4454.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/8a3ccc6f844249f5921c697b83f0ed5f.png
[a02fe4c11d954e78babda8784427c0a3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/02/18/c3dfca3976f54a1b902f4a97f5464fe4.png