2.【kafka运维】ConfigCommand运维脚本

墨蓝 2022-08-29 05:53 314阅读 0赞

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70_pic_center]

### 文章目录 ###

*   *  ConfigCommand
     *   *  1 查询配置
         *   *  Topic配置查询
             *  其他配置/clients/users/brokers/broker-loggers 的查询
             *   *  broker-loggers 查询指定Broker的Logger相关配置
             *  查询kafka版本信息
         *  2 增删改 配置 \`--alter\`
         *   *  Topic添加/修改动态配置
             *  Topic删除动态配置
             *  添加/删除配置同时执行
             *  其他配置同理,只需要类型改下\`--entity-type\`
             *   *  broker-loggers 查询指定Broker的Logger相关配置
         *  3. 默认配置
     *  附件
     *  More

> **日常运维** 、**问题排查** 怎么能够少了滴滴开源的  
> [滴滴开源LogiKM一站式Kafka监控与管控平台][LogiKM_Kafka]

## ConfigCommand ##

> Config相关操作; 动态配置可以覆盖默认的静态配置;

### 1 查询配置 ###

#### Topic配置查询 ####

> 展示关于Topic的动静态配置

**1.1.查询单个Topic配置**(只列举动态配置)

> `sh bin/kafka-configs.sh --describe --bootstrap-server xxxxx:9092 --topic test_create_topic`  
> 或者  
> `sh bin/kafka-configs.sh --describe --bootstrap-server 172.23.248.85:9092 --entity-type topics --entity-name test_create_topic`

**1.2.查询所有Topic配置**(包括内部Topic)(只列举动态配置)

> `sh bin/kafka-configs.sh --describe --bootstrap-server 172.23.248.85:9092 --entity-type topics`

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70]  
![在这里插入图片描述][d10ce9b066f14c8b9a491aadb86ec1c5.png]

**1.3.查询Topic的详细配置(动态+静态)**

> 只需要加上一个参数`--all`

#### 其他配置/clients/users/brokers/broker-loggers 的查询 ####

> 同理 ；只需要将`--entity-type` 改成对应的类型就行了 (topics/clients/users/brokers/broker-loggers)

##### broker-loggers 查询指定Broker的Logger相关配置 #####

> `sh bin/kafka-configs.sh --describe --bootstrap-server xxxx:9090 --entity-type broker-loggers --entity-name 想要查询的BrokerID`

#### 查询kafka版本信息 ####

> `sh bin/kafka-configs.sh --describe --bootstrap-server xxxx:9092 --version`

**所有可配置的动态配置 请看最后面的 *附件* 部分**

### 2 增删改 配置 `--alter` ###

**–alter**

**删除配置**: `--delete-config` k1=v1,k2=v2  
**添加/修改配**置: `--add-config` k1,k2  
**选择类型**: `--entity-type` (topics/clients/users/brokers/broker-  
loggers)  
**类型名称**: `--entity-name`

#### Topic添加/修改动态配置 ####

`--add-config`

> `sh bin/kafka-configs.sh --bootstrap-server xxxxx:9092 --alter --entity-type topics --entity-name test_create_topic1 --add-config file.delete.delay.ms=222222,retention.ms=999999`

#### Topic删除动态配置 ####

`--delete-config`

> `sh bin/kafka-configs.sh --bootstrap-server xxxxx:9092 --alter --entity-type topics --entity-name test_create_topic1 --delete-config file.delete.delay.ms,retention.ms`

#### 添加/删除配置同时执行 ####

> `sh bin/kafka-configs.sh --bootstrap-server xxxxx:9092 --alter --entity-type brokers --entity-default --add-config log.segment.bytes=788888888 --delete-config log.retention.ms`

#### 其他配置同理,只需要类型改下`--entity-type` ####

> 类型有: (topics/clients/users/brokers/broker- loggers)

##### broker-loggers 查询指定Broker的Logger相关配置 #####

> `sh bin/kafka-configs.sh --describe --bootstrap-server xxxx:9090 --entity-type broker-loggers --entity-name 想要查询的BrokerID`

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70 1]

哪些配置可以修改 请看最后面的附件：**ConfigCommand 的一些可选配置**

### 3. 默认配置 ###

配置默认 **`--entity-default`**

> `sh bin/kafka-configs.sh --bootstrap-server xxxxx:9090 --alter --entity-type brokers --entity-default --add-config log.segment.bytes=88888888`

动态配置的默认配置是使用了节点 `<defalut>`;

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70 2]  
该图转自https://www.cnblogs.com/lizherui/p/12271285.html

**优先级** 指定动态配置>默认动态配置>静态配置

## 附件 ##

**ConfigCommand 的一些可选配置**

--------------------

Topic相关可选配置

<table> 
 <thead> 
  <tr> 
   <th>key</th> 
   <th>value</th> 
   <th>示例</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>cleanup.policy</td> 
   <td>清理策略</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>compression.type</td> 
   <td>压缩类型(通常建议在produce端控制)</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>delete.retention.ms</td> 
   <td>压缩日志的保留时间</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>file.delete.delay.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>flush.messages</td> 
   <td>持久化message限制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>flush.ms</td> 
   <td>持久化频率</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>follower.replication.throttled.replicas</td> 
   <td>flowwer副本限流 格式：分区号:副本follower号,分区号:副本follower号</td> 
   <td>0:1,1:1</td> 
  </tr> 
  <tr> 
   <td>index.interval.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>leader.replication.throttled.replicas</td> 
   <td>leader副本限流 格式：分区号:副本Leader号</td> 
   <td>0:0</td> 
  </tr> 
  <tr> 
   <td>max.compaction.lag.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>max.message.bytes</td> 
   <td>最大的batch的message大小</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>message.downconversion.enable</td> 
   <td>message是否向下兼容</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>message.format.version</td> 
   <td>message格式版本</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>message.timestamp.difference.max.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>message.timestamp.type</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>min.cleanable.dirty.ratio</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>min.compaction.lag.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>min.insync.replicas</td> 
   <td>最小的ISR</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>preallocate</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>retention.bytes</td> 
   <td>日志保留大小(通常按照时间限制)</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>retention.ms</td> 
   <td>日志保留时间</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>segment.bytes</td> 
   <td>segment的大小限制</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>segment.index.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>segment.jitter.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>segment.ms</td> 
   <td>segment的切割时间</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>unclean.leader.election.enable</td> 
   <td>是否允许非同步副本选主</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

Broker相关可选配置

<table> 
 <thead> 
  <tr> 
   <th>key</th> 
   <th>value</th> 
   <th>示例</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>advertised.listeners</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>background.threads</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>compression.type</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>follower.replication.throttled.rate</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>leader.replication.throttled.rate</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>listener.security.protocol.map</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>listeners</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.backoff.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.dedupe.buffer.size</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.delete.retention.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.io.buffer.load.factor</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.io.buffer.size</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.io.max.bytes.per.second</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.max.compaction.lag.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.min.cleanable.ratio</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.min.compaction.lag.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleaner.threads</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.cleanup.policy</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.flush.interval.messages</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.flush.interval.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.index.interval.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.index.size.max.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.message.downconversion.enable</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.message.timestamp.difference.max.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.message.timestamp.type</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.preallocate</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.retention.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.retention.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.roll.jitter.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.roll.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.segment.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>log.segment.delete.delay.ms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>max.connections</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>max.connections.per.ip</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>max.connections.per.ip.overrides</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>message.max.bytes</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>metric.reporters</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>min.insync.replicas</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>num.io.threads</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>num.network.threads</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>num.recovery.threads.per.data.dir</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>num.replica.fetchers</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>principal.builder.class</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>replica.alter.log.dirs.io.max.bytes.per.second</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.enabled.mechanisms</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.jaas.config</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.kerberos.kinit.cmd</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.kerberos.min.time.before.relogin</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.kerberos.principal.to.local.rules</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.kerberos.service.name</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.kerberos.ticket.renew.jitter</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.kerberos.ticket.renew.window.factor</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.login.refresh.buffer.seconds</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.login.refresh.min.period.seconds</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.login.refresh.window.factor</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.login.refresh.window.jitter</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>sasl.mechanism.inter.broker.protocol</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.cipher.suites</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.client.auth</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.enabled.protocols</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.endpoint.identification.algorithm</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.key.password</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.keymanager.algorithm</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.keystore.location</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.keystore.password</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.keystore.type</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.protocol</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.provider</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.secure.random.implementation</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.trustmanager.algorithm</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.truststore.location</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.truststore.password</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>ssl.truststore.type</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>unclean.leader.election.enable</td> 
   <td></td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

Users相关可选配置

<table> 
 <thead> 
  <tr> 
   <th>key</th> 
   <th>value</th> 
   <th>示例</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>SCRAM-SHA-256</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>SCRAM-SHA-512</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>consumer_byte_rate</td> 
   <td>针对消费者user进行限流</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>producer_byte_rate</td> 
   <td>针对生产者进行限流</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>request_percentage</td> 
   <td>请求百分比</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

clients相关可选配置

<table> 
 <thead> 
  <tr> 
   <th>key</th> 
   <th>value</th> 
   <th>示例</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>consumer_byte_rate</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>producer_byte_rate</td> 
   <td></td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>request_percentage</td> 
   <td></td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

## More ##

Kafka专栏持续更新中…(源码、原理、实战、运维、视频、面试视频)

--------------------

[【kafka运维】Kafka全网最全最详细运维命令合集(精品强烈建议收藏!!!)\_石臻臻的杂货铺-CSDN博客][kafka_Kafka_-CSDN]

[【kafka实战】分区重分配可能出现的问题和排查问题思路(生产环境实战,干货!!!非常干!!!建议收藏)][kafka]

[【kafka异常】kafka 常见异常处理方案(持续更新! 建议收藏)][kafka_kafka _]

[【kafka运维】分区从分配、数据迁移、副本扩缩容 (附教学视频)][kafka_]

[【kafka源码】ReassignPartitionsCommand源码分析(副本扩缩、数据迁移、副本重分配、副本跨路径迁移][kafka_ReassignPartitionsCommand]

[【kafka】点击更多…][kafka 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70_pic_center]: /images/20220829/4535530aed9f4e2c831b2c41d6b6b6be.png
[LogiKM_Kafka]: https://github.com/didi/LogiKM
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70]: /images/20220829/7ad3e667897445969bd3abfbf34bd2ef.png
[d10ce9b066f14c8b9a491aadb86ec1c5.png]: /images/20220829/1d30bd59a3914d288b04266164c648d4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70 1]: /images/20220829/c53548ef09694a94aa228edb6cd95f6c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA2MzQwNjY_size_16_color_FFFFFF_t_70 2]: /images/20220829/3021a882ae334639b600bdea94e90ac9.png
[kafka_Kafka_-CSDN]: https://blog.csdn.net/u010634066/article/details/118215928?spm=1001.2014.3001.5501
[kafka]: https://blog.csdn.net/u010634066/article/details/118631272?spm=1001.2014.3001.5501
[kafka_kafka _]: https://blog.csdn.net/u010634066/article/details/118105676?spm=1001.2014.3001.5501
[kafka_]: https://blog.csdn.net/u010634066/article/details/118028403?spm=1001.2014.3001.5501
[kafka_ReassignPartitionsCommand]: https://blog.csdn.net/u010634066/article/details/118051963
[kafka 1]: https://mp.weixin.qq.com/mp/appmsgalbum?__biz=Mzg4ODY1NTcxNg==&action=getalbum&album_id=1966026980307304450#wechat_redirect