Kafka动态调整topic副本因子replication-factor

素颜马尾好姑娘i 2022-05-03 02:58 1112阅读 0赞

实际项目中我们可能在创建topic时没有设置好正确的replication-factor，导致kafka集群虽然是高可用的，但是该topic在有broker宕机时，可能发生无法使用的情况。topic一旦使用又不能轻易删除重建，因此动态增加副本因子就成为最终的选择。

**说明**：kafka 1.0版本配置文件默认没有default.replication.factor=x， 因此如果创建topic时，不指定–replication-factor 想， 默认副本因子为1. 我们可以在自己的server.properties中配置上常用的副本因子，省去手动调整。例如设置default.replication.factor=3， 详细内容可参考官方文档https://kafka.apache.org/documentation/\#replication

**原因分析：**

假设我们有3个kafka broker分别brokerA、brokerB、brokerC.

1.  当我们创建的topic有3个分区partition时并且replication-factor为1，基本上一个broker上一个分区。当一个broker宕机了，该topic就无法使用了，因为三个分区只有两个能用，
2.  当我们创建的topic有3个分区partition时并且replication-factor为2时，可能分区数据分布情况是  
    brokerA， partiton0，partiton1，  
    brokerB， partiton1，partiton2  
    brokerC， partiton2，partiton0，  
    每个分区有一个副本，当其中一个broker宕机了，kafka集群还能完整凑出该topic的三个分区，例如当brokerA宕机了，可以通过brokerB和brokerC组合出topic的三个分区。

**如何动态给已经创建的topic添加replication-factor？**  
可能很多人想使用kafka-topics.sh脚本，那么事情情况如何了？

[root@xszeree6p5z bin]# ./kafka-topics.sh --alter --topic yqtopic01 --zookeeper localhost:2181 --replication-factor 3
    Option "[replication-factor]" can't be used with option"[alter]"
    Option                                   Description
    ------                                   -----------
    --alter                                  Alter the number of partitions,
                                               replica assignment, and/or
                                               configuration for the topic.
    --config <String: name=value>            A topic configuration override for the

截图  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_27_color_FFFFFF_t_70]

可以看出kafka-topics.sh不能用来增加副本因子replication-factor。实际应该使用kafka [bin目录下面的kafka-reassign-partitions.sh][bin_kafka-reassign-partitions.sh]。  
**a,** 首先我们配置topic的副本，保存为json文件（）  
例如， 我们想把yqtopic01的部分设置为3，（我的kafka集群有3个broker，id分别为0,1,2）， json文件名称为increase-replication-factor.json

\{“version”:1,  
“partitions”:\[  
\{“topic”:“yqtopic01”,“partition”:0,“replicas”:\[0,1,2\]\},  
\{“topic”:“yqtopic01”,“partition”:1,“replicas”:\[0,1,2\]\},  
\{“topic”:“yqtopic01”,“partition”:2,“replicas”:\[0,1,2\]\}  
\]\}  
**b,** 然后执行脚本  
./kafka-reassign-partitions.sh -zookeeper 127.0.0.1:2181 --reassignment-json-file increase-replication-factor.json --execute

kafka-reassign-partitions.sh执行截图  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_27_color_FFFFFF_t_70 1]

我们可以通过执行  
[kafka-topics.sh][] \--describe --zookeeper localhost:2181 --topic yqtopic01查看现在该topic的副本因子。

**总结**  
所有文档官方文档最权威。[https://kafka.apache.org/documentation/\#basic\_ops\_increase\_replication\_factor][https_kafka.apache.org_documentation_basic_ops_increase_replication_factor]

摘录如下：  
Increasing replication factor  
Increasing the replication factor of an existing partition is easy. Just specify the extra replicas in the custom reassignment json file and use it with the --execute option to increase the replication factor of the specified partitions.

For instance, the following example increases the replication factor of partition 0 of topic foo from 1 to 3. Before increasing the replication factor, the partition’s only replica existed on broker 5. As part of increasing the replication factor, we will add more replicas on brokers 6 and 7.

The first step is to hand craft the custom reassignment plan in a json file:

> cat increase-replication-factor.json
    { "version":1,
    "partitions":[{ "topic":"foo","partition":0,"replicas":[5,6,7]}]}

Then, use the json file with the --execute option to start the reassignment process:

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
    Current partition replica assignment
     
    { "version":1,
    "partitions":[{ "topic":"foo","partition":0,"replicas":[5]}]}
    
     
    Save this to use as the --reassignment-json-file option during rollback
    Successfully started reassignment of partitions
    { "version":1,
    "partitions":[{ "topic":"foo","partition":0,"replicas":[5,6,7]}]}

The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option:

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
    Status of partition reassignment:
    Reassignment of partition [foo,0] completed successfully

You can also verify the increase in replication factor with the kafka-topics tool:

> bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe
    Topic:foo   PartitionCount:1    ReplicationFactor:3 Configs:
      Topic: foo    Partition: 0    Leader: 5   Replicas: 5,6,7 Isr: 5,6,7

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_27_color_FFFFFF_t_70]: /images/20220503/161980f5d91a43ccbf5f11d085a59e7f.png
[bin_kafka-reassign-partitions.sh]: http://xn--binkafka-reassign-partitions-c738byx40a0gyg30cl740b.sh
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3J1c3NsZQ_size_27_color_FFFFFF_t_70 1]: /images/20220503/518845e1c5a94b2a8175497304fb8b30.png
[kafka-topics.sh]: http://kafka-topics.sh
[https_kafka.apache.org_documentation_basic_ops_increase_replication_factor]: https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor