Kafka：服役（添加）新节点

今天药忘吃喽~ 2024-03-24 14:14 39阅读 0赞

**1）新节点准备（新节点配置一模一样的环境即可****）(该步骤可以忽略)**

（1）关闭 hadoop104，并右键执行克隆操作。（正常操作是配置一模一样的环境即可）

![bb9b8e85082249839643561dae047334.png][]

![06cf29bc59064ba9bc65fc98a07b426a.png][]

![28364295732541b38fd378bee2a10b21.png][]

（2）开启 hadoop105（克隆的hadoop104），并修改 IP 地址。

[root@hadoop104 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
    
    DEVICE=ens33
    TYPE=Ethernet
    ONBOOT=yes
    BOOTPROTO=static
    NAME="ens33"
    
    # 修改下面ip即可
    IPADDR=192.168.10.105
    
    PREFIX=24
    GATEWAY=192.168.10.2
    DNS1=192.168.10.2

（3）在 hadoop105 上，修改主机名称为 hadoop105。

[root@hadoop104 ~]# vim /etc/hostname
    
    hadoop105

（4）重新启动 hadoop104、hadoop105。

（5）修改 haodoop105 中 kafka 下的config 。

vim server.properties
    
    # 修改 broker.id 为 3
    # 因为现在是第四个broker

（6）删除 hadoop105 中 kafka 下的 datas 和 logs。

[root@hadoop105 kafka]$ rm -rf datas/* logs/*

（7）启动 hadoop102、hadoop103、hadoop104 上的 kafka 集群。

[root@hadoop102 ~]$ zk.sh start
    [root@hadoop102 ~]$ kf.sh start

（8）单独启动 hadoop105 中的 kafka。

[root@hadoop105 kafka]$ bin/kafka-server-start.sh -daemon ./config/server.properties

**2）执行负载均衡操作**（解决刚刚加入的新节点，怎么负载以前主题的数据）(选择以前任意一个节点)

（1）创建一个要均衡的主题。

[root@hadoop102 kafka]$ vim topics-to-move.json
    {
     "topics": [
     {"topic": "first"}
     ],
     "version": 1
    }

（2）生成一个负载均衡的计划。

[root@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
    
    Current partition replica assignment
    {"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,0,2],"log_dirs":["any","any","any"]}]}
    # 以下计划等会用于拷贝
    Proposed partition reassignment configuration
    {"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

（3）创建副本存储计划（所有副本存储在 broker0、broker1、broker2、broker3 中）。

[root@hadoop102 kafka]$ vim increase-replication-factor.json

输入如下内容：

# 下面内容，就是上面的Proposed partition reassignment configuration下面的内容
    {"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

（4）执行副本存储计划。

[root@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute

（5）验证副本存储计划。

[root@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify
    
    Status of partition reassignment:
    Reassignment of partition first-0 is complete.
    Reassignment of partition first-1 is complete.
    Reassignment of partition first-2 is complete.
    
    Clearing broker-level throttles on brokers 0,1,2,3
    Clearing topic-level throttles on topic first

[bb9b8e85082249839643561dae047334.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/3c3661218f0f491cb501f1bf47caa021.png
[06cf29bc59064ba9bc65fc98a07b426a.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/7a90adbc112443d6b259b35df39e726a.png
[28364295732541b38fd378bee2a10b21.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/6b82f643f26f451e9684b4a19dea02c9.png