Kafka集群安装

心已赠人 2023-08-17 15:35 133阅读 0赞

### 1.1       复制软件到安装服务器 ###

### 使用ansible  vim /etc/ansible/hosts将需要安装的服务器IP配置起来　　 ###

[kafka]
    
    220.100.23.[157:159] ansible_ssh_user=linux ansible_ssh_pass="XXXXXX"

**copy\_kafka.sh 文件**

#!/bin/bash
    
    
    echo "copy kafka soft"
    
    #将软件复制到其他服务器
    ansible kafka -m copy -a "src=/usr/local/soft/kafka_2.12-2.2.0.tgz dest=/usr/local/soft"
    #解压文件
    ansible kafka -m shell -a "tar -xzvf /usr/local/soft/kafka_2.12-2.2.0.tgz -C /usr/local/soft "
    
    echo "copy kafka end"

### 1.2      配置zookeeper文件 ###

**zookeeper-prod.properties****部分内容**

#dataDir=/tmp/zookeeper
    dataDir=/usr/local/soft/zookeeper/data
    # the port at which the clients will connect
    clientPort=2181
    # disable the per-ip limit on the number of connections since this is a non-production config
    dataLogDir=/usr/local/soft/zookeeper/logs
    #最大客户端连接数
    maxClientCnxns=20
    #是作为Zookeeper服务器之间或客户端与服务器之间维持心跳的时间间隔
    tickTime=2000
    #此配置表示，允许follower(相对于Leaderer言的“客户端”)连接并同步到Leader的初始化连接时间，以tickTime为单位。当初始化连接时间超过该值，则表示连接失败。
    initLimit=10
    #此配置项表示Leader与Follower之间发送消息时，请求和应答时间长度。如果follower在设置时间内不能与leader通信，那么此follower将会被丢弃。
    syncLimit=5
    #server.myid=ip:followers_connect to the leader:leader_election # server 是固定的，myid 是需要手动分配，第一个端口是follower是链接到leader的端口，第二个是用来选举leader 用的port
    server.1=220.100.23.157:2888:3888
    server.2=220.100.23.158:2888:3888
    server.3=220.100.23.159:2888:3888

### 1.3      创建zookeeper配置需要的文件 ###

**zookeeper\_properties.sh**

#!/bin/bash
    
    echo "copy zookeeper properties"
    #创建文件夹
    ansible kafka -m shell -a "mkdir /usr/local/soft/zookeeper/{data,logs} -p"
    ansible kafka1 -m shell -a "echo 1 > /usr/local/soft/zookeeper/data/myid"
    ansible kafka2 -m shell -a "echo 2 > /usr/local/soft/zookeeper/data/myid"
    ansible kafka3 -m shell -a "echo 3 > /usr/local/soft/zookeeper/data/myid"
    #将zookeeper配置文件复制
    ansible kafka -m copy -a "src=/usr/local/soft/kafka-controller/zookeeper-prod.properties dest=/usr/local/soft/kafka_2.12-2.2.0/config"
    echo "copyproperties end"

**注意：**myid 文件内的值是不一致 的，需要与配置文件server.1对应起来

### 1.4      启动zookeeper ###

**zookeeper\_start.sh**

#!/bin/bash
    
    echo "start zookeeper"
    #执行
    ansible kafka -m shell -a "/usr/local/soft/kafka_2.12-2.2.0/bin/zookeeper-server-start.sh /usr/local/soft/kafka_2.12-2.2.0/config/zookeeper-prod.properties &"
    
    echo "end"

ps aux|grep zookeeper   **是否启动**

echo stat|nc 127.0.0.1 2181 **来查看哪个节点被选择作为follower****或者leader**

netstat -anlp | grep 2181  端口是否开放

nc -vv 110.10.13.11 2181

### 2.1      配置Kafka ###

**server-157.properties****需要修改配置内容**

broker.id=0
    listeners=PLAINTEXT:// 220.100.23.157:9092
    log.dirs=/usr/local/soft/kafka/logs
    # Zookeeper connection string (see zookeeper docs for details).
    zookeeper.connect=220.100.23.157:2181, 220.100.23.158:2181, 220.100.23.159:2181

其他机器除了broker.id 与 listeners=PLAINTEXT:// XXX:9092不同，其他配置都一样

### 2.2     Kafka配置文件发送 ###

**kafka\_properties.sh****需要修改配置内容**

#!/bin/bash
    
    echo "copy server properties properties"
    #创建文件夹
    #ansible kafka -m shell -a "mkdir /usr/local/soft/kafka/logs -p"
    ansible kafka1 -m copy -a "src=/usr/local/soft/kafka-controller/server-152.properties dest=/usr/local/soft/kafka_2.12-2.2.0/config"
    ansible kafka2 -m copy -a "src=/usr/local/soft/kafka-controller/server-153.properties dest=/usr/local/soft/kafka_2.12-2.2.0/config"
    ansible kafka3 -m copy -a "src=/usr/local/soft/kafka-controller/server-154.properties dest=/usr/local/soft/kafka_2.12-2.2.0/config"
    echo "copy server properties end"

### 2.3      Kafka启动 ###

**kafka\_start.sh****需要修改配置内容**

#!/bin/bash
    
    echo "start kafka"
    #启动kafka
    ansible kafka1 -m shell -a "/usr/local/soft/kafka_2.12-2.2.0/bin/kafka-server-start.sh /usr/local/soft/kafka_2.12-2.2.0/config/server-152.properties &"
    ansible kafka2 -m shell -a "/usr/local/soft/kafka_2.12-2.2.0/bin/kafka-server-start.sh /usr/local/soft/kafka_2.12-2.2.0/config/server-153.properties &"
    ansible kafka3 -m shell -a "/usr/local/soft/kafka_2.12-2.2.0/bin/kafka-server-start.sh /usr/local/soft/kafka_2.12-2.2.0/config/server-154.properties &"
    
    echo "end kafka"

netstat -tnlp | grep 9092 查询是否启动
     #创建主题
    bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
    #查看主题
    bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
    发送消息
    bin/kafka-console-producer.sh --broker-list 220.100.23.157:9092 --topic test
    接收消息
    bin/kafka-console-consumer.sh --bootstrap-server 220.100.23.157:9092 --from-beginning --topic test

转载于:https://www.cnblogs.com/wushenghfut/p/11586779.html