docker-compose部署kafka集群（阿里云ECS也适用）

悠悠 2022-12-20 02:27 195阅读 0赞

**记录下怎么使用docker-compose部署kafka集群并 通过SpringBoot进行测试。**

前言：根据网络里的大佬使用docker-compose构建kafka集群但还是遇到了许多问题，要么是服务器跑通了但是测试用例跑不通等奇奇怪怪的原因。遂当自己构建成功的时候觉得时候觉得是时候回报社会了。

遇到的问题：（文末附链接地址以及解决方法）  
1.两个kafka之间服务通信不了  
2.springboot访问不到kafka的host等

## 1.首先构建docker的内部网络用于kafka节点间的通讯 ##

docker network create --subnet=172.18.0.0/24 br17218

查看构建的docker网络

docker network ls

当看到下图中显示的内容时则表示docker已经安装成功了  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center]

## 2.创建目录存放编排脚本并运行 ##

**2.1**  
创建一个目录,刷新后可以看到刚刚创建的目录结构

mkdir -p /usr/local/docker/kafka

**2.2**  
在编辑框内编写服务编排脚本 docker-compose.yml

vi docker-compose.yml
    :set paste # 用于把下面的yaml复制粘贴

docker-compose.yml文件

version: '3'
    
    services:
      zoo1:
        image: wurstmeister/zookeeper
        restart: always
        hostname: zoo1
        container_name: zoo1
        ports:
          - 2181:2181
        environment:
          ZOO_MY_ID: 1
          ZOO_SERVERS: server.1=0.0.0.0:2888:3888 
        networks:
          br17218:
            ipv4_address: 172.18.0.11
    
      kafka1:
        image: wurstmeister/kafka
        restart: always
        hostname: kafka1
        container_name: kafka1
        ports:
          - 9092:9092
        environment:
          KAFKA_ADVERTISED_HOST_NAME: kafka1
          KAFKA_ADVERTISED_PORT: 9092
          KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://xx.xx.xx.xx:9092 #自己的虚拟机ip地址 (注意这里一定要自己的虚拟机ip地址不然java程序访问不到)
          KAFKA_LISTENERS: PLAINTEXT://kafka1:9092
          KAFKA_HEAP_OPTS: "-Xmx256M -Xms128M" # 我的阿里云虚拟机内存小所以设置kafka的内存大小
        external_links:
          - zoo1
        networks:
          br17218:
            ipv4_address: 172.18.0.14
    
      kafka2:
        image: wurstmeister/kafka
        restart: always
        hostname: kafka2
        container_name: kafka2
        ports:
          - 9093:9093
        environment:
          KAFKA_ADVERTISED_HOST_NAME: kafka2
          KAFKA_ADVERTISED_PORT: 9093
          KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://xx.xx.xx.xx:9093 #自己的虚拟机ip地址
          KAFKA_LISTENERS: PLAINTEXT://kafka2:9093
          KAFKA_HEAP_OPTS: "-Xmx256M -Xms128M" 
        external_links:
          - zoo1
        networks:
          br17218:
            ipv4_address: 172.18.0.15
      kafka-manager:
        image: sheepkiller/kafka-manager              ## 镜像：开源的web管理kafka集群的界面
        environment:
            ZK_HOSTS: xx.xx.xx.xx                   ## 修改:宿主机IP
        ports:
          - "9000:9000"                               ## 暴露端口
    
    networks:
      br17218:
        external:
          name: br17218

**2.3**  
在 /usr/local/docker/kafka目录下  
执行docker-compose的启动命令

docker-compose up -d

查看docker容器，当有下图的时候就是成功了

docker ps

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 1]

## 3.在kafka里创建topic1、topic2用于测试 ##

**3.1**  
进入dockder里的kafka容器

docker exec -it kafka1  /bin/sh

创建topic1

kafka-topics.sh --create --zookeeper 47.112.252.218:2181 --replication-factor 2 --partitions 2 --topic topic1
    Created topic topic1

创建topic2

kafka-topics.sh --create --zookeeper 47.112.252.218:2181 --replication-factor 2 --partitions 2 --topic topic2
    Created topic topic2

看到下图就是创建成功了

kafka-topics.sh --list --zookeeper xx.xx.xx.xx:2181 #xx填写自己的虚拟机ip地址

![在这里插入图片描述][20201109225101817.png_pic_center]

**3.2**  
阿里云ECS的小伙伴得去安全组开放相应的端口  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 2]  
虚拟机的小伙伴得关闭防火墙开放端口

## 4.构建Springboot项目测试 ##

**4.1**  
引入下面的依赖

<dependency>
     <groupId>org.springframework.kafka</groupId>
     <artifactId>spring-kafka</artifactId>
     </dependency>

只需要增加这一个依赖

这里有大坑，版本不能随便用。kafka的client包必须和安装的kafka版本对应

下面是spring官方给的对应关系[https://spring.io/projects/spring-kafka][https_spring.io_projects_spring-kafka]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 3]  
我装的kafka是2.1.0版本，所以应该使用的kafka-client是1.0.x或1.1.x或2.0.0

但是spring-kafka已经依赖了kafka-client，我们需要确认依赖进的是哪个版本

项目的spring-boot-starter-parent版本：2.0.5.RELEASE

点spring-kafka进去看到

<groupId>org.apache.kafka</groupId>
     <artifactId>kafka-clients</artifactId>
     <version>1.0.2</version>

kafka-client的版本是1.0.2，满足要求

项目结构  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 4]

yml配置文件：

server:
      port: 8190
    spring:
      application:
        name: service-kafka
      kafka:
        producer:
          bootstrap-servers: xx.xx.xx.xx:9092, xx.xx.xx.xx:9093 # xxx是自己的虚拟机的ip地址
        consumer:
          bootstrap-servers: xx.xx.xx.xx:9092, xx.xx.xx.xx:9093
          group-id: topic1
          enable-auto-commit: true
          auto-offset-reset: latest

生产者 KafkaProducer

/** * 生产者 * 使用@EnableScheduling注解开启定时任务 */
    @Component
    @EnableScheduling
    public class KafkaProducer { 
        @Autowired
        private KafkaTemplate kafkaTemplate;
        /** * 定时任务 */
        @Scheduled(cron = "0/5 * * * * ?")
        public void send(){ 
            String message = UUID.randomUUID().toString();
            ListenableFuture future = kafkaTemplate.send("topic1", message);
            future.addCallback(o -> System.out.println("send-消息发送成功：" + message), throwable -> System.out.println("消息发送失败：" + message));
        }
    }

消费者 KafkaConsumer

@Component
    public class KafkaConsumer { 
        @KafkaListener(topics = { "topic1"})
        public void receive(String message){ 
            System.out.println("消费消息:" + message);
        }
    }

启动类

@SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
    @ComponentScan({ "com.atguigu.guli"})
    @EnableScheduling
    public class ServiceKafkaApplication { 
    
        public static void main(String[] args) { 
            SpringApplication.run(ServiceKafkaApplication.class, args);
        }
    
    }

成功跑通的结果图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 5]

附链接地址:  
zookeeper搭建：[https://zhuanlan.zhihu.com/p/110677319][https_zhuanlan.zhihu.com_p_110677319]  
kafka集群：[https://zhuanlan.zhihu.com/p/110905106][https_zhuanlan.zhihu.com_p_110905106]  
使用docker部署kafka集群【Contos7】:https://blog.csdn.net/weixin\_42831855/article/details/91980398  
docker-compose部署zk集群、kafka集群以及kafka-manager，及其遇到的问题和解决:[https://blog.csdn.net/weixin\_34004750/article/details/94625801][https_blog.csdn.net_weixin_34004750_article_details_94625801]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/faa1c5bb3ca349dcb4d3484f43d147e7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/a97b54bcce0043f1bd19460739f1ffab.png
[20201109225101817.png_pic_center]: https://img-blog.csdnimg.cn/20201109225101817.png#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/20201110085953295.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ==,size_16,color_FFFFFF,t_70#pic_center
[https_spring.io_projects_spring-kafka]: https://spring.io/projects/spring-kafka
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 3]: https://img-blog.csdnimg.cn/2020110922562081.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 4]: https://img-blog.csdnimg.cn/20201109225720873.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ_size_16_color_FFFFFF_t_70_pic_center 5]: https://img-blog.csdnimg.cn/20201110090559455.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTQyMDkxOQ==,size_16,color_FFFFFF,t_70#pic_center
[https_zhuanlan.zhihu.com_p_110677319]: https://zhuanlan.zhihu.com/p/110677319
[https_zhuanlan.zhihu.com_p_110905106]: https://zhuanlan.zhihu.com/p/110905106
[https_blog.csdn.net_weixin_34004750_article_details_94625801]: https://blog.csdn.net/weixin_34004750/article/details/94625801