kafka安装部署详解

ゝ一世哀愁。 2024-04-01 15:52 69阅读 0赞

#### kafka安装部署详解 ####

*  1. zookeeper安装部署
 *   *  1.1 zookeeper下载
     *  1.2 zookeeper配置启动
     *  1.3 zookeeper验证
 *  2. kafka安装部署
 *   *  2.1 kafka下载
     *  2.2 单机版配置启动
     *  2.3 集群版配置启动
     *  2.4 测试验证

## 1. zookeeper安装部署 ##

### 1.1 zookeeper下载 ###

zookeeper官网地址：[https://www.apache.org/dyn/closer.cgi/zookeeper/][https_www.apache.org_dyn_closer.cgi_zookeeper]

zookeeper下载地址：[https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/][https_dlcdn.apache.org_zookeeper_zookeeper-3.6.3]

下载文件apache-zookeeper-3.5.7-bin.tar.gz 才是启动zookeepr需要的，千万别下载错。

zookeeper必须下载含有bin开头的tar包，若下载的不是bin开头的tar包，则会抛出找不到主类异常。  
Zookeeper：无法加载主类，org.apache.zookeeper.server.quorum.QuorumPeerMain  
若启动zkServer报错：找不到或无法加载主类，肯定是tar包版本下载不对。

### 1.2 zookeeper配置启动 ###

1.环境准备
    jdk环境
    
    2.解压
    tar -zxvf zookeeper-3.4.10.tar.gz -C /opt/module/
    
    3.修改配置
    mv zoo_sample.cfg zoo.cfg
    vim zoo.cfg
    dataDir=/opt/module/zookeeper-3.4.10/zkData
    
    4.启动
    bin/zkServer.sh start
    
    5.查看
    jps

### 1.3 zookeeper验证 ###

1． 启动客户端
    [atguigu@hadoop103 zookeeper-3.4.10]$ bin/zkCli.sh
    2．显示所有操作命令
    [zk: localhost:2181(CONNECTED) 1] help
    3． 查看当前 znode 中所包含的内容
    [zk: localhost:2181(CONNECTED) 0] ls /
    [zookeeper]
    4． 查看当前节点详细数据
    [zk: localhost:2181(CONNECTED) 1] ls2 /
    [zookeeper]
    5． 分别创建 2 个普通节点
    [zk: localhost:2181(CONNECTED) 3] create /sanguo "jinlian"
    Created /sanguo
    [zk: localhost:2181(CONNECTED) 4] create /sanguo/shuguo
    "liubei"
    Created /sanguo/shuguo
    6．获得节点的值
    [zk: localhost:2181(CONNECTED) 5] get /sanguo
    jinlian

参考文档：[zookeeper分布式协调服务框架][zookeeper]

## 2. kafka安装部署 ##

### 2.1 kafka下载 ###

下载地址：[https://kafka.apache.org/downloads.html][https_kafka.apache.org_downloads.html]

### 2.2 单机版配置启动 ###

# 1.解压到op目录下
    tar -zxvf kafka_2.11-0.11.0.0.tgz -C /op
    
    # 2.修改名称
    mv kafka_2.11-0.11.0.0/ kafka
    
    # 3.在/opt/module/kafka 目录下创建 logs 文件夹
    mkdir logs
    
    # 4.修改配置文件，具体参考如下
    cd config/
    vim server.properties
    
    # 5.修改两个点，一个是kafka数据存放目录，一个zookeeper连接地址
    log.dirs=/tmp/kafka-logs
    zookeeper.connect=localhost:2181
    
    # 6.单机启动
    # 前台运行：
    bin/kafka-server-start.sh config/server.properties
    
    # 后台运行：
    nohup bin/kafka-server-start.sh config/server.properties > kafka-run.log 2>&1 &
    
    # 验证
    jps

### 2.3 集群版配置启动 ###

### 2.4 测试验证 ###

4.创建topic
    
    创建kafka topic：
    bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
    
    显示kafka topic：
    bin/kafka-topics.sh -list -zookeeper  localhost:2181
    
    5.创建生产者和消费者
    
    创建kafka生产者：
    bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
    
    创建kafka消费者：
    bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
    
    设置kafka消费组名：
    bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=test-group

参考文档：[消息中间件Kafka详解][Kafka]  
http://t.zoukankan.com/CoolJayson-p-10601515.html

[https_www.apache.org_dyn_closer.cgi_zookeeper]: https://www.apache.org/dyn/closer.cgi/zookeeper/
[https_dlcdn.apache.org_zookeeper_zookeeper-3.6.3]: https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/
[zookeeper]: https://blog.csdn.net/m0_37583655/article/details/113085876
[https_kafka.apache.org_downloads.html]: https://kafka.apache.org/downloads.html
[Kafka]: https://blog.csdn.net/m0_37583655/article/details/122563066