kafka的安装和基本使用

叁歲伎倆 2023-06-13 09:36 9阅读 0赞

## kafka概述 ##

详见：[http://kafka.apache.org/intro][http_kafka.apache.org_intro]

## Kafka架构： ##

**producer**：生产者，就是生产馒头（老妈）

**cunsumer**：消费者，吃馒头（你）

**broker**：篮子

**topic**：主题，给馒头带一个标签，topic a是给你吃的，topic b 是给你弟弟吃的

## Kafka部署以及使用 ##

**本文主要讲`Kafka`单节点部署和使用**

参考[http://kafka.apache.org/quickstart][http_kafka.apache.org_quickstart]

安装`kafka`之前要先[安装zookeeper][zookeeper]

### 安装过程： ###

#### 1. 下载： ####

[官网][Link 1]

或 [这里][Link 2]

#### 2. 解压： ####

tar -zvxf kafka_2.11-0.9.0.0.tgz  -C ~/app/

#### 3. 配置环境变量： ####

export KAFKA_HOME=/home/hadoop/app/kafka_2.11-0.9.0.0
    export PATH=$KAFKA_HOME/bin:$PATH

#### 4. 刷新`source ~/.bash_profile` ####

#### 5. 配置`server.properties` ####

cd /home/hadoop/app/kafka_2.11-0.9.0.0/config
    vi server.properties

**要修改的地方**

#修改主机名
    host.name=hadoop000
    #修改kafka日志文件目录
    log.dirs=/home/hadoop/app/tmp/kafka-logs
    #修改zookeeper的连接信息
    zookeeper.connect=hadoop000:2181

#### 6. 启动： ####

zkServer.sh start        #先启动zookeeper：
    
    #bin/kafka-server-start.sh  #尝试单独启动
    #USAGE: bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*   #提示应该指定server.properties	
    #启动kafka
    kafka-server-start.sh $KAFKA_HOME/config/server.properties

**查看进程**  
**jps**

[hadoop@hadoop000 ~]$ jps
    8917 ZooKeeperMain
    9158 Jps
    9063 Kafka    #kafka进程
    9004 QuorumPeerMain
    8525 ZooKeeperMain
    [hadoop@hadoop000 ~]$

**jps -m**

[hadoop@hadoop000 ~]$ jps -m
    8917 ZooKeeperMain
    9206 Jps -m
    9063 Kafka /home/hadoop/app/kafka_2.11-0.9.0.0/config/server.properties    #加载的配置
    9004 QuorumPeerMain /home/hadoop/app/zookeeper-3.4.5-cdh5.7.0/bin/../conf/zoo.cfg
    8525 ZooKeeperMain

#### 7. kafka单节点练习 ####

##### 7.1 创建topic（指定zookeeper） #####

kafka-topics.sh --create --zookeeper hadoop000:2181 --replication-factor 1 --partitions 1 --topic topic_hello

##### 7.2 查看所有topic #####

[hadoop@hadoop000 ~]$ kafka-topics.sh --list --zookeeper hadoop000:2181
    hello_topic
    kafka_streaming_topic
    my-replicated-topic
    streamingtopic
    topic_hello

##### 7.3 生产消息(和broker-list打交道：9092端口，详情可以看config/server.properties) #####

[hadoop@hadoop000 ~]$ kafka-console-producer.sh --broker-list hadoop000:9092 --topic topic_hello
    hello hadoop
    hello 
    spark

##### 7.4 消费消息：（zookeeper） #####

[hadoop@hadoop000 ~]$ kafka-console-consumer.sh --zookeeper hadoop000:2181 --topic topic_hello  --from-beginning
    hello hadoop
    hello 
    spark

`--from-beginning`代表从头开始消费

单节点操作的其他命令

kafka-topic.sh --describe --zookeeper hadoop000:2181   #查看主题信息

[hadoop@hadoop000 bin]$ kafka-topics.sh --describe --zookeeper hadoop000:2181
    Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
    	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
    Topic:kafka_streaming_topic	PartitionCount:1	ReplicationFactor:1	Configs:
    	Topic: kafka_streaming_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
    Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
    	Topic: my-replicated-topic	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1
    Topic:streamingtopic	PartitionCount:1	ReplicationFactor:1	Configs:
    	Topic: streamingtopic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
    Topic:topic_hello	PartitionCount:1	ReplicationFactor:1	Configs:
    	Topic: topic_hello	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

注:`Partition: 0` 代表分区从0开始

看单独的一个`topic`

[hadoop@hadoop000 bin]$ kafka-topics.sh --describe --zookeeper hadoop000:2181 --topic hello_topic
    Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
    	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

[http_kafka.apache.org_intro]: http://kafka.apache.org/intro
[http_kafka.apache.org_quickstart]: http://kafka.apache.org/quickstart
[zookeeper]: https://blog.csdn.net/qq_35885488/article/details/103085808
[Link 1]: https://archive.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
[Link 2]: https://download.csdn.net/download/qq_35885488/11978453