Kafka常用命令(带SASL权限版)

蔚落 2022-05-19 08:18 603阅读 0赞

### 前言 ###

一般需要使用的Kafka原生命令多位于`bin`目录，其下使用较多的脚本如下

*  kafka-topics.sh
 *  kafka-console-producer.sh
 *  kafka-console-consumer.sh
 *  kafka-acls.sh
 *  kafka-consumer-groups.sh
 *  kafka-server-start.sh
 *  kafka-server-stop.sh

### Kafka Orders ###

*  Topic相关(创建/查询/删除/修改分片/平衡leader)

# 查询
    ./kafka-topics.sh --list --zookeeper 192.168.100.85:2181,192.168.100.85:2182
    
    # 详情查询
    ./kafka-topics.sh --zookeeper 192.168.100.85:2181,192.168.100.85:2182 --describe --topic topic-sean
    
    # 创建(3分片 复制2份 注意:replication-factor个数包括本来的个数)
    ./kafka-topics.sh --create --zookeeper 192.168.100.85:2181 --replication-factor 2 --partitions 3 --topic topic-sean
    
    # 修改分片个数(只能增加，不能删除。删除需要进行迁移。)
    ./kafka-topics.sh --zookeeper 192.168.100.85:2181 --alter --topic topic-sean --partitions 12
    
    # 删除 (注意 1.如果有权限需要把权限删除/2.如果需要完全删除需要开启server.properties删除属性)
    ./kafka-topics.sh --zookeeper 192.168.100.85:2181 --delete --topic topic-sean
    
    # 平衡leader（注意平衡后需要使用查询命令查看集群状况）
    ./kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot

*  生成/消费(原始)

# 生产者
    ./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic topic-sean
    # 消费者
    ./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic topic-sean --from-beginning
    
    # 新生产者（支持0.9版本+） 
    ./kafka-console-producer.sh --broker-list localhost:9092 --topic topic-sean --producer.config config/producer.properties 
    
    ## 新消费者（支持0.9版本+） 
    ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-sean --new-consumer --from-beginning --consumer.config config/consumer.properties 
    
    ## 高级点的用法(具体到某个分片)
    ./kafka-simple-consumer-shell.sh --brist localhost:9092 --topic topic-sean --partition 0 --offset 1234 --max-messages 10

*  生成/消费(sasl权限控制)

# sasl consumer
    ./kafka-console-consumer.sh  --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093  --topic bmc-events-topic-dev --consumer.config ../config/consumer.properties --new-consumer
    
    # sasl producer
    ./kafka-console-producer.sh --broker-list 192.168.100.87:9093  --topic bmc-events-topic-dev --producer.config ../config/consumer.properties

*  sasl 权限

# 增加权限
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=192.168.100.85:2181 --add --allow-principal User:alice --operation Read --operation Write --topic bmc-events-topic-test
    
    # 查询权限
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=192.168.100.85:2181 --list --topic sean-security
    
    # 删除权限
    ./kafka-acls.sh --authorizer-properties zookeeper.connect=192.168.100.85:2181 --remove --allow-principal User:alice --allow-principal User:Alice  --operation Read --operation Write --topic bmc-events-topic-dev

*  生成/消费(offset详情查询与统计)

# 无权限版本查询(ConsumerOffsetChecker)
    ./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper 192.168.100.85:2181 --group test --topic hello-sean
    
    # 无权限版本查询(GetOffsetShell)
    ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.100.87:9093 --topic hello-sean --time -1
    
    # 含权限版本查询
    ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093 --command-config ../config/consumer.properties --list
    
    # 查询单个consumer group
    ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-93774'
    
    # 查询单个consumer group(含统计)
     ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-21925' | awk ' {sum += $3};END {print sum}'
    
    # 使用脚本新生产一个consumer group(为了方便查询)
    ./kafka-console-consumer.sh  --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093  --topic hello-sean --consumer.config ../config/consumer.properties --new-consumer
    
    ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093 --command-config ../config/consumer.properties --describe --group 'event_processor_group'
    
    ./kafka-topics.sh --list --zookeeper 192.168.100.85:2181
    
    ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-67854'| awk ' {sum += $3};END {print sum}'
    
    ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-67854'| awk ' {sum += $4};END {print sum}'

--------------------

### Reference ###

\[1\]. [JIRA-KAFKA-3151(SASL)][JIRA-KAFKA-3151_SASL]  
\[2\]. [kafka命令大全(orchome.com)][kafka_orchome.com]  
\[3\]. [Kafka命令行常用命令说明][Kafka]  
\[4\]. [Kafka常用命令(csdn.yanxml)][Kafka_csdn.yanxml]  
\[5\]. [Kafka常用命令行总结][Kafka 1]

--------------------

### Others ###

`bin`目录下的所有脚本，分析作用如下：

# version kafka-1.0.0
    # kakfa 相关
    connect-standalone.sh         
    connect-distributed.sh   
    
    # consumer 
    kafka-consumer-groups.sh   
    kafka-consumer-perf-test.sh          
    kafka-console-consumer.sh     
    kafka-simple-consumer-shell.sh  
    
    # producer
    kafka-console-producer.sh   
    kafka-producer-perf-test.sh          
    
    # broker 启/停 
    kafka-server-stop.sh            
    kafka-server-start.sh  
    kafka-broker-api-versions.sh  
    
    # topic
    kafka-topics.sh                     
    
    # acls 权限
    kafka-acls.sh                 
    
    # others
    
    # kafka config 配置
    kafka-configs.sh              
    
    # 重新分片 partition 
    kafka-reassign-partitions.sh
    
    # reset kafka streaming 
    kafka-streams-application-reset.sh  
    
    kafka-replay-log-producer.sh    
    kafka-delete-records.sh              
    kafka-replica-verification.sh   
    kafka-verifiable-consumer.sh        
    kafka-log-dirs.sh                    
    kafka-verifiable-producer.sh
    kafka-mirror-maker.sh                
    trogdor.sh
    kafka-preferred-replica-election.sh  
    
    # run-class
    kafka-run-class.sh              
    
    # 内置ZooKeeper相关
    zookeeper-shell.sh
    zookeeper-server-start.sh
    zookeeper-server-stop.sh
    zookeeper-security-migration.sh
    
    # windows脚本 文件夹
    windows/

[JIRA-KAFKA-3151_SASL]: https://issues.apache.org/jira/browse/KAFKA-3151
[kafka_orchome.com]: http://orchome.com/454
[Kafka]: https://www.xuebuyuan.com/2152009.html
[Kafka_csdn.yanxml]: https://blog.csdn.net/u010416101/article/details/53905593
[Kafka 1]: https://www.jianshu.com/p/cfedb7122e38