1、Kafka伪分布安装

蔚落 2022-08-01 06:14 156阅读 0赞

1、Kafka伪分布安装

下载的地址： [http://kafka.apache.org/downloads.html][http_kafka.apache.org_downloads.html]  
  
  
下载的软件：kafka\_2.10-0.8.2.1.tgz  
  
  
1、编辑文件：config/server.properties  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# pwd  
/usr/local/kafka\_2.10-0.8.2.1  
\[root@baozi kafka\_2.10-0.8.2.1\]\# ll  
总用量 28  
drwxr-xr-x. 3 root root  4096 2月  27 06:17 bin  
drwxr-xr-x. 2 root root  4096 6月  20 20:35 config  
drwxr-xr-x. 2 root root  4096 2月  27 06:17 libs  
\-rw-r--r--. 1 root root 11358 2月  27 06:02 LICENSE  
\-rw-r--r--. 1 root root   162 2月  27 06:02 NOTICE  
\[root@baozi kafka\_2.10-0.8.2.1\]\#  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# vim config/server.properties  
  
host.name=192.168.1.200  
log.dirs=/mykafka/kafka-logs  
  
  
2、启动zookeeper：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-server-start.sh config/zookeeper.properties &  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# jps  
2028 QuorumPeerMain  
2122 Jps  
\[root@baozi kafka\_2.10-0.8.2.1\]\#  
  
  
3、后台启动kafka：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# jps  
2134 Kafka  
2222 Jps  
2028 QuorumPeerMain  
\[root@baozi kafka\_2.10-0.8.2.1\]\#  
  
  
4、查看所有topic列表：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181  
  
//这个是在下面执行完创建topic命令之后输入的：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181  
\[2015-06-20 20:59:52,034\] INFO Accepted socket connection from /192.168.1.200:48535 (org.apache.zookeeper.server.NIOServerCnxnFactory)  
\[2015-06-20 20:59:52,040\] INFO Client attempting to establish new session at /192.168.1.200:48535 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 20:59:52,042\] INFO Established session 0x14e10fd34b3000b with negotiated timeout 30000 for client /192.168.1.200:48535 (org.apache.zookeeper.server.ZooKeeperServer)  
test1  
test2  
\[2015-06-20 20:59:52,236\] INFO Processed session termination for sessionid: 0x14e10fd34b3000b (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 20:59:52,240\] INFO Closed socket connection for client /192.168.1.200:48535 which had sessionid 0x14e10fd34b3000b (org.apache.zookeeper.server.NIOServerCnxn)  
\[root@baozi kafka\_2.10-0.8.2.1\]\#  
  
  
5、创建topic：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-topics.sh --create --zookeeper 192.168.1.200:2181 --replication-factor 1 --partitions 3 --topic test1  
  
  
6、describe命令：  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-topics.sh --describe --zookeeper 192.168.1.200:2181 --topic test1  
  
  
显示：  
\[2015-06-20 20:55:07,901\] INFO Accepted socket connection from /192.168.1.200:48529 (org.apache.zookeeper.server.NIOServerCnxnFactory)  
\[2015-06-20 20:55:07,907\] INFO Client attempting to establish new session at /192.168.1.200:48529 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 20:55:07,909\] INFO Established session 0x14e10fd34b30005 with negotiated timeout 30000 for client /192.168.1.200:48529 (org.apache.zookeeper.server.ZooKeeperServer)  
Topic:test1     PartitionCount:3        ReplicationFactor:1     Configs:  
        Topic: test1    Partition: 0    Leader: 0       Replicas: 0     Isr: 0  
        Topic: test1    Partition: 1    Leader: 0       Replicas: 0     Isr: 0  
        Topic: test1    Partition: 2    Leader: 0       Replicas: 0     Isr: 0  
\[2015-06-20 20:55:08,831\] INFO Processed session termination for sessionid: 0x14e10fd34b30005 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 20:55:08,833\] INFO Closed socket connection for client /192.168.1.200:48529 which had sessionid 0x14e10fd34b30005 (org.apache.zookeeper.server.NIOServerCnxn)  
  
  
7、查看所有topic分区情况：  
创建topic ->test2  
bin/kafka-topics.sh --create --zookeeper 192.168.1.200:2181 --replication-factor 1 --partitions 3 --topic test2  
  
  
命令：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-topics.sh --describe --zookeeper 192.168.1.200:2181  
\[2015-06-20 20:57:22,513\] INFO Accepted socket connection from /192.168.1.200:48533 (org.apache.zookeeper.server.NIOServerCnxnFactory)  
\[2015-06-20 20:57:22,521\] INFO Client attempting to establish new session at /192.168.1.200:48533 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 20:57:22,524\] INFO Established session 0x14e10fd34b30009 with negotiated timeout 30000 for client /192.168.1.200:48533 (org.apache.zookeeper.server.ZooKeeperServer)  
Topic:test1     PartitionCount:3        ReplicationFactor:1     Configs:  
        Topic: test1    Partition: 0    Leader: 0       Replicas: 0     Isr: 0  
        Topic: test1    Partition: 1    Leader: 0       Replicas: 0     Isr: 0  
        Topic: test1    Partition: 2    Leader: 0       Replicas: 0     Isr: 0  
Topic:test2     PartitionCount:3        ReplicationFactor:1     Configs:  
        Topic: test2    Partition: 0    Leader: 0       Replicas: 0     Isr: 0  
        Topic: test2    Partition: 1    Leader: 0       Replicas: 0     Isr: 0  
        Topic: test2    Partition: 2    Leader: 0       Replicas: 0     Isr: 0  
\[2015-06-20 20:57:23,678\] INFO Processed session termination for sessionid: 0x14e10fd34b30009 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 20:57:23,682\] INFO Closed socket connection for client /192.168.1.200:48533 which had sessionid 0x14e10fd34b30009 (org.apache.zookeeper.server.NIOServerCnxn)  
\[root@baozi kafka\_2.10-0.8.2.1\]\#  
  
  
8、删除主题topic：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic test1 --zookeeper 192.168.1.200:2181  
  
  
查看所有主题：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181                                              
\[2015-06-20 21:08:51,864\] INFO Accepted socket connection from /192.168.1.200:48538 (org.apache.zookeeper.server.NIOServerCnxnFactory)  
\[2015-06-20 21:08:51,870\] INFO Client attempting to establish new session at /192.168.1.200:48538 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 21:08:51,872\] INFO Established session 0x14e10fd34b3000e with negotiated timeout 30000 for client /192.168.1.200:48538 (org.apache.zookeeper.server.ZooKeeperServer)  
test1 - marked for deletion  
test2  
\[2015-06-20 21:08:52,066\] INFO Processed session termination for sessionid: 0x14e10fd34b3000e (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:08:52,071\] INFO Closed socket connection for client /192.168.1.200:48538 which had sessionid 0x14e10fd34b3000e (org.apache.zookeeper.server.NIOServerCnxn)  
\[root@baozi kafka\_2.10-0.8.2.1\]\#  
  
  
9、进入生产者状态：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-console-producer.sh --broker-list 192.168.1.200:9092 --topic test2  
\[2015-06-20 21:11:18,523\] WARN Property topic is not valid (kafka.utils.VerifiableProperties)  
  
  
\[root@baozi ~\]\# jps  
2134 Kafka  
2780 Jps  
2028 QuorumPeerMain  
2759 ConsoleProducer  
\[root@baozi ~\]\#  
  
  
10、进入消费者：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-console-consumer.sh --zookeeper 192.168.1.200:2181 --topic test2 --from-beginning  
  
  
\[root@baozi ~\]\# jps  
2793 ConsoleConsumer  
2134 Kafka  
2843 Jps  
2028 QuorumPeerMain  
2759 ConsoleProducer  
\[root@baozi ~\]\#  
  
  
11、在生产者这里写入：  
hello world  
hadoop hive hbase i am baolibin  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-console-producer.sh --broker-list 192.168.1.200:9092 --topic test2  
\[2015-06-20 21:11:18,523\] WARN Property topic is not valid (kafka.utils.VerifiableProperties)  
\[2015-06-20 21:14:05,037\] INFO Accepted socket connection from /192.168.1.200:48539 (org.apache.zookeeper.server.NIOServerCnxnFactory)  
\[2015-06-20 21:14:05,045\] INFO Client attempting to establish new session at /192.168.1.200:48539 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 21:14:05,048\] INFO Established session 0x14e10fd34b3000f with negotiated timeout 30000 for client /192.168.1.200:48539 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 21:14:05,621\] INFO Accepted socket connection from /192.168.1.200:48540 (org.apache.zookeeper.server.NIOServerCnxnFactory)  
\[2015-06-20 21:14:05,622\] INFO Client attempting to establish new session at /192.168.1.200:48540 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 21:14:05,626\] INFO Established session 0x14e10fd34b30010 with negotiated timeout 6000 for client /192.168.1.200:48540 (org.apache.zookeeper.server.ZooKeeperServer)  
\[2015-06-20 21:14:05,858\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x2 zxid:0x50 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/ids Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/ids (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:14:05,870\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x3 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:14:06,547\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x1b zxid:0x55 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/owners/test2 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/owners/test2 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:14:06,549\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x1c zxid:0x56 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/owners (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:15:05,679\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x28 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/0 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/0 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:15:05,681\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x29 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:15:05,747\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x2d zxid:0x61 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/2 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/2 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:15:05,754\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x2e zxid:0x62 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-43150/offsets/test2 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:15:05,771\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x30 zxid:0x64 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/1 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/1 (org.apache.zookeeper.server.PrepRequestProcessor)  
\[2015-06-20 21:15:05,774\] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x31 zxid:0x65 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-43150/offsets/test2 (org.apache.zookeeper.server.PrepRequestProcessor)  
hello world  
hadoop hive hbase i am baolibin  
  
  
  
在消费者这里会接收到：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-console-consumer.sh --zookeeper 192.168.1.200:2181 --topic test2 --from-beginning  
hello world  
hadoop hive hbase i am baolibin  
  
  
12、关闭kafka：  
关闭broker：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-server-stop.sh config/server.properties &  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/kafka-server-stop.sh config/server1.properties &  
  
  
关闭zookeeper：  
\[root@baozi kafka\_2.10-0.8.2.1\]\# bin/zookeeper-server-stop.sh config/zookeeper.properties >/dev/null 2>&1 &  
  
  
\[root@baozi kafka\_2.10-0.8.2.1\]\# jps  
3751 Jps

\[root@baozi kafka\_2.10-0.8.2.1\]\#

[http_kafka.apache.org_downloads.html]: http://kafka.apache.org/downloads.html