bigdata-02-kafka

下载：http://kafka.apache.org/downloads
安装：tar -zxvf kafka_2.11-0.9.0.0.tgz -C ~/app

配置环境变量
    vi ~/.bash_profile

    export KAFKA_HOME=/Users/liujinjie/app/kafka_2.11-0.9.0.0
    export PATH=$KAFKA_HOME/bin:$PATH

使环境变量生效
    source ~/.bash_profile

~~~~~~~~~~~~~~~~~~~~~单节点单broker的部署(一个机器上配置一个broker)~~~~~~~~~~~~~~~~~~~~~~

修改配置文件
$KAFKA_HOME/config/server.properties
    broker.id=0   # 一个kafka一个实例
    listeners=PLAINTEXT://:9092
    host.name=localhost
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs
    zookeeper.connect=localhost:2181

官方启动文档
http://kafka.apache.org/quickstart

kafka-server-start.sh
USAGE: /Users/liujinjie/app/kafka_2.11-0.9.0.0/bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*

一、启动
kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

二、建立topic
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello_topic

三、发送消息
kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic-180216-01

四、消费消息

（1）kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic-180216-01

（2）kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

--from-beginning 每次都从头消费

# 老版本的写法
kafka-console-consumer.sh --zookeeper localhost:2181 --topic test-topic-180216-01

五、查看描述
kafka-topics.sh --describe --zookeeper localhost:2181

返回
Topic:hello_topic    PartitionCount:1    ReplicationFactor:1    Configs:
    Topic: hello_topic    Partition: 0    Leader: 0    Replicas: 0    Isr: 0

六、查看全部topic

kafka-topics.sh --list --zookeeper localhost:2181

七、根据topic名称查看内容

kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello_topic

~~~~~~~~~~~~~~~~~~~~~~单节点多broker部署(一个机器上部署多个broker)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

cd $KAFKA_HOME/conf
cp server.proerties server-1.properties
cp server.proerties server-2.properties
cp server.proerties server-3.properties

vi server-1.properties
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs-1

vi server-2.properties
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs-2

vi server-1.properties
    broker.id=3
    listeners=PLAINTEXT://:9095
    log.dirs=/Users/liujinjie/app/tmp/kafka-logs-3

启动
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &

建立topic
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

看topic详情
    kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

返回
Topic:my-replicated-topic    PartitionCount:1    ReplicationFactor:3    Configs:
    Topic: my-replicated-topic    Partition: 0    Leader: 1    Replicas: 1,3,2    Isr: 1,3,2

PartitionCount:1         分区数量是1
ReplicationFactor:3     副本系数是3
Leader: 1                 broker-id=1是leader
Replicas:1,3,2           3个副本
Isr: 1,3,2              活着的3个

启动生产者
kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095 --topic my-replicated-topic

消费消息
kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic

Now let's test out fault-tolerance. Broker 1 was acting as the leader so let's kill it:

> ps aux | grep server-1.properties 7564 ttys002    0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java... > kill -9 7564