kafka中 FailedToSendMessageException解决

这两天在学kafka,把官网上的demo部署到本身的虚拟机上运行,正常;

而后部署到公司的研发线主机上,发现producer一直没法发送消息;

部分错误日志以下:

[2014-11-13 09:58:09,660] WARN Error while fetching metadata [{TopicMetadata for topic mor ->
No partition metadata for topic mor due to kafka.common.LeaderNotAvailableException}] for topic [mor]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2014-11-13 09:58:09,660] ERROR Failed to send requests for topics mor with correlation ids in [17,24] (kafka.producer.async.DefaultEventHandler)
[2014-11-13 09:58:09,660] ERROR Error in handling batch of 17 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
        at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
        at scala.collection.immutable.Stream.foreach(Stream.scala:526)
        at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
        at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

操做步骤彻底同样,只是在研发线上修改了zookeeper的监听端口,输入命令时也是按照对应的端口输入的,因此不是参数有问题致使的;

对比了研发线和虚拟机上的配置,发现除了这个端口以外,没有区别;

鼓捣了一天,没找到问题出在哪里;

次日来上班,把server.properties文件里

#host.name=localhost

的注释去掉,而后运行,发现问题解决了;

(可是在虚拟机里明明没有修改这个东西也是能够正常运行的;

怀疑是发行版不一样,一些内部的配置有区别致使的;

虚拟机:

Distributor ID:    Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:    trusty

研发线:

LSB Version:    :
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 5.4 (Tikanga)
Release:        5.4
Codename:       Tikanga

)

---

在解决问题的过程当中,还发现其余的问题,可是目前没发现影响运行;

就是zookeeper和broker跑起来以后,建立producer,建立话题,建立Consumer,的时候,zookeeper都会报异常,部分记录以下:

[2014-11-13 09:12:13,486] INFO Got user-level KeeperException when processing sessionid:0x149a6b3e36c0001 type:setData cxid:0x3 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config/topics/morning Error:KeeperErrorCode = NoNode for /config/topics/morning (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:12:13,506] INFO Got user-level KeeperException when processing sessionid:0x149a6b3e36c0001 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:12:13,535] INFO Processed session termination for sessionid: 0x149a6b3e36c0001 (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 09:19:50,958] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:50,982] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:50,998] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:51,295] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:setData cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/controller_epoch Error:KeeperErrorCode = NoNode for /controller_epoch (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 09:19:51,374] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf50000 type:delete cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)

[2014-11-13 10:31:50,651] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf5001a type:setData cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/test-consumer-group/offsets/mor/0 Error:KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets/mor/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2014-11-13 10:31:50,661] INFO Got user-level KeeperException when processing sessionid:0x149a6bbeaf5001a type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/test-consumer-group/offsets Error:KeeperErrorCode = NoNode for /consumers/test-consumer-group/offsets (org.apache.zookeeper.server.PrepRequestProcessor)

一开始producer没法发送消息,觉得是和这些异常有关,可是在虚拟机里运行的时候,一样出现这些异常,可是也没有影响producer发送消息;

在网上查了半天,有人说是不正确关闭zookeeper和server的缘由,有人说是没有删除/tmp下的zookeeper日志和kafka日志的缘由,

总之,我试了上面提到的方法,仍是报这些异常;

若是有哪位仁兄知道这些异常是什么缘由引发的,请告诉我,十分感谢;

---

另外,从本机运行java代码的Consumer链接研发线的时候,链接迅速被关闭了,收不到producer发来的消息;

缘由是配置的超时时间过短,zookeeper没有读完Consumer的数据,链接就被Consumer断开了,部分日志以下:

[2014-11-13 10:28:47,989] INFO Accepted socket connection from /192.168.50.33:2676 (org.apache.zookeeper.server.NIOServerCnxn)
[2014-11-13 10:28:47,989] WARN EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)

‍解决办法,就是把配置里的超时时间配置的长点儿,以下:

props.put("zookeeper.session.timeout.ms", "400000");

---

对了,还有个问题,kafka官网上关于java版producer部分的代码,有个地方写的不是很明确,以下:

props.put("metadata.broker.list", "broker1:9092,broker2:9092");

 

broker1和broker2表明broker对应的hostname,而不是broker的id.正确示例以下:

 props.put("metadata.broker.list", "localhost:9092,localhost:9093");