Centos7编译安装Kafka-manager-2.0.0.2

时间 2019-12-26

标签 centos7 centos 编译安装 kafka manager 2.0.0.2 栏目 Tomcat 繁體版

原文原文链接

1、Kafka manager简介

项目地址为：https://github.com/yahoo/kafka-managerjava

为了简化开发者和服务工程师维护Kafka集群的工做，yahoo构建了一个叫作Kafka管理器的基于Web工具，叫作 Kafka Manager。这个管理工具能够很容易地发现分布在集群中的哪些topic分布不均匀，或者是分区在整个集群分布不均匀的的状况。它支持管理多个集群、选择副本、副本从新分配以及建立Topic。同时，这个管理工具也是一个很是好的能够快速浏览这个集群的工具，有以下功能：node

管理kafka集群
方便集群状态监控 (包括topics, consumers, offsets, brokers, replica distribution, partition distribution)
方便选择分区副本
配置分区任务，包括选择使用哪些brokers
能够对分区任务重分配
提供不一样的选项来建立及删除topic
Topic list会指明哪些topic被删除
批量产生分区任务而且和多个topic和brokers关联
批量运行多个主题对应的多个分区
向已经存在的主题中添加分区
对已经存在的topic修改配置
能够在broker level和topic level的度量中启用JMX polling功能
能够过滤在ZK上没有ids/ owners/offsets/ directories的consumer

2、下载kafka-manager源码包

源码包下载地址：https://github.com/yahoo/kafka-manager/archive/2.0.0.2.tar.gzgit

github上没有提供安装包，须要咱们自行编译以后在进行安装，须要如今sbt编译工具。github

再此我已经编译好安装包了kafka-manager-2.0.0.2.zip，能够直接下载使用：https://pan.baidu.com/s/1XukGHykub5OdJvyeudqVPg 提取码：w4q4web

看到我这篇文章，就不要去编译，没意思，直接从百度网盘连接下载编译好的安装包就好了，编译步骤参考一下就好了。apache

3、安装sbt-1.3.5

[root@localhost ~]# curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo
[root@localhost ~]# mv bintray-sbt-rpm.repo /etc/yum.repos.d/
[root@localhost ~]# yum install sbt -y

由于Kafka-manager使用的Play框架，为了编译的速度更快，先配置sbt的maven仓库，因为默认仓库速度较慢，所以使用aliyun提供的maven仓库。api

修改仓库地址：（sbt 默认下载库文件很慢, 还时不时被打断，不行的话就重试），咱们能够在用户目录下建立 touch ~/.sbt/repositories, 填上阿里云的镜像 # vi ~/.sbt/repositories 服务器

cd ~
mkdir .sbt
touch ~/.sbt/repositories
vi ~/.sbt/repositories

内容：app

[repositories]
  local
  #oschina: http://maven.oschina.net/content/groups/public/  
  aliyun-nexus: http://maven.aliyun.com/nexus/content/groups/public/
  jcenter: http://jcenter.bintray.com/
  typesafe-ivy-releases: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
  maven-central: http://repo1.maven.org/maven2/

以上配置文件解释顺序是：本地→阿里云镜像→jcenter→typesafe-ivy-releases→Maven主镜像。若是须要添加公司的maven镜像，能够按照 key: value 的形式添加，key 的命名没有要求（暂时没注意到，可是最好也不要用什么特殊符号吧）框架

验证：检查sbt是否安装成功，查看命令输出，发现已经成功能够从maven.aliyun.com/nexus下载到依赖即表示成功

sbt -version

须要很长时间~耐心等待，我这里已经执行过了，再次执行显示以下：

[root@localhost ~]# sbt -version
[info] [launcher] getting org.scala-sbt sbt 1.3.5  (this may take some time)...
:: loading settings :: url = jar:file:/usr/share/sbt/bin/sbt-launch.jar!/org/apache/ivy/core/settings/ivysettings.xml
:: retrieving :: org.scala-sbt#boot-app
	confs: [default]
	81 artifacts copied, 0 already retrieved
[info] [launcher] getting Scala 2.12.10 (for sbt)...
:: retrieving :: org.scala-sbt#boot-scala
	confs: [default]
	6 artifacts copied, 0 already retrieved
sbt version in this project: 1.3.5
sbt script version: 1.3.5

4、解压编译kafka-manager源码包

在【2、下载kafka-manager源码包】步骤中咱们已经已下载了源码包，

解压kafka-manager源码包：

[root@localhost soft]# tar -zxvf kafka-manager-2.0.0.2.tar.gz

解压后显示目录以下

[root@localhost soft]# ll
总用量 56
drwxrwxr-x. 9 root root   109 4月  12 2019 app
-rw-rw-r--. 1 root root  4242 4月  12 2019 build.sbt
drwxrwxr-x. 2 root root   108 4月  12 2019 conf
drwxrwxr-x. 2 root root   156 4月  12 2019 img
-rw-rw-r--. 1 root root 11307 4月  12 2019 LICENSE
drwxrwxr-x. 2 root root    49 4月  12 2019 project
drwxrwxr-x. 5 root root    54 4月  12 2019 public
-rw-rw-r--. 1 root root  8686 4月  12 2019 README.md
-rwxrwxr-x. 1 root root 21353 4月  12 2019 sbt
drwxrwxr-x. 4 root root    37 4月  12 2019 src
drwxrwxr-x. 5 root root    51 4月  12 2019 test

而后执行：

./sbt clean dist

编译时间会很长，须要耐心等待，能够到~/.sbt/boot/update.log 查看sbt更新日志。sbt更新好，就开始下载各类jar包，最后看到：[info] Your package is ready in /home/soft/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip 证实编译好了。

我这个已经编译过了，因为记录笔记，我再次进行了编译就很快了，日志以下：

[root@localhost kafka-manager-2.0.0.2]# ./sbt clean dist
Downloading sbt launcher for 1.2.8:
  From  http://repo.scala-sbt.org/scalasbt/maven-releases/org/scala-sbt/sbt-launch/1.2.8/sbt-launch.jar
    To  /root/.sbt/launchers/1.2.8/sbt-launch.jar
Getting org.scala-sbt sbt 1.2.8  (this may take some time)...
:: retrieving :: org.scala-sbt#boot-app
	confs: [default]
	79 artifacts copied, 0 already retrieved (28496kB/1360ms)
Getting Scala 2.12.7 (for sbt)...
:: retrieving :: org.scala-sbt#boot-scala
	confs: [default]
	5 artifacts copied, 0 already retrieved (19715kB/347ms)
[info] Loading settings for project kafka-manager-2-0-0-2-build from plugins.sbt ...
[info] Loading project definition from /home/soft/kafka-manager-2.0.0.2/project
[info] Updating ProjectRef(uri("file:/home/soft/kafka-manager-2.0.0.2/project/"), "kafka-manager-2-0-0-2-build")...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Loading settings for project root from build.sbt ...
[info] Set current project to kafka-manager (in build file:/home/soft/kafka-manager-2.0.0.2/)
[success] Total time: 0 s, completed 2019-12-25 12:27:23
[info] Packaging /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/kafka-manager_2.12-2.0.0.2-sources.jar ...
[info] Done packaging.
Warning: node.js detection failed, sbt will use the Rhino based Trireme JavaScript engine instead to run JavaScript assets compilation, which in some cases may be orders of magnitude slower than using node.js.
[info] Updating ...
[info] downloading http://maven.aliyun.com/nexus/content/groups/public/org/scala-lang/modules/scala-parser-combinators_2.12/1.0.7/scala-parser-combinators_2.12-1.0.7.jar ...
[info] 	[SUCCESSFUL ] org.scala-lang.modules#scala-parser-combinators_2.12;1.0.7!scala-parser-combinators_2.12.jar(bundle) (2108ms)
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Wrote /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/kafka-manager_2.12-2.0.0.2.pom
[info] Main Scala API documentation to /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/api...
[info] Non-compiled module 'compiler-bridge_2.12' for Scala 2.12.8. Compiling...
[info]   Compilation completed in 38.745s.
model contains 604 documentable templates
[info] Main Scala API documentation successful.
[info] Compiling 131 Scala sources and 2 Java sources to /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/classes ...
[info] Done compiling.
[info] Packaging /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/kafka-manager_2.12-2.0.0.2-javadoc.jar ...
[info] Done packaging.
[info] LESS compiling on 1 source(s)
[info] Packaging /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/kafka-manager_2.12-2.0.0.2.jar ...
[info] Done packaging.
[info] Packaging /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/kafka-manager_2.12-2.0.0.2-web-assets.jar ...
[info] Done packaging.
[info] Packaging /home/soft/kafka-manager-2.0.0.2/target/scala-2.12/kafka-manager_2.12-2.0.0.2-sans-externalized.jar ...
[info] Done packaging.
[success] All package validations passed
[info] Your package is ready in /home/soft/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip
[success] Total time: 355 s, completed 2019-12-25 12:33:19

5、安装kafka-manager

因为使用kafka-manager是在JDK8基础上的，因此先安装JDK8，JDK安装再也不叙述。

因为【4、解压编译kafka-manager源码包】步骤已经编译完成，咱们能够把 /home/soft/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip 的安装包移动到你要安装的地方解压。

在此我解压到/usr/local/目录下

unzip kafka-manager-2.0.0.2.zip -d /usr/local/

解压后查看目录以下：

[root@localhost kafka-manager-2.0.0.2]# ll
总用量 28
drwxr-xr-x. 2 root root 4096 12月 25 14:57 bin
drwxr-xr-x. 2 root root  108 12月 25 14:57 conf
drwxr-xr-x. 2 root root 8192 12月 25 14:57 lib
-rw-r--r--. 1 root root 8686 4月  12 2019 README.md
drwxr-xr-x. 3 root root   17 12月 25 14:57 share

接下来就是配置kafka-manager了

vi conf/application.conf 
#修改kafka-manager.zkhosts列表为本身的zk节点
kafka-manager.zkhosts="192.168.184.133:2181"
#添加http访问端口配置，默认9000
http.port=9090

想要看到读取,写入速度须要开启JMX，修改kafka-server-start.sh 添加一行便可：添加JMX端口8999

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="8999"
fi

注意：每一个kafka broker都须要修改，修改后进行重启kafka。

最后，启动kafka-manager

cd kafka-manager/bin

nohup ./kafka-manager -Dconfig.file=../conf/application.conf >/dev/null 2>&1 &

6、测试kafka-manager

访问：http://192.168.184.133:9090/

一、新建Cluster

点击【Cluster】>【Add Cluster】打开以下添加集群配置界面：输入集群的名字（如KafkaCluster）和 Zookeeper 服务器地址（如192.168.184.133:2181/kafka），选择最接近的Kafka版本

其余broker的配置能够根据本身须要进行配置，默认状况下，点击【保存】时，会提示几个默认值为1的配置错误，须要配置为>=2的值。提示以下。

添加完集群以后查看

topics相关：

更多页面功能本身点点就好了。。。

更多信息查看github上kafka-manager的README.md：https://github.com/yahoo/kafka-manager/blob/master/README.md