ELK之Elasticsearch、logstash部署及配置
ElasticSearch是一个搜索引擎,用来搜索、分析、存储日志;node
Logstash用来采集日志,把日志解析为json格式交给ElasticSearch;linux
Kibana是一个数据可视化组件,把处理后的结果经过web界面展现;git
Beats在这里是一个轻量级日志采集器,早期的ELK架构中使用Logstash收集、解析日志,可是Logstash对内存、cpu、io等资源消耗比较高.相比 Logstash,Beats所占系统的CPU和内存几乎能够忽略不计.github
1.准备环境以及安装包web
hostname:linux-elk1 ip:10.0.0.22
hostname:linux-elk2 ip:10.0.0.33
# 两台机器保持hosts一致,以及关闭防火墙和selinux
cat /etc/hosts
10.0.0.22 linux-elk1
10.0.0.33 linux-elk2
systemctl disable firewalld.service
systemctl disable NetworkManager
echo "*/5 * * * * /usr/sbin/ntpdate time1.aliyun.com &>/dev/null" >>/var/spool/cron/root
systemctl restart crond.service
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-x86_64.rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.4.0.rpm
ll
total 342448
-rw-r--r-- 1 root root 33211227 May 15 2018 elasticsearch-5.4.0.rpm
-rw-r--r-- 1 root root 167733100 Feb 3 12:13 jdk-8u121-linux-x64.rpm
-rw-r--r-- 1 root root 56266315 May 15 2018 kibana-5.4.0-x86_64.rpm
-rw-r--r-- 1 root root 93448667 May 15 2018 logstash-5.4.0.rpm
# 安装配置elasticsearch
yum -y install jdk-8u121-linux-x64.rpm elasticsearch-5.4.0.rpm
grep "^[a-Z]" /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-cluster
node.name: elk-node1
path.data: /data/elkdata
path.logs: /data/logs
bootstrap.memory_lock: true
network.host: 10.0.0.22
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.0.0.22", "10.0.0.33"]
mkdir /data/{elkdata,logs}
chown -R elasticsearch.elasticsearch /data
# 打开启动脚本中的某行/usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity
# 修改最大、最小内存为3g
grep -vE "^$|#" /etc/elasticsearch/jvm.options
-Xms3g
-Xmx3g
...
systemctl restart elasticsearch.service
tail /data/logs/elk-cluster.log # 最后几行有个started时,说明服务已经启动
访问http://10.0.0.22:9200/,出现es信息,说明服务正常
在linux上验证es服务正常的命令:curl http://10.0.0.22:9200/_cluster/health?pretty=true
2.安装es插件headdocker
wget https://nodejs.org/dist/v8.10.0/node-v8.10.0-linux-x64.tar.xz
tar xf node-v8.10.0-linux-x64.tar.xz
mv node-v8.10.0-linux-x64 /usr/local/node
vim /etc/profile
export NODE_HOME=/usr/local/node
export PATH=$PATH:$NODE_HOME/bin
source /etc/profile
which node
/usr/local/node/bin/node
which npm
/usr/local/node/bin/npm
npm install -g cnpm --registry=https://registry.npm.taobao.org # 生成了一个cnpm
/usr/local/node/bin/cnpm -> /usr/local/node/lib/node_modules/cnpm/bin/cnpm
npm install -g grunt-cli --registry=https://registry.npm.taobao.org # 生成了一个grunt
/usr/local/node/bin/grunt -> /usr/local/node/lib/node_modules/grunt-cli/bin/grunt
wget https://github.com/mobz/elasticsearch-head/archive/master.zip
unzip master.zip
cd elasticsearch-head-master/
vim Gruntfile.js
90 connect: {
91 server: {
92 options: {
93 hostname: '10.0.0.22',
94 port: 9100,
95 base: '.',
96 keepalive: true
97 }
98 }
99 }
vim _site/app.js
4360行将"http://localhost:9200"改成"http://10.0.0.22:9200";
cnpm install
grunt --version
vim /etc/elasticsearch/elasticsearch.yml # 增长以下两行
http.cors.enabled: true
http.cors.allow-origin: "*"
systemctl restart elasticsearch
systemctl enable elasticsearch
grunt server &
在elasticsearch 2.x之前的版本能够经过:npm
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head来安装head插件;json
在elasticsearch 5.x以上版本须要经过npm进行安装bootstrap
docker方式启动插件(前提是装好了es)vim
yum -y install docker systemctl start docker && systemctl enable docker docker run -p 9100:9100 mobz/elasticsearch-head:5
报错:Fatal error: Unable to find local grunt.
网上大多解决方法是:npm install grunt --save-dev,这是没有装grunt的时候的解决办法,在上面的步骤中已经安装了grunt,只是启动时没有在项目目录中,到项目目录执行该命令grunt server &便可.
3.logstash部署及基本语法
cd /usr/local/src
yum -y install logstash-5.4.0.rpm
# 使用rubydebug方式前台输出展现以及测试
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout { codec => rubydebug} }'
The stdin plugin is now waiting for input:
00:37:04.711 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
hello
{
"@timestamp" => 2019-02-03T16:43:48.149Z,
"@version" => "1",
"host" => "linux-elk1",
"message" => "hello"
}
# 测试输出到文件
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { file { path => "/tmp/test-%{+YYYY.MM.dd}.log"} }'
# 开启gzip压缩输出
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output{ file { path => "/tmp/test-%{+YYYY.MM.dd}.log.tar.gz" gzip => true } }'
# 测试输出到elasticsearch
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { elasticsearch { hosts => ["10.0.0.22:9200"] index => "logstash-test-%{+YYYY.MM.dd}" } }'
systemctl enable logstash
systemctl restart logstash
在删除数据时,在该界面删除,切勿在服务器目录上删除,由于集群节点上都有这样的数据,删除某一个,可能会致使elasticsearch没法启动.
Elasticsearch环境准备-参考博客:http://blog.51cto.com/jinlong/2054787
logstash部署及基本语法-参考博客:http://blog.51cto.com/jinlong/2055024