ELK安装过程
官方安装文档：https://www.elastic.co/guide/en/elasticsearch/reference/current/zip-targz.htmlhtml
官方硬件和配置项推荐：https://www.elastic.co/guide/en/elasticsearch/guide/master/hardware.html前端
事件--->input---->codec--->filter--->codec--->outputjava
二、环境设置
系统：Centos7.4 IP地址：11.11.11.30 JDK:1.8 Elasticsearch-6.4.3 Logstash-6.4.0 kibana-6.4.0
注释：截至2018年11月8日最新版为6.4.3
iptables -F
2.一、前期准备
一、安装JDK
JDK的下载地址：https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlnode
[root@localhost ~]# tar zxvf jdk-8u152-linux-x64.tar.gz -C /usr/local/ 解压后配置全局变量 [root@localhost ~]# vi /etc/profile.d/jdk.sh JAVA_HOME=/usr/local/jdk1.8 CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib PATH=$JAVA_HOME/bin:$PATH export JAVA_HOME CLASSPATH PATH [root@localhost ~]# source /etc/profile.d/jdk.sh [root@localhost ~]# java -version java version "1.8.0_152" Java(TM) SE Runtime Environment (build 1.8.0_152-b16) Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
注释：也能够直接yum安装java也是能够的！！！ linux
二、修改全局参数
vi /etc/sysctl.conf vm.max_map_count=655360 sysctl -p vi /etc/security/limits.conf * soft nproc 65536 * hard nproc 65536 * soft nofile 65536 * hard nofile 65536
#如下两个参数不是必须的，缘由在于配置第二台elk的时候启动elasticsearch的时候报JVM内存没法分配内存错误
* soft memlock unlimited * hard memlock unlimited
三、建立elk用户及下载所需文件
#建立运行ELK的用户 useradd elk passwd elk #建立目录存放软件 [root@elk ~]# su - elk
[elk@elk ~]$ mkdir /home/elk/{Application,Data,Log}
注释： 软件安装目录：/home/elk/Application 软件数据目录：/home/elk/Data 软件日志目录：/home/elk/Log #下载并解压kibana，elasticsearch，logstash到/home/elk/Application目录下 下载地址：https://www.elastic.co/start #因最新版本下载过慢，下载以前同事已经下载好的...... #解压文件 [elk@elk ~]$ cd /usr/local/src/ [elk@elk src]$ ll 总用量 530732 -rw-r--r--. 1 root root 97872736 11月 7 14:55 elasticsearch-6.4.3.tar.gz -rw-r--r--. 1 root root 10813704 11月 7 15:41 filebeat-6.4.0-x86_64.rpm -rw-r--r--. 1 root root 55751827 11月 7 15:41 kafka_2.11-2.0.0.tgz -rw-r--r--. 1 root root 187936225 11月 7 15:39 kibana-6.4.0-linux-x86_64.tar.gz -rw-r--r--. 1 root root 153887188 11月 7 15:41 logstash-6.4.0.tar.gz -rw-r--r--. 1 root root 37191810 11月 7 15:41 zookeeper-3.4.13.tar.gz
[elk@elk src]$ tar xf elasticsearch-6.4.3.tar.gz -C /home/elk/Application/ [elk@elk src]$ tar xf kibana-6.4.0-linux-x86_64.tar.gz -C /home/elk/Application/ [elk@elk src]$ tar xf logstash-6.4.0.tar.gz -C /home/elk/Application/
[elk@elk src]$ ll /home/elk/Application/ 总用量 0 drwxr-xr-x. 8 elk elk 143 10月 31 07:22 elasticsearch-6.4.3 drwxrwxr-x. 11 elk elk 229 8月 18 07:50 kibana-6.4.0-linux-x86_64 drwxrwxr-x. 12 elk elk 255 11月 7 15:46 logstash-6.4.0 [elk@elk src]$
三、服务配置
3.一、配置elasticsearch
#内存大的话能够根据需求要修改 [elk@elk Application]$ vi elasticsearch/config/jvm.options -Xms8g -Xmx8g [elk@elk Application]$ vi elasticsearch/config/elasticsearch.yml #集群时使用的集群名称 cluster.name: my-es #本节点名称 node.name: node-1 #数据存储的目录 #path.data: /data/es-data #多个目录能够用逗号隔开 path.data: /home/elk/Data/elasticsearch #日志存放目录 path.logs: /home/elk/Log/elasticsearch #数据不容许放入sawp分区 bootstrap.memory_lock: true #本地监听地址 network.host: 11.11.11.30
#设置集群中master节点的初始列表，能够经过这些节点来自动发现新加入的集群节点
discovery.zen.ping.unicast.hosts: ["11.11.11.30", "11.11.11.31"]
#监听端口 http.port: 9200
# 设置节点间交互的tcp端口（集群）,(默认9300)
transport.tcp.port: 9300
#增长参数，使head插件能够访问es
http.cors.enabled: true
http.cors.allow-origin: "*"
#看下本身修改过的配置 [elk@elk Application]$ grep '^[a-Z]' elasticsearch/config/elasticsearch.yml 启动elasticsearch(要使用elk这个用户启动) nohup /home/elk/Application/elasticsearch/bin/elasticsearch >> /home/elk/Log/elasticsearch/elasticsearch.log 2>&1 & 日志地址： [elk@elk Application]$ tail /home/elk/Log/elasticsearch/my-es.log
3.1.一、部署第二台elk，修改的配置文件以下，其余跟elk配置一致
[elk@elk02 Application]$ grep '^[a-Z]' elasticsearch/config/elasticsearch.yml cluster.name: my-es #必定要跟第一台一致 node.name: node-2 #必定要跟第一台不一样 path.data: /home/elk/Data/elasticsearch path.logs: /home/elk/Log/elasticsearch bootstrap.memory_lock: true network.host: 11.11.11.31 http.port: 9200 transport.tcp.port: 9300 http.cors.enabled: true http.cors.allow-origin: "*"
mkdir /home/elk/Log/elasticsearch/
启动elasticsearch nohup /home/elk/Application/elasticsearch/bin/elasticsearch >> /home/elk/Log/elasticsearch/elasticsearch.log 2>&1 &
3.1.二、根据附录中的1，安装完head后的结果以下图
3.二、配置logstash（拓展请查看附录3）
这是收集数据，转发数据，对日志进行过滤ios
logstash官方说明：https://www.elastic.co/guide/en/logstash/current/introduction.htmlnginx
写入插件：https://www.elastic.co/guide/en/logstash/current/input-plugins.htmlgit
输出插件：https://www.elastic.co/guide/en/logstash/current/output-plugins.htmlgithub
[elk@elk Application]$ vi logstash/config/jvm.options -Xms4g -Xmx4g [elk@elk Application]$ vi logstash/config/logstash.yml #pipeline线程数，默认为cpu核数
pipeline.workers: 8
#batcher一次批量获取的待处理文档数 pipeline.batch.size: 1000
#batcher等待时长，默认50ms pipeline.batch.delay: 50
示例：web
#logstash启动的时候要读取的配置文件
[elk@elk config]$ cat logstash-sample.conf
#从哪获取数据 input { file { path => ["/var/log/messages","/var/log/secure"] type => "system-log" start_position => "beginning" } }
#对进来的文件内容进行过滤 filter { }
#过滤后的数据发到哪里 output { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "system-log-%{+YYYY.MM}" #user => "elastic" #password => "changeme" } }
注释：若是日志量不大的话，不建议使用天计算（+YYYY.MM.dd）；kikana到时很差添加！！！！
#启动
[elk@elk ~]$ mkdir /home/elk/Log/logstash -p
nohup /home/elk/Application/logstash/bin/logstash -f /home/elk/Application/logstash/config/logstash-sample.conf --config.reload.automatic >> /home/elk/Log/logstash/logstash.log 2>&1 &
#查看是否运行
[elk@elk config]$ jobs -l
[1]+ 16289 运行中 nohup /home/elk/Application/logstash/bin/logstash -f /home/elk/Application/logstash/config/logstash-sample.conf --config.reload.automatic >> /home/elk/Log/logstash/logstash.log 2>&1 &
1 [elk@elk Application]$ vi logstash/config/logstash.conf 2 input { 3 kafka { 4 bootstrap_servers => "192.168.2.6:9090" 5 topics => ["test"] 6 } 7 8 } 9 filter{ 10 json{ 11 source => "message" 12 } 13 14 ruby { 15 code => "event.set('index_day_hour', event.get('[@timestamp]').time.localtime.strftime('%Y.%m.%d-%H'))" 16 } 17 mutate { 18 rename => { "[host][name]" => "host" } 19 } 20 mutate { 21 lowercase => ["host"] 22 } 23 24 25 } 26 output { 27 elasticsearch { 28 index => "%{host}-%{app_id}-%{index_day_hour}" 29 hosts => ["http://192.168.2.7:9200"] 30 } 31 }
3.三、配置kibana（过滤file）
[elk@elk Application]$ vi kibana/config/kibana.yml server.port: 15045 #页面访问端口 server.host: "11.11.11.30" #监听IP地址 elasticsearch.url: "http://11.11.11.30:9200" elasticsearch.pingTimeout: 1500
[root@elk ~]# grep '^[a-Z]' /home/elk/Application/kibana/config/kibana.yml
#服务启动的端口 server.port: 15045
#监听地址
server.host: "11.11.11.30" #elas端口，必定要对上
elasticsearch.url: "http://11.11.11.30:9200" #kibana的日志也写入els内，这是个索引的名字，会在head上显示
kibana.index: ".kibana" elasticsearch.pingTimeout: 1500 [root@elk ~]#
[root@elk ~]# mkdir /home/elk/Log/kibana -p
[root@elk ~]# nohup /home/elk/Application/kibana/bin/kibana >> /home/elk/Log/kibana/kibana.log 2>&1 &
[root@elk ~]# netstat -luntp
tcp 0 0 11.11.11.30:15045 0.0.0.0:* LISTEN 16437/node
tcp6 0 0 :::9100 :::* LISTEN 14477/grunt
tcp6 0 0 11.11.11.30:9200 :::* LISTEN 15451/java
tcp6 0 0 11.11.11.30:9300 :::* LISTEN 15451/java
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 16289/java
注释：星星是主，圆圈是副
node-2这台主机重启后新启动了系统，可是在head显示不出来，重启下node-1的els就能够看到了。。。。。！
四、使用
4.一、添加索引
4.二、logstash过滤java日志
[root@elk ~]# cat /home/elk/Application/logstash/config/logstash.conf input { file { path => "/var/log/messages" type => "system-log" start_position => "beginning" } file { path => "/home/elk/Log/elasticsearch/my-es.log" type => "es-log" } file { path => "/home/elk/Log/logstash/logstash.log" type => "logstash-log" } } output { if [type] == "system-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "system-log-%{+YYYY.MM}" #user => "elastic" #password => "changeme" } } if [type] == "es-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "es-log-%{+YYYY.MM}" } } if [type] == "logstash-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "logstash-log-%{+YYYY.MM}" } } } [root@elk ~]# [root@elk ~]# nohup /home/elk/Application/logstash/bin/logstash -f /home/elk/Application/logstash/config/logstash.conf --config.reload.automatic >> /home/elk/Log/logstash/logstash.log 2>&1 & [root@elk ~]# netstat -luntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 874/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1112/master tcp 0 0 11.11.11.30:15045 0.0.0.0:* LISTEN 16437/node tcp6 0 0 :::9100 :::* LISTEN 14477/grunt tcp6 0 0 11.11.11.30:9200 :::* LISTEN 15451/java tcp6 0 0 11.11.11.30:9300 :::* LISTEN 15451/java tcp6 0 0 :::22 :::* LISTEN 874/sshd tcp6 0 0 ::1:25 :::* LISTEN 1112/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 20746/java udp 0 0 127.0.0.1:323 0.0.0.0:* 526/chronyd udp6 0 0 ::1:323 :::* 526/chronyd [root@elk ~]#
4.2.一、java日志的处理（多行归一行）
多行并一行，官方地址：https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html
测试规则的可用性
[elk@elk config]$ cat codec.conf input { stdin { codec => multiline{ pattern => "^\[" negate => true what => "previous" } } } filter { } output { stdout { codec => rubydebug } } [elk@elk config]$ [root@elk logstash]# /home/elk/Application/logstash/bin/logstash -f /home/elk/Application/logstash/config/codec.conf
用于处理真正的logstash配置：
#删除已有的缓存，删除后会从新收集 [root@elk ~]# find / -name .sincedb* /home/elk/Application/logstash/data/plugins/inputs/file/.sincedb_97cbda73a2aaa9193a01a3b39eb761f3 /home/elk/Application/logstash/data/plugins/inputs/file/.sincedb_5227a954e2d5a4a3f157592cbe63c166 /home/elk/Application/logstash/data/plugins/inputs/file/.sincedb_452905a167cf4509fd08acb964fdb20c [root@elk ~]# rm -rf /home/elk/Application/logstash/data/plugins/inputs/file/.sincedb* #修改事后的文件以下 [elk@elk config]$ cat logstash.conf input { file { path => "/var/log/messages" type => "system-log" start_position => "beginning" } file { path => "/home/elk/Log/elasticsearch/my-es.log" type => "es-log" start_position => "beginning" codec => multiline{ pattern => "^\[" negate => true what => "previous" } } } output { if [type] == "system-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "system-log-%{+YYYY.MM}" #user => "elastic" #password => "changeme" } } if [type] == "es-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "es-log-%{+YYYY.MM}" } } if [type] == "logstash-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "logstash-log-%{+YYYY.MM}" } } } [elk@elk config]$ #使用root帐号启动 [root@elk config]# nohup /home/elk/Application/logstash/bin/logstash -f /home/elk/Application/logstash/config/logstash.conf >> /home/elk/Log/logstash/logstash.log 2>&1 & #刷新head
#els界面查看
五、收集nginx日志（转json格式）
关键在于logstash收集到的日志格式，必定要是json格式的
方法1：nginx配置日志格式，而后输入到els
log_format access_log_json '{"user_ip":"$http_x_real_ip","lan_ip":"$remote_addr","log_time":"$time_iso8601","user_req":"$request","http_code":"$status","body_bytes_sent":"$body_bytes_sent","req_time":"$request_time","user_ua":"$http_user_agent"}'; access_log /var/log/nginx/access_json.log access_log_json;
方法2：文件直接获取，写入Redis 而后再使用Python脚本读取Redis，写成Json后再写入ELS
5.一、安装nignx并建立访问日志
[root@elk02 ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo [root@elk02 ~]# yum -y install nginx [root@elk02 ~]# systemctl start nginx.service [root@elk02 ~]# netstat -luntp [root@elk02 ~]# ab -n 1000 -c 1 http://11.11.11.31/ #访问一千次，并发为1 [root@elk02 ~]# tail -5 /var/log/nginx/access.log 11.11.11.31 - - [09/Nov/2018:18:09:08 +0800] "GET / HTTP/1.0" 200 3700 "-" "ApacheBench/2.3" "-" 11.11.11.31 - - [09/Nov/2018:18:09:08 +0800] "GET / HTTP/1.0" 200 3700 "-" "ApacheBench/2.3" "-" 11.11.11.31 - - [09/Nov/2018:18:09:08 +0800] "GET / HTTP/1.0" 200 3700 "-" "ApacheBench/2.3" "-" 11.11.11.31 - - [09/Nov/2018:18:09:08 +0800] "GET / HTTP/1.0" 200 3700 "-" "ApacheBench/2.3" "-" 11.11.11.31 - - [09/Nov/2018:18:09:08 +0800] "GET / HTTP/1.0" 200 3700 "-" "ApacheBench/2.3" "-" [root@elk02 ~]#
注释：用户真是IP 远程用户 时间 请求 状态码 发送数据大小 从哪跳转过来的 **** 获取代理的IP地址
[root@elk02 nginx]# vim nginx.conf
#添加以下日子规则******
log_format access_log_json '{"user_ip":"$http_x_real_ip","lan_ip":"$remote_addr","log_time":"$time_iso8601","user_req":"$request","http_code":"$status","body_bytes_sent":"$body_bytes_sent","req_time":"$request_time","user_ua":"$http_user_agent"}';
access_log /var/log/nginx/access.log access_log_json;
[root@elk02 nginx]# systemctl reload nginx [root@elk02 ~]# ab -n 1000 -c 1 http://11.11.11.31/ [root@elk02 nginx]# tailf /var/log/nginx/access_json.log {"user_ip":"-","lan_ip":"11.11.11.31","log_time":"2018-11-09T20:18:12+08:00","user_req":"GET / HTTP/1.0","http_code":"200","body_bytes_sent":"3700","req_time":"0.000","user_ua":"ApacheBench/2.3"} {"user_ip":"-","lan_ip":"11.11.11.31","log_time":"2018-11-09T20:18:12+08:00","user_req":"GET / HTTP/1.0","http_code":"200","body_bytes_sent":"3700","req_time":"0.000","user_ua":"ApacheBench/2.3"} {"user_ip":"-","lan_ip":"11.11.11.31","log_time":"2018-11-09T20:18:12+08:00","user_req":"GET / HTTP/1.0","http_code":"200","body_bytes_sent":"3700","req_time":"0.000","user_ua":"ApacheBench/2.3"} {"user_ip":"-","lan_ip":"11.11.11.31","log_time":"2018-11-09T20:18:12+08:00","user_req":"GET / HTTP/1.0","http_code":"200","body_bytes_sent":"3700","req_time":"0.000","user_ua":"ApacheBench/2.3"}
5.二、开始收集、分析
5.2一、收集测试
#logstash服务启动的端口是9600，若是已存在kill掉，使用这个nginx配置文件启动 [root@elk02 config]# pwd /home/elk/Application/logstash/config [root@elk02 config]# vim nginx.conf input { file { path => "/var/log/nginx/access_json.log" codec => "json" } } output { stdout { codec => rubydebug } } [root@elk02 config]# ../bin/logstash -f nginx.conf 而后使用11.30这台服务器过ad压力测试 [root@elk config]# ab -n 10 -c 1 http://11.11.11.31/ 在elk02即11.31上能够看到以下效果
5.2.二、写入配置els内
这里为了方便直接修改nginx.conf这个文件内了，实际最好写入到logstash.conf文件内
[root@elk02 config]# pwd /home/elk/Application/logstash/config [root@elk02 config]# cat nginx.conf input { file { type => "nginx-access-log" path => "/var/log/nginx/access_json.log" codec => "json" } } output { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "nginx-access-log-%{+YYYY.MM}" } } [root@elk02 config]#
[root@elk02 config]# ../bin/logstash -f nginx.conf #运行logstash指定使用nginx.conf这个配置文件内。
[root@elk ~]# ab -n 10 -c 1 http://11.11.11.31/ #必定要产生日志才能被发现，由于是从文件末尾开始收集的；
若是没有显示处理方法：
一、在原有的nginx.conf配置文件内添加以下：
input { file { type => "nginx-access-log" path => "/var/log/nginx/access_json.log" codec => "json" } } output { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "nginx-access-log-%{+YYYY.MM}"
}
stdout { codec => rubydebug } }
[root@elk02 config]# ../bin/logstash -f nginx.conf
[root@elk ~]# ab -n 10 -c 1 http://11.11.11.31/
二、找到它并删除它从新获取数据
[root@elk02 config]# find / -name .sincedb*
六、web页面kibana的使用
6.一、搜索关键值
6.二、视图的用法
6.2.一、makdown的使用方法
6.2.二、统计某一值得方法
6.2.三、统计IP的访问次数
6.2.四、可视化平台展现
七、日志收集
官方参考地址：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html
7.一、rsyslog系统日志
一、测试是否能收集到syslog日志{11.31}
[root@elk02 config]# cat syslog.conf input { syslog { type => "system-syslog" port => 514 } } output { stdout { codec => rubydebug } } [root@elk02 config]#
二、运行配置后查看端口是否开启{11.31}
[root@elk02 config]# pwd /home/elk/Application/logstash/config [root@elk02 config]# ../bin/logstash -f systlog.conf Sending Logstash logs to /home/elk/Application/logstash/logs which is now configured via log4j2.properties [2018-11-13T11:19:49,206][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-11-13T11:19:53,753][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.0"} [2018-11-13T11:20:08,746][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-11-13T11:20:10,815][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x64bb4375 run>"} [2018-11-13T11:20:12,403][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:514"} [2018-11-13T11:20:12,499][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:514"} [2018-11-13T11:20:12,749][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2018-11-13T11:20:14,078][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} #查看是否有9600和514端口 [root@elk02 config]# netstat -luntp
三、编辑/etc/rsyslog.conf日志{11.30}
#在最后添加以下一行 [root@elk ~]# tail -2 /etc/rsyslog.conf *.* @@11.11.11.31:514 # ### end of the forwarding rule ### [root@elk ~]# 注释：里边有些地址后边有横杠，表明有新的消息先写入内存缓存再写入对应的目录内。 #重启rsys的服务 [root@elk ~]# systemctl restart rsyslog.service #重启完成后就能在11.31上看到有新的消息输出了
看下是不是实时输出
7.一、syslog实际部署
[root@elk02 config]# cat systlog.conf input { syslog { type => "system-syslog" port => 514 } } output { elasticsearch { hosts => ["http://11.11.11.31:9200"] index => "system-syslog-%{+YYYY.MM}" } } [root@elk02 config]# pwd /home/elk/Application/logstash/config [root@elk02 config]#
[root@elk02 config]# ../bin/logstash -f systlog.conf
注释：而后看下11.31是否开启了514端口用于接收数据
7.二、TCP日志
测试可行性
[root@elk02 config]# cat tcp.conf input { tcp { type => "tcp" port => "6666" mode => "server" } } output { stdout { codec => rubydebug } } [root@elk02 config]# pwd /home/elk/Application/logstash/config [root@elk02 config]#
[root@elk02 config]# ../bin/logstash -f tcp.conf
注释：这时就能够查看是否有6666端口存在，存在则表明已经启动了。
写入ES内
[root@elk02 config]# cat tcp.conf input { tcp { type => "tcp" port => "6666" mode => "server" } } output { elasticsearch { hosts => ["http://11.11.11.31:9200"] index => "system-syslog-%{+YYYY.MM}" } } [root@elk02 config]# pwd /home/elk/Application/logstash/config [root@elk02 config]#
八、grok（httpd日志）
参考地址：https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
#查看httpd的日志格式
[root@elk ~]# yum -y install httpd [root@elk ~]# systemctl start httpd.service [root@elk conf]# pwd /etc/httpd/conf [root@elk conf]# vim httpd.conf LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common
注释： 访问IP地址 - 用户 时间戳
http://httpd.apache.org/docs/current/mod/mod_log_config.html
8.一、官方模板存参考位置：
/home/elk/Application/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns 下 [root@elk patterns]# ll 总用量 112 -rw-r--r--. 1 elk elk 1831 8月 18 08:23 aws -rw-r--r--. 1 elk elk 4831 8月 18 08:23 bacula -rw-r--r--. 1 elk elk 260 8月 18 08:23 bind -rw-r--r--. 1 elk elk 2154 8月 18 08:23 bro -rw-r--r--. 1 elk elk 879 8月 18 08:23 exim -rw-r--r--. 1 elk elk 10095 8月 18 08:23 firewalls -rw-r--r--. 1 elk elk 5338 8月 18 08:23 grok-patterns -rw-r--r--. 1 elk elk 3251 8月 18 08:23 haproxy -rw-r--r--. 1 elk elk 987 8月 18 08:23 httpd -rw-r--r--. 1 elk elk 1265 8月 18 08:23 java -rw-r--r--. 1 elk elk 1087 8月 18 08:23 junos -rw-r--r--. 1 elk elk 1037 8月 18 08:23 linux-syslog -rw-r--r--. 1 elk elk 74 8月 18 08:23 maven -rw-r--r--. 1 elk elk 49 8月 18 08:23 mcollective -rw-r--r--. 1 elk elk 190 8月 18 08:23 mcollective-patterns -rw-r--r--. 1 elk elk 614 8月 18 08:23 mongodb -rw-r--r--. 1 elk elk 9597 8月 18 08:23 nagios -rw-r--r--. 1 elk elk 142 8月 18 08:23 postgresql -rw-r--r--. 1 elk elk 845 8月 18 08:23 rails -rw-r--r--. 1 elk elk 224 8月 18 08:23 redis -rw-r--r--. 1 elk elk 188 8月 18 08:23 ruby -rw-r--r--. 1 elk elk 404 8月 18 08:23 squid [root@elk patterns]#
8.二、测试拆分格式（重点查看上边给的目录下的 httpd文档）
试拆分格式（重点查看官方文档grok）
这个是参考的官方网页文档
[root@elk config]# pwd /home/elk/Application/logstash/config [root@elk config]# cat grok.conf input { stdin {} } filter { grok { match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" } } } output { stdout { codec => rubydebug } } [root@elk config]#
8.三、实际收集日志测试
httpd的日志类型为默认类型
[root@elk config]# cat apache.conf input { file { path => "/var/log/httpd/access_log" start_position => "beginning" } } filter { grok { match => { "message" => "%{HTTPD_COMMONLOG}"} } } output { stdout { codec => rubydebug } } [root@elk config]# pwd /home/elk/Application/logstash/config [root@elk config]#
注释：HTTPD_COMMONLOG 是在httpd文件内拷贝出来的，后边已经定义了变量，直接引用便可！！！
8.四、最终版（apache）
[root@elk config]# cat apache.conf input { file { path => "/var/log/httpd/access_log" start_position => "beginning" type => "apache-access-log" } } filter { grok { match => { "message" => "%{HTTPD_COMMONLOG}"} } } output { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "apache-access-log-%{+YYYY.MM}" } } [root@elk config]# pwd /home/elk/Application/logstash/config [root@elk config]#
九、消息队列（低耦合）
冗余架构官方地址：https://www.elastic.co/guide/en/logstash/2.3/deploying-and-scaling.html#deploying-scaling
raid消息列队output内的插件：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html
9.一、测试logstash写入radis
[root@elk02 ~]# yum -y install redis [root@elk02 ~]# vim /etc/redis.conf bind 11.11.11.31 #监听地址 daemonize yes #后台运行 systemctl start redis systemctl enable redis [root@elk02 config]# netstat -luntp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 11.11.11.31:6379 0.0.0.0:* LISTEN 130702/redis-server tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 871/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1107/master tcp6 0 0 11.11.11.31:9200 :::* LISTEN 1309/java tcp6 0 0 11.11.11.31:9300 :::* LISTEN 1309/java tcp6 0 0 :::22 :::* LISTEN 871/sshd tcp6 0 0 ::1:25 :::* LISTEN 1107/master udp 0 0 127.0.0.1:323 0.0.0.0:* 533/chronyd udp6 0 0 ::1:323 :::* 533/chronyd [root@elk02 ~]# cd /home/elk/Application/logstash/ [root@elk02 config]# vim radis.conf input { stdin {} } output { redis { host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "demo" } } [root@elk02 config]# ../bin/logstash -f radis.conf Sending Logstash logs to /home/elk/Application/logstash/logs which is now configured via log4j2.properties [2018-11-14T19:05:53,506][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-11-14T19:05:57,477][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.0"} [2018-11-14T19:06:14,992][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-11-14T19:06:16,366][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6c86dc5a run>"} The stdin plugin is now waiting for input: [2018-11-14T19:06:16,772][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2018-11-14T19:06:18,719][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} heheh01 hehe02 [root@elk02 config]# redis-cli -h 11.11.11.31 11.11.11.31:6379> set name huangyanqi OK 11.11.11.31:6379> info # Keyspace db0:keys=1,expires=0,avg_ttl=0 db6:keys=1,expires=0,avg_ttl=0 11.11.11.31:6379> SELECT 6 #进入db6 OK 11.11.11.31:6379[6]> keys * #keys类型 1) "demo" 11.11.11.31:6379[6]> type demo @demo格式 list 11.11.11.31:6379[6]> llen demo #消息长度 (integer) 2 11.11.11.31:6379[6]> lindex demo -1 #查看最后一行 "{\"message\":\" heheh01\",\"@timestamp\":\"2018-11-14T11:06:46.034Z\",\"@version\":\"1\",\"host\":\"elk02\"}" 11.11.11.31:6379[6]>
9.二、 apache的日志写入redis内
应为elk（11.30）已经安装了http，就使用这台作实验了，radis安装在了elk2（11.31）上
[root@elk ~]# cd /home/elk/Application/logstash/config/ [root@elk config]# vim apache.conf input { file { path => "/var/log/httpd/access_log" start_position => "beginning" } } output { redis { host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "apache-access-log" } } #这时须要访问下11.30上的http服务，以便产生新的日志，radis才能读取到 #而后就能够访问radis了 [root@elk02 ~]# redis-cli -h 11.11.11.31 #登录redis 11.11.11.31:6379> select 6 #进入库6 OK 11.11.11.31:6379[6]> keys * #查看哪些key值，生产千万不能查全部key 1) "apache-access-log" 2) "demo" 11.11.11.31:6379[6]> type apache-access-log #key的类型 list 11.11.11.31:6379[6]> llen apache-access-log #key的长度，就多少条信息 (integer) 3 11.11.11.31:6379[6]> lindex apache-access-log -1 #这个key的最后一条内容 "{\"message\":\"11.11.11.1 - - [16/Nov/2018:04:45:07 +0800] \\\"GET / HTTP/1.1\\\" 200 688 \\\"-\\\" \\\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\\\"\",\"@timestamp\":\"2018-11-15T20:45:08.705Z\",\"@version\":\"1\",\"path\":\"/var/log/httpd/access_log\",\"host\":\"elk\"}" 11.11.11.31:6379[6]>
9.三、把redis内的日志写入logstash内
raid消息列队input内的插件 https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html
[root@elk02 ~]# cat redis_port.conf input { redis { host => "11.11.11.31" port => "6379" db =>"6" data_type => "list" key => "apache-access-log" } } filter { grok { match => { "message" => "%{HTTPD_COMMONLOG}"} } } output { stdout { codec => rubydebug } } [root@elk02 ~]# [root@elk02 ~]# /home/elk/Application/logstash/bin/logstash -f redis_port.conf
9.四、实测
elk（11.30）安装了httpd、logstash、els、kibans、head
elk02(11.31)安装了logstash、els、redis
过程：在11.30上运行httpd使用logstash收集而后传输到11.31上的redis内；而后使用11.31上的logstash读取后写入11.30的els内。
[root@elk config]# cat apache.conf input { file { path => "/var/log/httpd/access_log" start_position => "beginning" } } output { redis { host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "apache-access-log" } } [root@elk config]# pwd /home/elk/Application/logstash/config [root@elk config]# [root@elk config]# ../bin/logstash -f apache.conf [root@elk02 ~]# cat redis_port.conf input { redis { host => "11.11.11.31" port => "6379" db =>"6" data_type => "list" key => "apache-access-log" } } output { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "access01-log-%{+YYYY.MM.dd}" } } [root@elk02 ~]# [root@elk02 ~]# /home/elk/Application/logstash/bin/logstash -f redis_port.conf
十、生产建议后设置
需求分析： 访问日志：apache访问日志、nginx访问日志、tomcat file--》filter 错误日志：error log、java日志 直接收，java异常须要处理 系统日志：/var/log/* syslong rsyslog 运行日志： 程序写的 file json格式 网络日志： 防火墙、交换机、路由器日志 syslong 标准化： 日志放哪（/data/logs/） 格式是什么(JSON格式) 命名规则（access_log error_log runtime_log 三个目录） 日志怎么切割（按天或小时）（access error crontab进行切分） 全部原始文本---rsync到别处，后删除最近三天前的日志 工具化： 如何使用logstash进行收集方案 若是使用redis list 做为ELKstack的消息队列，那么请对全部list key的长度进行监控 llen key_name 根据实际状况，若是超过10万就告警
#11.11.11.30上设置后输入11.31上的redis上。
[root@elk config]# pwd /home/elk/Application/logstash/config [root@elk config]# cat apache1.conf input { file { path => "/var/log/httpd/access_log" start_position => "beginning" type => "apache-2-access-log" } file { path => "/home/elk/Log/elasticsearch/my-es.log" type => "es-2-log" start_position => "beginning" codec => multiline{ pattern => "^\[" negate => true what => "previous" } } } filter { grok { match => { "message" => "%{HTTPD_COMMONLOG}"} } } output { if [type] == "apache-2-access-log" { redis { host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "apache-2-access-log" } } if [type] == "es-2-log" { redis { host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "es-2-log" } } } [root@elk config]#
[root@elk config]# ../bin/logstash -f apache.conf
[root@elk02 ~]# cat redis_port.conf input { syslog { type => "system-2-syslog" port=> 514 } redis { type => "apache-2-access-log" host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "apache-2-access-log" } redis { type => "es-2-log" host => "11.11.11.31" port => "6379" db => "6" data_type => "list" key => "es-2-log" } } filter { if [type] == "apache-2-access-log" { grok { match => { "message" => "%{HTTPD_COMMONLOG}"} } } } output { if [type] == "apache-2-access-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "apache-2-access-log-%{+YYYY.MM}" } } if [type] == "es-2-log" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "es-2-log-%{+YYYY.MM.dd}" } } if [type] == "system-2-syslog" { elasticsearch { hosts => ["http://11.11.11.30:9200"] index => "system-2-syslog-%{+YYYY.MM}" } } } [root@elk02 ~]#
附录：
一、继3.1 启动elasticsearch以后安装的插件（方法）
https://www.cnblogs.com/Onlywjy/p/Elasticsearch.html
1.一、安装haed插件，启动./elasticsearch以后
- 下载head插件
wget -O /usr/local/src/master.zip https://github.com/mobz/elasticsearch-head/archive/master.zip
- 安装node
wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz
http://nodejs.org/dist/v4.4.7/ node下载地址
tar -zxvf node-v4.4.7-linux-x64.tar.gz
- 配置下环境变量,编辑/etc/profile添加
#set node envirnoment
export NODE_HOME=/usr/local/src/node-v4.4.7-linux-x64 export PATH=$PATH:$NODE_HOME/bin export NODE_PATH=$NODE_HOME/lib/node_modules
执行 source /etc/profile
- 安装grunt
grunt是基于Node.js的项目构建工具，能够进行打包压缩、测试、执行等等的工做，head插件就是经过grunt启动
unzip master.zip [root@elk src]# cd elasticsearch-head-master/ npm install -g grunt-cli //执行后会生成node_modules文件夹 [root@elk elasticsearch-head-master]# npm install -g grunt-cli npm WARN engine atob@2.1.2: wanted: {"node":">= 4.5.0"} (current: {"node":"4.4.7","npm":"2.15.8"}) /usr/local/src/node-v4.4.7-linux-x64/bin/grunt -> /usr/local/src/node-v4.4.7-linux-x64/lib/node_modules/grunt-cli/bin/grunt grunt-cli@1.3.2 /usr/local/src/node-v4.4.7-linux-x64/lib/node_modules/grunt-cli ├── grunt-known-options@1.1.1 ├── interpret@1.1.0 ├── v8flags@3.1.1 (homedir-polyfill@1.0.1) ├── nopt@4.0.1 (abbrev@1.1.1, osenv@0.1.5) └── liftoff@2.5.0 (flagged-respawn@1.0.0, extend@3.0.2, rechoir@0.6.2, is-plain-object@2.0.4, object.map@1.0.1, resolve@1.8.1, fined@1.1.0, findup-sync@2.0.0) [root@elk elasticsearch-head-master]# grunt -version grunt-cli v1.3.2 grunt v1.0.1 修改head插件源码 #修改服务器监听地址:Gruntfile.js
#修改链接地址
[root@elk elasticsearch-head-master]# vim _site/app.js
#设置开机启动
[root@elk ~]# cat es_head_run.sh
PATH=$PATH:$HOME/bin:/usr/local/src/node-v4.4.7-linux-x64/bin/grunt
export PATH
cd /usr/local/src/elasticsearch-head-master/
nohup npm run start >/usr/local/src/elasticsearch-head-master/nohup.out 2>&1 &
[root@elk ~]# vim /etc/rc.d/rc.local
/usr/bin/sh /root/es_head_run.sh 2>&1
一、离线安装插件方法 https://www.elastic.co/guide/en/marvel/current/installing-marvel.html#offline-installation
二、插件收索方法： 还有kopf （记获得里边查看版本兼容性）
[root@elk elasticsearch]# ./bin/elasticsearch-plugin install lukas-vlcek/bigdesk
二、健康集群是否正常（状态）
[elk@elk ~]$ curl -XGET 'http://11.11.11.30:9200/_cluster/health?pretty=true' { "cluster_name" : "my-es", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } [elk@elk ~]$
提供更好的健康检查方法： https://www.elastic.co/guide/en/elasticsearch/guide/current/_cat_api.html
[elk@elk ~]$ curl -XGET 'http://11.11.11.30:9200/_cat/health?pretty=true' 1541662928 15:42:08 my-es green 2 2 0 0 0 0 0 0 - 100.0% [elk@elk ~]$
三、配置logstash后续测试
参考文档：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
写入插件：https://www.elastic.co/guide/en/logstash/current/input-plugins.html
输出插件：https://www.elastic.co/guide/en/logstash/current/output-plugins.html
这里选输出elasticsearch插件 ： https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
[root@elk ~]# /home/elk/Application/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["11.11.11.30:9200"] index => "logstash-%{+YYYY.MM.dd}" } }' ...中间会出现一堆东西，不用管... [2018-11-08T16:28:05,537][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} hehe hehehehe #数据已经写入els上了，能够在head页面内查看写入的数据
四、线上集群的被建议作法
1、每有一个ES上都运行一个kibanaweb 2、每一个kibana都链接本身本地的ES 3、前端Nginx负载均衡+验证
五、Kafka消息队列（待续）
https://www.unixhot.com/article/61
https://www.infoq.cn/
六、另外一个能够代替head的插件
软件地址： https://github.com/lmenezes/cerebro/tags
下载到系统内、解压后 [root@elk cerebro-0.8.1]# ./bin/cerebro -Dhttp.port=1234 -Dhttp.address=11.11.11.30
相关文章
- 1. ELK安装过程总结
- 2. ELK安装流程和ELK安装遇到的所有坑
- 3. ELK的安装教程 20190426
- 4. ELK安装使用教程
- 5. ELK Stack (1) —— ELK + Redis安装
- 6. 【ELK】ELK安装与配置
- 7. 【ELK】ELK安装部署
- 8. docker安装ELK
- 9. Docker安装ELK
- 10. elk安装
- 更多相关文章...
- • Docker 安装 Nginx - Docker教程
- • Docker 安装 Node.js - Docker教程
- • Composer 安装与使用
- • IntelliJ IDEA安装代码格式化插件
相关标签/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. Appium入门
- 2. Spring WebFlux 源码分析(2)-Netty 服务器启动服务流程 --TBD
- 3. wxpython入门第六步（高级组件）
- 4. CentOS7.5安装SVN和可视化管理工具iF.SVNAdmin
- 5. jedis 3.0.1中JedisPoolConfig对象缺少setMaxIdle、setMaxWaitMillis等方法，问题记录
- 6. 一步一图一代码，一定要让你真正彻底明白红黑树
- 7. 2018-04-12—（重点）源码角度分析Handler运行原理
- 8. Spring AOP源码详细解析
- 9. Spring Cloud（1）
- 10. python简单爬去油价信息发送到公众号
欢迎关注本站公众号,获取更多信息
相关文章
- 1. ELK安装过程总结
- 2. ELK安装流程和ELK安装遇到的所有坑
- 3. ELK的安装教程 20190426
- 4. ELK安装使用教程
- 5. ELK Stack (1) —— ELK + Redis安装
- 6. 【ELK】ELK安装与配置
- 7. 【ELK】ELK安装部署
- 8. docker安装ELK
- 9. Docker安装ELK
- 10. elk安装