搭建 ELK 问题排查

最近在电脑上开了三个虚拟机鼓捣了一下 ELK，配置成功以后，关闭虚拟机从新打开老是出现一些奇怪的问题，使得 kibana 处于不健康的状态，真是让人操碎了心。

1、前言

在搭建过程当中，本人是依据如下两篇文章进行的，步骤明确，效果清晰。

一、  搭建ELK日志分析平台（上）—— ELK介绍及搭建 Elasticsearch 分布式集群

二、  搭建ELK日志分析平台（下）—— 搭建kibana和logstash服务器

如下记录本人在实现过程当中遇到的问题以及最终解决的思路。

 

2、elasticsearch 集群状态不健康

一、问题描述

elasticsearch （如下简称 es）集群状态处于 yellow 或者 red 状态，2 个数据节点未成功接入主节点，number_of_nodes 数量仍为 1，kibana 界面报错 503。

[root@server ~]# curl '192.168.100.15:9200/_cluster/health?pretty'
{
  "cluster_name" : "server-node",
  "status" : "red",             # 为 green 则表明健康没问题，若是是 yellow 或者 red 则是集群有问题
  "timed_out" : false,          # 是否有超时
  "number_of_nodes" : 1,        # 集群中的节点数量
  "number_of_data_nodes" : 0,   # 集群中data节点的数量
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 12,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 0.0          # 可用性百分比，此处为 0 不可用
}

 

二、排查思路

1）首先确保 es 主节点最早启动，随后启动数据节点；

2）容许 selinux（非必要），关闭 iptables；

3）确保数据节点的配置文件正确。

 

三、排查后状态

[root@server ~]# curl '192.168.100.15:9200/_cluster/health?pretty'        
{
  "cluster_name" : "server-node",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

 

3、索引状态不健康

一、问题描述

es 启动正常，kibana 报错 503，management 页面无显示，点击无反应，查看索引状态为 red。

[root@server ~]# curl '192.168.100.15:9200/_cat/indices?v'
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   system-syslog-2018.09 JPDsnK_qSym-sjOiZS9zAw   5   1        548            0    719.6kb        345.3kb

 

二、排查思路

1）确认 logstash 是否正常启动，端口（9600以及各日志索引配置端口）是否存在；

2）删除不正常索引，从新启动 logstash；

[root@server ~]# curl -XDELETE http://localhost:9200/system-syslog-2018.09
{"acknowledged":true}

3）确认 kibana 状态

[root@server ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-09-29 09:01:44 CST; 31min ago
 Main PID: 646 (node)
   CGroup: /system.slice/kibana.service
           └─646 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml...

Sep 29 09:01:44 server systemd[1]: Started Kibana.
Sep 29 09:01:44 server systemd[1]: Starting Kibana...

4）从新查看索引状态

 

三、排查后状态

[root@server ~]# curl '192.168.100.15:9200/_cat/indices?v'                
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   system-syslog-2018.09 TR_gdOb8RDSRtHj_g4a4_g   5   1          3            0     62.2kb         31.1kb

 

4、es 的 node 日志报错

一、问题描述

kibana 界面报错，菜单点击无反应，日志信息部分以下。

[2018-09-28T21:23:20,487][DEBUG][o.e.a.s.TransportSearchAction] [server] All shards failed for phase: [query]
[2018-09-28T21:23:20,488][WARN ][r.suppressed             ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) ~[elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.4.1.jar:6.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.1.jar:6.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
[2018-09-28T21:23:20,487][WARN ][r.suppressed             ] path: /.kibana/doc/config%3A6.4.1, params: {index=.kibana, id=config:6.4.1, type=doc}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.4.1]: routing [null]]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:207) ~[elasticsearch-6.4.1.jar:6.4.1]

 

二、排查思路

1）删除索引

[root@server ~]# curl -XDELETE http://localhost:9200/.kibana
{"acknowledged":true}

2）在 kibana 界面开启或关闭

 

三、排查后状态

[root@server ~]# curl '192.168.100.15:9200/_cat/indices?v'        
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   system-syslog-2018.09 TR_gdOb8RDSRtHj_g4a4_g   5   1         20            0    240.6kb        120.3kb
green  open   .kibana               GGWwf7gdTwCKMn3BqRaGcQ   1   1          2            0       22kb           11kb

 

 

参考资料

1. 聊一聊Elasticsearch的健康状态

2. Elasticsearch系列篇之删除索引

3. Elasticsearchallshardsfailed:[unsupported_operation_exception]null