基于Kibana的可视化监控报警插件sentinl入门

sentinl是什么

• Kibi/Kibana Alert & Reporting App
• Watching your data, 24/7/365

sentinl是一个免费的kibana预警与报告插件，与付费软件X-Pack功能相似。

Some Examples for illustration:

• HIT COUNT PER HOUR
• QUESTION: How many hits does index X receive hourly?
• WATCHER: query index and return count of hits in last hour
• ACTION: Notify with number of Hits per hour
• METRIC THRESHOLDS
• QUESTION: Is any of my monitored metrics surpassing a certain value?
• WATCHER: query index and type for specific values, aggregated by an arbitrary field.
• ACTION: Notify with aggs bucket details every time a threshold is surpassed or spike anomaly detected.
• BLACKLISTS HITS
• QUESTION: Is any of my users trying to reach blacklisted destinations?
• WATCHER: query firewall logs comparing destination IPs to a blacklist.
• ACTION: Notify admin via email if any IP >= 10 matches returned
• FAILED LOGINS
• QUESTION: Are there recurring failure attempts authenticating users on my network?
• WATCHER: query active directory logs for login failures in last hour and compare to user index. .
• ACTION: Notify admin via webhook if >= 10 matches returned
• LEAK DETECTION (chain)
• QUESTION: Are there any public leaks about my data I was not aware of?
• WATCHER: query for user emails included in published leaks ingested from third parties.
• ACTION: Save hits in secondary result Index. Notify via email if leak was not known in a secondary Watcher

安装

./kibana-plugin install file:./sentinl-v6.0.1.zip

安装完成后，要重启kibana

fuser -n tcp 5601  
ps -ef | grep node
kill -9 pid

./kibana &

使用步骤

使用包括5个步骤

• Step 1: New Watcher

give our Watcher a name and choose an execution frequency

• Step 2: Input Query

es的搜索与聚合

• Step 3: Condition

validate if the results received back are worth processing

语法与x-pack script condition语法相似

至关于过滤条件

"condition": {  
      "script": {  
        "script": "payload.hits.total>=1"  //当报警条件为***出现的次数大于1  
      }  
    }  


"condition": {
      "script": {
        "script": "payload.hits.hits[0]._source.responsetime > 0.01" // 检索条件 响应时间大于 0.01秒
      }
    }

• Step 4: Transform

Our data might need adjustments or post processing. Process our payload using a javascript expression/script

过后处理

• Step 5: Actions

Let's form a notification using the mustache templating language。

能够采用多种方式发送通知。

transform

How to Adapt or Post-Process data

Post Process过后的处理。

The transform script is the wild member of the family and can be used to inject simple or complex logic into the pipeline before delivery to actions using pure javascript.

From converting format types, through generating brand new payload keys and interpolating data, transform is the way up. The script expects a boolean condition to trigger actions. A false condition can be forced to stop the execution. BONUS: Transforms can be saved and used across Watchers! "transform": { "script": { "script": "payload.newvar = payload.aggs.some.values['95.0']" } }

action举例之邮件发送

• Troubleshooting

kibana.yml

logging.verbose: true
sentinl:
  settings:
    email:
      active: true
      host: smtp.exmail.qq.com
      ssl: false
    report:
      active: true
      tmp_path: /tmp/
      

上面是官网的，下面是实践已OK      
sentinl:
  settings:
    email:
      active: true
      user: tanyk@huawangtech.com
      password: Dd@2016
      host: smtp.exmail.qq.com
      ssl: true
      timeout: 10000
    report:
      active: true
      tmp_path: /tmp/

先测试

mailx -S smtp=<smtp-server-address> -r <from-address> -s <subject> -v <to-address> < body.txt

yum -y install sendmail
yum install -y sendmail-cf

/etc/init.d/sendmail start
chkconfig sendmail on
yum install -y mailx

vim /etc/mail.rc（optional）

set from=tanyk@mail.com
set smtp=smtp.exmail.qq.com
set smtp-auth-user=tanyk@mail.com
set smtp-auth-password=******
set smtp-auth=login
set nss-config-dir="/etc/pki/nssdb/"

test

echo "This is the message body and contains the message" | mailx -v -r "tanyk@mail.com" -s "This is the subject" -S smtp="smtp.exmail.qq.com" -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user="tanyk@mail.com" -S smtp-auth-password="******" -S ssl-verify=ignore -S nss-config-dir="/etc/pki/nssdb/" tanyk@163.com

参考文献

• 基于Kibana的可视化监控报警插件 KAAE 的配置
• http://sentinl.readthedocs.io/en/latest/
• Kibana 可视化监控报警插件 KAAE 的介绍与使用