配置nagios来监控HA集群（二）
终于开始监控集群了，先说下实验环境：之前搭建的ha集群，192.168.10.101和192.168.10.102，运行的是http服务，nagios安装在192.168.10.1，利用nrpe监控101和102。
好了，开始。老习惯，先找资料：
First off, we need to define what we mean by a "cluster". The simplest way to understand this is with an example. Let's say that your organization has five hosts which provide redundant DNS services to your organization. If one of them fails, its not a major catastrophe because the remaining servers will continue to provide name resolution services. If you're concerned with monitoring the availability of DNS service to your organization, you will want to monitor five DNS servers. This is what I consider to be a service cluster. The service cluster consists of five separate DNS services that you are monitoring. Although you do want to monitor each individual service, your main concern is with the overall status of the DNS service cluster, rather than the availability of any one particular service.
这段讲的重点其实就是我们之所以要搭建集群就是为了保证服务在其中节点宕机的情况下还能正常运行，所以用nagios监控集群的关键是将监控集群本身作为nagios的一个服务去看待，而这个服务的目标在于对集群这个整体运行状态的监控，而不是具体针对其中哪一台机器出了问题。
here are several ways you could potentially monitor service or host clusters. I'll describe the method that I believe to be the easiest. Monitoring service or host clusters involves two things:
*Monitoring individual cluster elements
*Monitoring the cluster as a collective entity
Monitoring individual host or service cluster elements is easier than you think. In fact, you're probably already doing it. For service clusters, just make sure that you are monitoring each service element of the cluster. If you've got a cluster of five DNS servers, make sure you have five separate service definitions (probably using the check_dns plugin). For host clusters, make sure you have configured appropriate host definitions for each member of the cluster (you'll also have to define at least one service to be monitored for each of the hosts). Important: You're going to want to disable notifications for the individual cluster elements (host or service definitions). Even though no notifications will be sent about the individual elements, you'll still get a visual display of the individual host or service status in the status CGI. This will be useful for pinpointing the source of problems within the cluster in the future.
Monitoring the overall cluster can be done by using the previously cached results of cluster elements. Although you could re-check all elements of the cluster to determine the cluster's status, why waste bandwidth and resources when you already have the results cached? Where are the results cached? Cached results for cluster elements can be found in the status file (assuming you are monitoring each element). The check_cluster plugin is designed specifically for checking cached host and service states in the status file. Important: Although you didn't enable notifications for individual elements of the cluster, you will want them enabled for the overall cluster status check.
而这一段具体说明了监控集群的两个方面：一个是节点的监控（如我们上一篇所实验的那样，具体的监控）；第二个是集群的监控。节点的监控举例告诉我们如果有5个节点的集群跑DNS服务，那么在localhost.cfg文件里面需要定义5个独立的服务，利用check_dns插件来完成。并且强调了：在这里的监控关闭通知，因为我们能在CGI界面上看到相应的监控状态。那么nagios需要通知的是什么呢？想象下：一个维护人员不可能一天二十四小时都趴在机房里等着出问题吧？刚刚说集群如果宕机了一个或者几个节点，但是服务还是在正常的运行，没必要现在就赶着去把那几台宕掉的机器弄好吧？或者那几台被fence device重启了你也大老远的白跑一趟么？所以只有当集群或者服务真的出问题了或者运行得不稳定了我们才要引起重视，没必要为单独的一台服务器的小毛病把自己的生活都弄得乱七八糟吧，维护人员是要掌握系统，而不是被系统掌握！
这里还提到了用来监控集群的模块check_cluster，我们再来想下，本来我们在进行节点监控的时候已经获得了很多监控数据了，那现在监控集群又要再多去监控一遍么？没必要吧，所以这里提到check_cluster模块是利用nagios本身的缓存cached的数据，这就不用再重新去获得数据，而直接对数据进行分析，大大节省了资源。那么这些cached的数据存放在哪里呢？/usr/localhost/nagios/var/目录下一个叫status.dat的文件里：
我们可以看到大量的数据，而check_cluster就是从这些数据里得到相应的然后进行分析的。废话不多说了，开始定义check_cluster（command.cfg里默认是没有定义的，所以要手动写进去）：
参数的意思我们可以用--help来查看下，这里设置了4个参数：第一个我也不知道什么意思，估计是个名字标识吧随便写了，第二个warning的级别定义，第三个critical的级别，第四个等下我们来解释，下面再定义一个命令用来监控服务：
然后定义localhost.cfg:
首先是监控服务的定义warning的级别是1，critical是2，也就是1台宕机了warning，2台就critical了，后面的$SERVICESTATEID $HOSTSTATEID是是服务检查中产生的宏，存放了service检测的返回值，后面冒号接定义的host然后冒号接定义的服务名，这里注意要用$结尾，然后逗号写下一个。$HOSTSTATEID也是同样的道理，不过就不用写服务了。
重启一边nagios然后看状态：
其中服务中提示有一条critical，我查看下是：
当然会critical，因为ha集群只有一台服务在运行状态。现在我把node1的httpd给stop掉再看：
Node2开始接管了，我没有调整nagios收集信息的参数，所以延时还是蛮大的，哎。。手动执行了，等不了了：
继续，我现在把node1给shutdown，按照我们的设置应该会cluster_host提示warning，cluster_service还是OK状态吧？看看吧：
红红的一片，node1已经down了，再来看cluster_host和cluster_service：
奇怪了，应该cluster_host会warning啊，怎么后面写着1up，1down状态还是ok的啊？查了资料，这里我弄错了，原来warning是指的大于设置的值，我设置1那么就要大于1台down才会报警。好了，关掉node2然后再看状态：
同时报警，warning而不是critical，可以看出critical也是大于所设定的值。好了，终于完成了。不过这也只是冰山一角，nagios还有很多的配置参数，很多插件，还有特别是它的报警机制可以值得去研究的，确实是公司企业，运维必备的利器！！
参考：http://blog.chinaunix.net/u2/89218/showart_2019588.html
http://vincentwen.blog.51cto.com/824577/385231