使用Monit替代Supervisor自动化管理和监控服务小结

前言

对于进程的监控最多见的需求就是进程挂了如何被自动拉起来，如今能够由Kubernetes等先进的容器化技术来自动化管理，那原来再物理服务器或者虚拟机中的进程有什么好的办法呢？答案就是Monit/Supervisor等第三方应用来解决，由于线上环境分别使用Monit来监控Core Logical Service，Supervisor用在Codis Dashboard/FE/Proxy上，使用下来的感觉和网上的对比分析报告相似，具体内容会在文章内引用，推荐你们使用Monit替代Supervisor自动化管理和监控服务。

使用Monit替代Supervisor自动化管理和监控服务小结

更新历史

2020年01月15日 - 初稿

阅读原文 - https://wsgzao.github.io/post...

扩展阅读

Monit
Supervisor

---

Monit介绍

NAME

Monit - utility for monitoring services on a Unix system

SYNOPSIS

monit [options] <arguments>

DESCRIPTION

Monit is a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and filesystems for changes, such as timestamps changes, checksum changes or size changes.

Monit is controlled via an easy to configure control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions via customisable alert messages. Monit can perform various TCP/IP network checks, protocol checks and can utilise SSL for such checks. Monit provides a HTTP(S) interface and you may use a browser to access the Monit program.

WHAT TO MONITOR?

You can use Monit to monitor daemon processes or similar programs running on localhost. Monit is particularly useful for monitoring daemon processes, such as those started at system boot time. For instance sendmail, sshd, apache and mysql. In contrast to many other monitoring systems, Monit can act if an error situation should occur, e.g.; if sendmail is not running, monit can start sendmail again automatically or if apache is using too many resources (e.g. if a DoS attack is in progress) Monit can stop or restart apache and send you an alert message. Monit can also monitor process characteristics, such as how much memory or cpu cycles a process is using.

You can also use Monit to monitor files, directories and filesystems on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons - you can monitor the md5 or sha1 checksum of files that should not change and get an alert or perform an action if they should change.

Monit can monitor network connections to various servers, either on localhost or on remote hosts. TCP, UDP and Unix Domain Sockets are supported. Network test can be performed on a protocol level; Monit has built-in tests for the main Internet protocols, such as HTTP, SMTP etc. Even if a protocol is not supported you can still test the server because you can configure Monit to send any data and test the response from the server.

Monit can be used to test programs or scripts at certain times, much like cron, but in addition, you can test the exit value of a program and perform an action or send an alert if the exit value indicates an error. This means that you can use Monit to perform any type of check you can write a script for.

Finally, Monit can be used to monitor general system resources on localhost such as overall CPU usage, Memory and System Load.

https://mmonit.com/monit/docu...

Supervisor介绍

Supervisor: A Process Control System

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.

http://supervisord.org/

Monit VS Supervisor

Monit是什么

• Monit 是一个管理和监控 Unix 系统的小型开源组件.
• Monit 能够在出现错误的状况下, 自动维护, 修复和作一些有意义的行为

为何选择Monit

除了 Monit 还有一些其余的第三方监控方案(eg. Supervisor), 咱们考虑选择额 Monit 做为监控的缘由有

• 超轻量, 稳定, 高可用
• 依赖少, 安装配置方便, 尽可能减小运维及学习成本(即便没有任何 Monit 基础的人, 都能轻易的读懂大部分监控文件)
• 非侵入式, 被监控的程序能够不用知道监控程序的存在(若是使用 Supervisor 监控, 则服务必须从 Supervisor 启动)
• 基本功能完备(9 种类型监控, 邮件报警, 支持用户自定义 shell 扩展)

Supervisor优缺点

优势

1. 轻量、特性丰富、内存友好(好客套的优势。。。)
2. 对被监控进程的状态获取迅速且精确——经过子进程管理，没办法不精确

缺点

1. 被监控进程必须运行在前台(能够理解为有本身的控制终端)——这也是最为致命的一点
2. 没法管理依赖，也就是说没法控制服务启动前后顺序
3. 没法管理被监控进程建立的子进程——重启服务时被监控进程的子进程没法正常退出，隐患大
4. 没法控制进程失败重试的间隔时间——有些进程须要清理资源，不过这点还好

Monit优缺点

优势

1. 安装配置简单，一样轻量(彷佛也是很客套了)
2. 能够监控前台进程和非前台进程——完美的弥补了supervisor的致命缺点
3. 除了监控进程还能监控文件、文件系统，甚至系统资源，CPU利用率，内存使用也是能够的
4. 被监控的进程能够设置依赖，控制启动顺序

缺点

1. 没法监控没有pid文件的进程，如shell脚本
2. 对进程监控的状态感知有延时，即精度不够——采用轮训决定了它没法像supervisor同样实时感知被监控进程状态

这样看起来仍是monit更为普适一点。

不过这催生了一个大胆的想法，使用supervisor管理容器内多进程，monit做为一个被监控进程挂在supervisor之下。这样对于没法前台运行的程序，就能够经过monit监控，而对服务中断感知强烈的则直接挂在supervisor之下。看起来彷佛是个好办法，有机会试试，哈哈哈。

从实际容器中运行的表现看，monit常常出现各类未知异常，而supervisor表现得十分稳定。

Monit基本用法

Monit经常使用命令

# monit -h
Usage: monit [options]+ [command]
Options are as follows:
 -c file       Use this control file
 -d n          Run as a daemon once per n seconds
 -g name       Set group name for monit commands
 -l logfile    Print log information to this file
 -p pidfile    Use this lock file in daemon mode
 -s statefile  Set the file monit should write state information to
 -I            Do not run in background (needed when run from init)
 --id          Print Monit's unique ID
 --resetid     Reset Monit's unique ID. Use with caution
 -B            Batch command line mode (do not output tables or colors)
 -t            Run syntax check for the control file
 -v            Verbose mode, work noisy (diagnostic output)
 -vv           Very verbose mode, same as -v plus log stacktrace on error
 -H [filename] Print SHA1 and MD5 hashes of the file or of stdin if the
               filename is omited; monit will exit afterwards
 -V            Print version number and patchlevel
 -h            Print this text
Optional commands are as follows:
 start all             - Start all services
 start <name>          - Only start the named service
 stop all              - Stop all services
 stop <name>           - Stop the named service
 restart all           - Stop and start all services
 restart <name>        - Only restart the named service
 monitor all           - Enable monitoring of all services
 monitor <name>        - Only enable monitoring of the named service
 unmonitor all         - Disable monitoring of all services
 unmonitor <name>      - Only disable monitoring of the named service
 reload                - Reinitialize monit
 status [name]         - Print full status information for service(s)
 summary [name]        - Print short status information for service(s)
 report [up|down|..]   - Report state of services. See manual for options
 quit                  - Kill the monit daemon process
 validate              - Check all services and start if not running
 procmatch <pattern>   - Test process matching pattern

想要让 Monit 可靠的为咱们工做, 学习成本很是低, 只须要学习一些 Monit 命令行和配置文件写法

# options - 选项
- monit
- monit -t
- monit -c /var/monit/monitrc  # 指定配置文件
- monit -g <groupname> start/stop # Monit 能够对各个监控分组, 若是须要对某个分组统一操做, 能够用这个命令

# arguments - 参数
- monit reload
- monit quit
- monit start/stop/restart/monitor/unmonitor <name>/all  # <name>: 每一个监控都有一个独一无二的名字, 具体后面会提到; all: 全部监控服务

Monit服务监控配置文件格式

详细配置, 共计 9 种, 全部配置中, 都符合如下规则

• 若是指定的 path 不存在, 并且配置块里包含 start 方法, 会调用这个 start 方法
• 若是 path 指定的文件类型不对, Monit 不能监控这个项目

1. Process

CHECK PROCESS <unique name> <PIDFILE <path> | MATCHING <regex>>

<path> pid-file 的绝对路径. 不存在 pid-file 文件或者 pid-file 文件没有对应的正在运行的程序, Monit 会执行 start 方法

<regex> 进程名称的正则表达来监控进程, 能够经过命令行测试正则是否写对了: monit procmatch "regex-pattern"

1. File

CHECK FILE <unique name> PATH <path>

<path> file 的绝对路径.

1. Fifo

CHECK FIFO <unique name> PATH <path>
<path> fifo 的绝对路径.

1. Filesystem

CHECK FILESYSTEM <unique name> PATH <path>
<path> 设备/磁盘, 挂载点的路径 或 NFS/CIFS/FUSE 连接字符串. 若是文件系统不可用, Monit 会执行 start 方法

1. Directory

CHECK DIRECTORY <unique name> PATH <path>

<path> 目录问价的绝对路径

1. Remote host

CHECK HOST <unique name> ADDRESS <host>

<host> 能够是域名或者 IP 地址. eg: "tildeslash.com" or "64.87.72.95".

1. System

CHECK SYSTEM <unique name>

<unique name> 一般来讲是本机名称(能够用 $HOST), 也能够是其余名称. 用于邮件报警或者 M/Monit 的初始化名称
这类配置能够监控系统资源(CPU, memory, load average...)

1. Program

CHECK PROGRAM <unique name> PATH <executable file> [TIMEOUT <number> SECONDS]

<path> 可执行程序或脚本的绝对路径. 容许检查程序退出状态.若是程序没能在 <number> 秒内执行完成, Monit 会终结这个程序, 默认是 300s
程序的输出会被记录, 用于用户界面或者报警, 默认 512 bytes(能够经过 set limits 修改)

1. Network

CHECK NETWORK <unique name> <ADDRESS <ipaddress> | INTERFACE <name>>

# <ipaddress> 是被监控的 IPv4/IPv6 网卡地址. 用 eth0 也是能够的

更多配置信息能够参考Monit官方文档和实例

https://mmonit.com/documentat...

https://mmonit.com/wiki/Monit...

Monit配置实践

1. 建立templates模板，利用python生成monit配置文件
2. 使用ansible推送到目标服务器中

# 建立通用配置，配置日志，邮件告警
vim basic.j2

# log to monit.log
set logfile /var/log/monit.log

set daemon {{ monit_poll_interval }}

set eventqueue basedir /var/lib/monit/events slots 5000

set mailserver smtp.xxx.com port 465

set alert xxx@xxx.com { nonexist, timeout, resource }

set mail-format {
  from: xxx@xxx.com
  subject: monit alert -- $SERVICE $EVENT at $DATE
  message: $EVENT Service $SERVICE
                Date:        $DATE
                Action:      $ACTION
                Host:        $HOST
                Description: $DESCRIPTION

           Your faithful employee,
           Monit
}

# 建立标准应用监控
vim daemon_set.j2

check process xxx with pidfile /run/xxx/daemon.pid
    start program = "/usr/bin/python2  /bin/xxx restart"
    stop program = "/usr/bin/python2 /bin/xxx stop"

    if 10 restarts within 10 cycles then unmonitor

check process xxxx with matching xxxx
    start program = "/etc/init.d/xxxx start"
    stop program = "/etc/init.d/xxxx stop"

    if 10 restarts within 10 cycles then unmonitor

# 建立非标准应用监控
vim logic_service.j2

check process {{ service_name }} with pidfile {{ root_dir }}/{{ service_name }}/deploy/{{ monit_name }}.pid
    start program = "/bin/bash -c 'cd {{ root_dir }}/{{ service_name }}/deploy && ./start.sh &>start.log '"
    stop program = "/bin/bash -c 'cd {{ root_dir }}/{{ service_name }}/deploy && ./stop.sh &>stop.log '"

    if 5 restarts within 15 cycles then unmonitor
    {% if memory_usage is defined %}
    if total memory usage > {{ memory_usage }} for 10 cycles then restart
    {% endif %}

参考文章

Monit Document

Monit 笔记

Docker容器内多进程管理——supervisor VS monit