利用Keepalived实现lvs-dr的简单示例:
一共使用5台服务
a.时间服务器、调度器(Director1):
CentOS 7.2A
b.调度器(Director2)
CentOS 7.2B
c.后端服务器(Real Server1):
CentOS 7.2C
d.后端服务器(Real Server1):
CentOS 7.2D
f.进行测试的客户端:
CentOS 7.5Ahtml
1.修改对应的主机名;为了方便观察 Director(CentOS 7.2A): ~]# hostnamectl set-hostname drct1 Director(CentOS 7.2B): ~]# hostnamectl set-hostname drct2 Real Server1(CentOS 7.2C): ~]# hostnamectl set-hostname rs1 Real Server2(CentOS 7.2D): ~]# hostnamectl set-hostname rs2 2.对比无keepalived的lvs-dr模型,无需在调度器上的DIP配置对应的VIP,由于后续会在keepalived的主配置文件中进行VIP的配置; 3.在rs一、rs2上利用脚本进行配置 #!/bin/bash # VIP=172.16.72.254 MASK=255.255.255.255 case $1 in setup) #调整ARP相关的内核参数: echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce #在lo的标签接口上配置VIP ifconfig lo:0 $VIP netmask $MASK broadcast $VIP up #为了可以使响应报文从lo:0标签接口向外封装发送数据,须要指定一条特殊的静态路由: route add -host $VIP dev lo:0 ;; delete) ifconfig lo:0 down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage: $(basename $0) { setup | delete }" ;; esac 4.关闭防火墙、SELinux,确保调度器装上ipvsadm、keepalived、mailx;http服务全部服务器都要装上; 5.在调度器上的配置: drct1: ~]# cd /etc/keepalived keepalived]# vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct1 vrrp_mcast_group4 224.0.100.18 } vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 36 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0 #只在lvs-dr模型中才这样写 } } virtual_server 172.16.72.254 80 { delay_loop 1 lb_algo wrr lb_kind DR protocol TCP real_server 172.16.72.3 80 { weight 1 HTTP_GET { url { path /index.html } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.16.72.4 80 { weight 3 HTTP_GET { url { path /index.html } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } drct2: ~]# cd /etc/keepalived keepalived]# vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct2 vrrp_mcast_group4 224.0.100.18 } vrrp_instance VI_1 { state BACKUP interface eno16777736 virtual_router_id 36 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0 #只在lvs-dr模型中才这样写 } } virtual_server 172.16.72.254 80 { delay_loop 3 lb_algo wrr lb_kind DR protocol TCP real_server 172.16.72.3 80 { weight 1 HTTP_GET { url { path /index.html } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.16.72.4 80 { weight 3 HTTP_GET { url { path /index.html } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } 6.rs一、rs2上都开启http服务,并配置好相关的文件,令访问rs一、rs2时显示的内容不相同,这样可以方便观察keepalived的VRRP;实际生产环境中是使后端服务器显示的内容都相同; 7.drct一、drct2上都开启keepalived服务,利用ipvsadm查看集群配置状况: keepalived]# systemctl start keepalived.service keepalived]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.72.254:80 wrr -> 172.16.72.3:80 Route 1 0 0 -> 172.16.72.4:80 Route 3 0 0 You have new mail in /var/spool/mail/root 8.客户端测试 当两台调度服务器都正常工做、后端提供的http服务也正常时; ~]# for i in {1..10} ; do curl http://172.16.72.254 ; done this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ this is CentOS 7.2C for /var/www/html/ 关闭其中一台调度服务器的keepalived、后端提供的http服务也正常时: ~]# for i in {1..10} ; do curl http://172.16.72.254 ; done This is CentOS 7.2D for /var/www/html this is CentOS 7.2C for /var/www/html/ This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html this is CentOS 7.2C for /var/www/html/ This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html this is CentOS 7.2C for /var/www/html/ 当两台调度服务器都正常工做、关闭rs1的http服务: 调度器上: keepalived]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.72.254:80 wrr -> 172.16.72.4:80 Route 3 0 0 客户端访问: ~]# for i in {1..10} ; do curl http://172.16.72.254 ; done This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html This is CentOS 7.2D for /var/www/html //客户端测试结果显示: 1.调度器可以实现高可用 2.后端RS也可以实现负载均衡
其余较为有用的配置命令:
一.在LVS CONFIGURATION配置段的virtual server的上下文添加:
sorry_server <IPADDR> <PORT>web
示例: sorry_server 127.0.0.1 80 做用:当后端RS都没法使用时,前段的调度器给客户端返回一个友好的错误页面; 1) 把RS的http服务都关闭,同时开启调度器的http服务,配置sorry_server显示的web站点;重启keepalived服务;若此前keepalived主配置文件已添加sorry_server的配置,无需重启服务; 2) 此时两台调度器都应该处于正常的keepalived状态, 在客户端测试 ~]# for i in {1..10} ; do curl http://172.16.72.254 ; done sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 sorry page for 1 在调度器(drct1)上: keepalived]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.72.254:80 wrr -> 127.0.0.1:80 Route 1 0 10 3) 关闭调度器1(drct1)的keepalived, 在客户端测试: ~]# for i in {1..10} ; do curl http://172.16.72.254 ; done sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 sorry page for 2222222222222222222 //利用keepalived的sorry_server的配置也可实现 二.在VRRPD CONFIGURATION配置段的上下文中添加: notify_master <STRING>|<QUOTED-STRING> notify_backup <STRING>|<QUOTED-STRING> notify_fault <STRING>|<QUOTED-STRING> //设置当当前服务器状态发生变化时,所触发的脚本文件的路径及相关参数; 状态变动通知脚本示例:/etc/keepalived/notify.sh #!/bin/bash # CONTACT='root@localhost' notify() { MAILSUBJECT="$(hostname) changed state to $1, VIP reflact." #发送电子邮件的主题 MAILBODY="$(date +'%F %T'): VRRP STATUS CHANGED, $(hostname) change state to $1" #邮件的正文 echo "$MAILBODY" | mail -s "$MAILSUBJECT" "$CONTACT" } #case $1的$1指的是执行脚本时后面给定的参数 #notify()函数的$1指的是case引用notify()函数执行命令时notify后面的参数 case $1 in master) notify MASTER ;; backup) notify BACKUP ;; fault) notify FAULT ;; *) echo "Usage: $(basename $0) { master | backup | fault}" ;; esac 示例:两台调度器都要完成相同的操做 keepalived]# vim notify.sh keepalived]# chmod +x notify.sh keepalived]# vim keepalived.conf VRRP的配置段: vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 36 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } 测试: 1) 两台调度器重启keepalived服务;若此前keepalived主配置文件已添加notify的配置,无需重启服务; 2) 两台调度器都重启后,利用mail查看 查看调度器1(drct1)的邮箱: keepalived]# mail >N 11 root Fri May 3 23:58 18/698 "drct1 changed state to MASTER, VIP reflact." //调度器1(drct1)进入MASTER 查看调度器2(drct2)的邮箱: keepalived]# mail >N 11 root Fri May 3 23:58 18/698 "drct2 changed state to BACKUP, VIP reflact." //调度器2(drct2)进入BACKUP 3) 中止调度器1(drct1)的keepalived,利用mail查看 查看调度器2(drct2)的邮箱: keepalived]# mail >N 12 root Fri May 3 23:58 18/698 "drct2 changed state to MASTER, VIP reflact." //由于中止了调度器1(drct1),因此调度器2(drct2)进入MASTER 三.利用外部辅助脚本,完成MASTER/BACKUP的手动切换; keepalived容许调用外部辅助脚本,完成对于资源的监控,并根据健康的返回状态结果实现动态调整; 以下指令应该使用在vrrp实例定义以前 vrrp_script <STRING> { script "" interval <INT> weight -<INT> } 以下述指令使用的vrrp示例上下文之中; track_script { SCRIPT_NAME } 示例:在调度器1(drct1)上实现手动的MASTER/BACKUP切换 keepalived]# vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct1 vrrp_mcast_group4 224.0.100.18 } vrrp_script check_state { #script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #用上述配置keepalived启动时会显示:Unable to access script `[[`;此条vrrp_script不可用,没法被执行启动;改成以下; script "/etc/keepalived/keep_down.sh" interval 1 } vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 36 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0 } track_script { check_state } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } keepalived]# vim keep_down.sh #!/bin/bash # if [ -f /etc/keepalived/down ];then weight -20 fi keepalived]# chmod +x keep_down.sh keepalived]# touch down //调度器1(drct1)的优先级从100降到80(weight -20);此时调度器2(drct2)的优先级为90, //调度器1(drct1)为BACKUP;调度器2(drct2)为MASTER;可用mail查看状态转换的信息; keepalived]# rm -f down //调度器1(drct1)的优先级从80回到100;此时调度器2(drct2)的优先级为90,调度器2(drct2)的优先级为90; //调度器1(drct1)为MASTER;调度器2(drct2)为BACKUP;可用mail查看状态转换的信息; 四.利用外部辅助脚本,高可用其余的系统服务: 用两台调度器上的进行http服务的高可用实现 抢占式(默认)下的高可用http服务实现: 1.编辑两台调度器的keepalived的主配置文件 调度器1(drct1): keepalived]# vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct1 vrrp_mcast_group4 224.0.100.18 } vrrp_script check_state { #script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #用上述配置keepalived启动时会显示:Unable to access script `[[`;此条vrrp_script不可用,没法被执行启动;改成以下; script "/etc/keepalived/keep_down.sh" interval 1 } vrrp_script check_httpd { script "/usr/bin/killall -0 httpd && exit 0 || exit 1" interval 1 weight -20 } vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 36 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.100 } track_script { check_state check_httpd } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } 调度器2(drct2): keepalived]# vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct2 vrrp_mcast_group4 224.0.100.18 } vrrp_script check_state { #script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #用上述配置keepalived启动时会显示:Unable to access script `[[`;此条vrrp_script不可用,没法被执行启动;改成以下; script "/etc/keepalived/keep_down.sh" interval 1 } vrrp_script check_httpd { script "/usr/bin/killall -0 httpd && exit 0 || exit 1" interval 1 weight -20 } vrrp_instance VI_1 { state BACKUP interface eno16777736 virtual_router_id 36 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.100 } track_script { check_state check_httpd } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } //两台调度器的主配置文件中主要是加入了vrrp_script check_httpd和track_script这两个配置 3.配置两个调度器的http服务页面 4.测试 a.先开启调度器2(drct2)的keepalived和http服务: keepalived]# systemctl start keepalived.service keepalived]# ip addr list //查看是否有172.16.72.100这个VIP b.用客户端测试: ~]# curl http://172.16.72.100 sorry page for 2222222222222222222 c.再打开调度器1(drct1)的keepalived和http服务; keepalived]# systemctl start keepalived.service keepalived]# ip addr list //查看是否有172.16.72.100这个VIP,相应的调度器2(drct2)上的VIP就没有了;由于此时调度器1(drct1)为MASTER,调度器2(drct2)为BACKUP; d.用客户端测试: ~]# curl http://172.16.72.100 sorry page for 1 //此时访问到的页面已是调度器1(drct1)上的http配置的页面 非抢占式下的高可用http服务实现: 与抢占式不一样的是: 1.当A调度器的http服务故障时,B调度器立刻就会变成BACKUP状态,后来A调度器的http服务修复恢复后,不会当即抢占为MASTER状态,依然处于BACKUP状态;即便A比B优先级高; 2.直到B调度器的http服务故障时,A才会从BACKUP状态回到MASTER drct1的配置文件:/etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct1 vrrp_mcast_group4 224.0.100.18 } vrrp_script check_state { script "/etc/keepalived/keep_down.sh" interval 1 } vrrp_script check_httpd { script "/usr/bin/killall -0 httpd && exit 0 || exit 1" interval 1 weight -20 } vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 36 nopreempt priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.100 } track_script { check_state check_httpd } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } //注意: 1.killall命令写进脚本的时候用绝对命令的绝对路径,不然可能会引发vrrp_script执行失败 2.drct1的初始状态为MASTER,须要填写在vrrp_instance VI_1填写nopreempt 3.vrrp_script check_httpd{}中须要写weight drct2的配置文件:/etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id drct2 vrrp_mcast_group4 224.0.100.18 } vrrp_script check_state { script "/etc/keepalived/keep_down.sh" interval 1 } vrrp_script check_httpd { script "/usr/bin/killall -0 httpd && exit 0 || exit 1" interval 1 } vrrp_instance VI_1 { state BACKUP interface eno16777736 virtual_router_id 36 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.72.100 } track_script { check_state check_httpd } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } //注意: 1.killall命令写进脚本的时候用绝对命令的绝对路径,不然可能会引发vrrp_script执行失败; 2.drct2做为初始状态为BACKUP,因此不须要填写vrrp_instance VI_1中的nopreempt 3.vrrp_script check_httpd{}中不须要写weight 状态通知和转变状况脚本: keepalived]# vim notify.sh #!/bin/bash # CONTACT='root@localhost' notify() { MAILSUBJECT="$(hostname) changed state to $1, VIP reflact." #发送电子邮件的主题 MAILBODY="$(date +'%F %T'): VRRP STATUS CHANGED, $(hostname) change state to $1" #邮件的正文 echo "$MAILBODY" | mail -s "$MAILSUBJECT" "$CONTACT" } #case $1的$1指的是执行脚本时后面给定的参数 #notify()函数的$1指的是case引用notify()函数执行命令时notify后面的参数 case $1 in master) notify MASTER ;; backup) notify BACKUP systemctl start httpd.service ;; fault) notify FAULT systemctl start httpd.service ;; *) echo "Usage: $(basename $0) { master | backup | fault}" ;; esac keepalived]# chmox +x notify.sh //两台调度器上都须要此脚本 测试: a.先开启调度器2(drct2)的keepalived和http服务,客户端测试 ~]# curl http://172.16.72.100 sorry page for 2222222222222222222 b.而后开启调度器1(drct1)的keepalived服务但不开启http服务,客户端测试: ~]# curl http://172.16.72.100 sorry page for 2222222222222222222 //此时代表访问到的依然是drct2上http服务,而不是drct1; //由于若是访问到的是drct1,因为没有开启http服务,客户端获得的返回结果将会是: curl: (7) Failed connect to 172.16.72.100:80; Connection refused //这也间接证实了非抢占式的实现 c.开启调度器1(drct1)的http服务,客户端再测试: ~]# curl http://172.16.72.100 sorry page for 2222222222222222222 //返回结果依然是drct2的http服务配置的页面信息; d.关闭调度器2(drct2)的http服务,客户端测试: ~]# curl http://172.16.72.100 sorry page for 1 f.打开关闭调度器2(drct2)的http服务,客户端测试: ~]# curl http://172.16.72.100 sorry page for 1 //根据e,f能够看出,调度器2(drct2)的http服务从故障到恢复正常后,依然访问调度器1(drct1)的http服务配置的页面信息; 测试总结: 1.当调度器2的keepalived、http第一开启,调度器1尚未开启keepalived、http服务,调度器2先处于MASTER状态; 2.直到调度2的http服务下线,调度器1的状态才能从BACKUP变成MASTER状态,调度器1开启的http服务才能被应用; 3.只要调度器1的http、keepalived服务一直正常工做,MASTER的状态就会一直不变,返回的信息依然是调度器1的http服务配置的内容信息; 设计非抢占式下的高可用http服务实现的要点: 1.killall命令写进脚本的时候用绝对命令的绝对路径,不然可能会引发vrrp_script执行失败; 2.drct1的初始状态为MASTER,须要填写在vrrp_instance VI_1填写nopreempt 3.vrrp_script check_httpd{}中须要写weight 4.drct2做为初始状态为BACKUP,因此不须要填写vrrp_instance VI_1中的nopreempt 5.vrrp_script check_httpd{}中不须要写weight 6.状态通知和转变状况脚本(/etc/keepalived/notify.sh)必须根据状态的改变进行对应的编写
我的心得总结:
1.抢占式下的高可用http服务(其余服务)实现属于强制性地选用某个调度器做为MASTER(在没有其余约束的条件下);所以我以为配合外部辅助脚本:MASTER/BACKUP的手动切换的脚本使用比较好;
抢占式 + MASTER/BACKUP的手动切换的脚本
2.非抢占式下的高可用http服务(其余服务)实现能够根据调度器自身的状况进行MASTER/BACKUP状态转换;
3.二者比较:
a.抢占式的设计来得易懂些,逻辑简单;自动化较非抢占式低一些;
b.非抢占式的设计逻辑比抢占式复杂一点,自动化程度更高些;vim