搭建LVS+Keepalived高可用负载均衡集群

LVS+Keepalived 介绍

LVS
LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最先出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；
十种调度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）。
Keepalvied
Keepalived在这里主要用做RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现

集群IP地址规划

节点类型	ip	主机名
Master DR	192.168.2.128	centos.peng.com
Backup DR	192.168.2.129	centos1.peng.com
Real Server (nginx)	192.168.2.130	centos2.peng.com
Real Server (nginx)	192.168.2.131	centos3.peng.com
VIP	192.168.2.200	www.peng.com

安装LVS

检查系统是否支持lvs

[root@centos ~]# modprobe -l |grep ip_vs

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko

/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko

若有相似以上输出，说明系统支持IPVS模块

安装ipvsadm

[root@centos ~]# tar zxvf ipvsadm-1.24.tar.gz

[root@centos ~]# cd ipvsadm-1.24

[root@centos ipvsadm-1.24]# ln -s /usr/src/kernels/2.6.18-348.el5-x86_64/ /usr/src/linux/

[root@centos ipvsadm-1.24]# make

[root@centos ipvsadm-1.24]# make install

安装keepalived

[root@centos1 ~]# tar zxvf keepalived-1.1.19.tar.gz

[root@centos1 ~]# cd keepalived-1.1.19

[root@centos1 keepalived-1.1.19]#./configure  --with-kernel-dir=/usr/src/kernels/2.6.18-348.el5-x86_64

[root@centos1 keepalived-1.1.19]# make

[root@centos1 keepalived-1.1.19]# make install

[root@centos1 keepalived-1.1.19]# ln -s /usr/local/sbin/keepalived /sbin/

Real Server 配置

LVS的DR和TUN模式下，用户的请求不通过DR直接返回给用户，所以须要在Real Server上添加VIP。用脚本realserver.sh在Real Server上执行，实现自动添加VIP的功能。

#!/bin/bash

VIP=192.168.2.200      

/etc/rc.d/init.d/functions  

case "$1" in

start)

      /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP

      /sbin/route add -host $VIP dev lo:0

      echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

      echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

      echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

      sysctl -p >/dev/null 2>&1

      echo "RealServer Start OK"

      ;;

stop)

      /sbin/ifconfig lo:0 down

      /sbin/route del $VIP >/dev/null 2>&1

      echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

      echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

      echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

      echo "RealServer Stoped"

      ;;

*)

      echo "Usage: $0 {start|stop}"

      exit 1

esac

exit 0

配置keepalived

Keepalived的配置文件为/etc/Keepalived/Keepalived.conf

配置以下所示：

! Configuration File for keepalived

global_defs {

  notification_email {

         79907277@qq.com

  }

  notification_email_from 79907277@qq.com

  smtp_server smtp.qq.com

  smtp_connect_timeout 30

  router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state MASTER

   interface eth0

   virtual_router_id 51

priority 100

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.2.200

   }

}

virtual_server 192.168.2.200 80 {

   delay_loop 6

   lb_algo rr

   lb_kind DR

   nat_mask 255.255.255.0

   persistence_timeout 50

protocol TCP

real_server 192.168.2.130 80 {

       weight 1

       TCP_CHECK {

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

                 }

       }

   real_server 192.168.2.131 80 {

       weight 1

       TCP_CHECK {

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

                 }

       }

}

Backup DR的配置文件基本相同，state MASTER 换成 state backup ，priority 100 换成prority 80 便可。

启动keepalived 和realserver.sh

[root@centos keepalived]# /etc/init.d/keepalived start

[root@centos3 ~]# ./realserver.sh

   启动realserver.sh脚本后，Realserver ip地址以下所示：

[root@centos2 ~]# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0C:29:76:A3:21  

         inet addr:192.168.2.130  Bcast:192.168.2.255  Mask:255.255.255.0

         inet6 addr: fe80::20c:29ff:fe76:a321/64 Scope:Link

         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

         RX packets:8902 errors:0 dropped:0 overruns:0 frame:0

         TX packets:6316 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:1000

         RX bytes:2713466 (2.5 MiB)  TX bytes:1065526 (1.0 MiB)

lo        Link encap:Local Loopback  

         inet addr:127.0.0.1  Mask:255.0.0.0

         inet6 addr: ::1/128 Scope:Host

         UP LOOPBACK RUNNING  MTU:16436  Metric:1

         RX packets:4559 errors:0 dropped:0 overruns:0 frame:0

         TX packets:4559 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:0

         RX bytes:10562584 (10.0 MiB)  TX bytes:10562584 (10.0 MiB)

lo:0      Link encap:Local Loopback  

         inet addr:192.168.2.200  Mask:255.255.255.255

         UP LOOPBACK RUNNING  MTU:16436  Metric:1

测试LVS+keepalived

在两台Real Server上利用nginx新建两台web服务器，都将监听的虚拟VIP指向www.peng.com。经过两台客户端（一台win 7，一台opensuse）访问www.peng.com。访问结果以下所示：

Win7 下的IE访问：

Opensuse下的firefox访问：

能够看到，经过不一样客户端访问到的是不一样的realserver，一台是centos2，一台是centos3。

利用ipvsadm查看以下：

Lvs监听到了192.168.2.130:80和192.168.2.131:80两台RealServer，131产生了4个连接，130产生了1个连接。

由此能够肯定经过LVS+keepalived搭建的高可用负载均衡web服务顺利完成。

模拟DR故障

当Master DR当机的时候，咱们经过关闭keepalived服务来模拟这一场景。看一下，Backup DR是怎样继续提供服务的。观察此时Backup DR日志，以下所示：

[root@centos1 ~]# tail -f /var/log/messages

Aug  6 05:45:23 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Aug  6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Aug  6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Aug  6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:45:24 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:24 centos1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:29 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

能够看到Backup DR 立刻将本身调整为MASTER STATE ，并接管了虚拟ip192.168.2.200，让web服务仍然可以正常访问。

当从新启动Master DR上的keepalived服务，咱们的Master DR从当机中恢复以后。观察Backup DR日志以下：

Aug  6 05:45:24 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:24 centos1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:45:29 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Aug  6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Aug  6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

Aug  6 05:52:34 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 removed

Aug  6 05:52:34 centos1 Keepalived_healthcheckers: Netlink reflector reports IP192.168.2.200 removed

能够看到Backup DR 立刻又乖乖的将本身变成了BACKUP STATE 并移除了虚拟ip192.168.2.200，继续作着万年的备胎。

模拟RealServer故障

当RealServer当机的时候，咱们经过关闭RealServer上的 nginx服务来模拟这一场景。看一下，DR是怎样运行的。观察此时Master DR日志，以下所示：

Aug  6 05:52:34 centos Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Aug  6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Aug  6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Aug  6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:52:35 centos Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:52:35 centos Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added

Aug  6 05:52:40 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200

Aug  6 05:58:34 centos Keepalived_healthcheckers: TCP connection to [192.168.2.130:80] failed !!!

Aug  6 05:58:34 centos Keepalived_healthcheckers: Removing service [192.168.2.130:80] from VS [192.168.2.200:80]

Aug  6 05:58:34 centos Keepalived_healthcheckers: SMTP connection ERROR to [127.0.0.1:25].

能够看到keepalived检测到192.168.2.130:80服务当机，立刻将其从VS中移除。此时VS中应该只有192.168.2.131:80提供服务，咱们经过浏览器看一下，以下所示：

果然如此，经过两个客户端访问到的都是centos3提供的服务。