实现lvs调度及lvs+keeplivead高可用

时间 2020-05-26

标签实现 lvs 调度 lvs+keeplivead keeplivead 可用栏目负载均衡繁體版

原文原文链接

1.简述下lvs四种集群特色及使用场景html

LVS 有三种负载均衡的模式，分别是VS/NAT（nat 模式）,VS/DR(路由模式),VS/TUN（隧道模式）,VS/FULLNAT
1、NAT模式（VS-NAT）
原理：就是把客户端发来的数据包的IP头的目的地址，在负载均衡器上换成其中一台RS的IP地址并发至此RS来处理,RS处理完后把数据交给负载均衡器,负载均衡器再把数据包原IP地址改成本身的IP将目的地址改成客户端IP地址便可期间,不管是进来的流量,仍是出去的流量,都必须通过负载均衡器
优势：集群中的物理服务器可使用任何支持TCP/IP操做系统，只有负载均衡器须要一个合法的IP地址
缺点：扩展性有限。当服务器节点（普通PC服务器）增加过多时,负载均衡器将成为整个系统的瓶颈
由于全部的请求包和应答包的流向都通过负载均衡器。当服务器节点过多时
大量的数据包都交汇在负载均衡器那，速度就会变慢！
2、IP隧道模式（VS-TUN）
原理：首先要知道，互联网上的大多Internet服务的请求包很短小，而应答包一般很大那么隧道模式就是，把客户端发来的数据包，封装一个新的IP头标记(仅目的IP)发给RSRS收到后,先把数据包的头解开,还原数据包,处理后,直接返回给客户端,不须要再通过负载均衡器。注意,因为RS须要对负载均衡器发过来的数据包进行还原,因此说必须支持IPTUNNEL协议，因此,在RS的内核中,必须编译支持IPTUNNEL这个选项
优势：负载均衡器只负责将请求包分发给后端节点服务器，而RS将应答包直接发给用户因此，减小了负载均衡器的大量数据流动，负载均衡器再也不是系统的瓶颈，就能处理很巨大的请求量这种方式，一台负载均衡器可以为不少RS进行分发。并且跑在公网上就能进行不一样地域的分发。
缺点：隧道模式的RS节点须要合法IP，这种方式须要全部的服务器支持”IP Tunneling”(IP Encapsulation)协议，服务器可能只局限在部分Linux系统上
3、直接路由模式（VS-DR）
原理：负载均衡器和RS都使用同一个IP对外服务但只有DR对ARP请求进行响应全部RS对自己这个IP的ARP请求保持静默也就是说,网关会把对这个服务IP的请求所有定向给DR而DR收到数据包后根据调度算法,找出对应的RS,把目的MAC地址改成RS的MAC（由于IP一致）并将请求分发给这台RS这时RS收到这个数据包,处理完成以后，因为IP一致，能够直接将数据返给客户则等于直接从客户端收到这个数据包无异,处理后直接返回给客户端
因为负载均衡器要对二层包头进行改换,因此负载均衡器和RS之间必须在一个广播域也能够简单的理解为在同一台交换机上
优势：和TUN（隧道模式）同样，负载均衡器也只是分发请求，应答包经过单独的路由方法返回给客户端与VSTUN相比，VS-DR这种实现方式不须要隧道结构，所以可使用大多数操做系统作为物理服务器。
缺点：（不能说缺点，只能说是不足）要求负载均衡器的网卡必须与物理网卡在一个物理段上。
4、fullnat模式
lvs-fullnat：经过同时修改请求报文的源IP地址和目标IP地址进行转发
(1) VIP是公网地址，RIP和DIP是私网地址，且一般不在同一IP网络；所以，RIP的网关通常不会指向DIP
(2) RS收到的请求报文源地址是DIP，所以，只需响应给DIP；但Director还要将其发往Client
(3) 请求和响应报文都经由Director
(4) 支持端口映射
注意：此类型kernel默认不支持

2.描述LVS-DR工做原理，并配置实现node

原理：负载均衡器和RS都使用同一个IP对外服务但只有DR对ARP请求进行响应全部RS对自己这个IP的ARP请求保持静默也就是说,网关会把对这个服务IP的请求所有定向给DR而DR收到数据包后根据调度算法,找出对应的RS,把目的MAC地址改成RS的MAC（由于IP一致）并将请求分发给这台RS这时RS收到这个数据包,处理完成以后，因为IP一致，能够直接将数据返给客户则等于直接从客户端收到这个数据包无异,处理后直接返回给客户端
因为负载均衡器要对二层包头进行改换,因此负载均衡器和RS之间必须在一个广播域也能够简单的理解为在同一台交换机上

规划：c1,c2,c3,c4都是centos7.6
c1  客户端
c2  vs
c3  web1
c4  web2

2.1 安装web服务器
[root@c3 ~]# yum install httpd -y
[root@c3 ~]# echo rs1 > /var/www/html/index.html
[root@c3 ~]# systemctl start httpd
[root@c3 ~]# curl c3
rs1

[root@c4 ~]# yum install httpd -y
[root@c4 ~]# echo rs2 > /var/www/html/index.html
[root@c4 ~]# systemctl start httpd
[root@c4 ~]# curl c4
rs2
2.2 配置rs服务器
[root@c3 ~]# yum install net-tools -y   ###使用ifconfig命令须要安装net-tools包
[root@c3 ~]# cat rs.sh 
#!/bin/bash
vip=10.0.0.100
mask='255.255.255.255'
dev=lo:1
case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev $vip netmask $mask #broadcast $vip up
    #route add -host $vip dev $dev
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ;;
*)
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac
[root@c3 ~]# sh rs.sh start
[root@c3 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:f1:37:a8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.244/24 brd 10.1.1.255 scope global noprefixroute dynamic eth0
       valid_lft 14582sec preferred_lft 14582sec
    inet6 fe80::5025:c937:77d0:2b28/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[root@c4 ~]# yum install net-tools -y
[root@c4 ~]# sh rs.sh start
[root@c4 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:05:32:f0 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.245/24 brd 10.1.1.255 scope global noprefixroute dynamic eth0
       valid_lft 16671sec preferred_lft 16671sec
    inet6 fe80::96c3:3cc3:b39e:dee3/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
2.3 配置vs服务器
[root@c2 ~]# yum install -y ipvsadm  ###安装lvs包
[root@c2 ~]# cat vs.sh 
#!/bin/bash
vip='10.0.0.100'
iface='lo:1'
mask='255.255.255.255'
port='80'
rs1='10.1.1.244'
rs2='10.1.1.245'
scheduler='rr'      ###为了测试容易出效果，采用rr轮询算法
type='-g'
case $1 in
start)
    ifconfig $iface $vip netmask $mask #broadcast $vip up
    iptables -F
    ipvsadm -A -t ${vip}:${port} -s $scheduler
    ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
    ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
    ;;
stop)
    ipvsadm -C
    ifconfig $iface down
    ;;
*)
    echo "Usage $(basename $0) start|stop"
    exit 1
esa
[root@c2 ~]# sh vs.sh start
[root@c2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
[root@c2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 rr
  -> 10.1.1.244:80                Route   1      0          0         
  -> 10.1.1.245:80                Route   1      0          0
2.4 测试：
[root@c1 ~]# route -n     ###无去往10.0.0.0网段的路由
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.1.254      0.0.0.0         UG    100    0        0 eth0
10.1.1.0        0.0.0.0         255.255.255.0   U     100    0        0 eth0
[root@c1 ~]# curl 10.0.0.100
^C
[root@c1 ~]# route add -host 10.0.0.100 dev eth0  ###增长去往10.0.0.0网段的路由
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2

3.实现LVS+Keepalived高可用web

keepalive使用vrrp（虚拟路由协议）实现负载冗余功能。路由器经过周期的组播通告宣称本身是主路由器，而后与网络中的路由器对比优先级，以选出主、备路由器。主路由器提供相对应的路由功能，备路由器在主路由器故障的时候再次对比优先级选出新的主路由器提供服务，其他的成为备份路由器。此场景在第2小节的基础上完成，c2是主路由器，c5是备路由器

3.1 实现c2与c5互为免密登陆
[root@c2 ~]# ssh-keygen -t rsa -P "" ###生成密钥
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:C1wDPspULjsjJQs/hSUjac50V4BLQXCQFkwxbViT/DM root@c2
The key's randomart image is:
+---[RSA 2048]----+
|+O#*=.=.         |
|.Oo&.= .         |
|B * O + o        |
| = O E o .       |
|  = * = S        |
|   o o . .       |
|        .        |
|                 |
|                 |
+----[SHA256]-----+
[root@c2 ~]# ssh-copy-id c5 ###传输密钥
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c5 (10.1.1.246)' can't be established.
ECDSA key fingerprint is SHA256:ilZ46J85JC8Xhr2dVvYsUxMGyj17SDhD6/JrhmNy6GY.
ECDSA key fingerprint is MD5:2f:c5:a9:d6:d7:5f:5e:4e:c3:94:7c:92:3a:d2:55:63.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c5's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'c5'"
and check to make sure that only the key(s) you wanted were added.

[root@c2 ~]# ssh c5   ###测试免密登陆
Last login: Mon May 25 21:40:03 2020 from 192.168.10.45
[root@c5 ~]# 

[root@c5 ~]# ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:abVEGoN7+mbpGU0aZY4VssjdndMC+cjLZYK5Icy+S/U root@c5
The key's randomart image is:
+---[RSA 2048]----+
|       .+ +.     |
|     ..o O.+ o   |
|     oo.++Bo= .  |
|      = =X+.+o   |
|     . +S++=     |
|      oo.*o      |
|      .oo.E      |
|     .. =o       |
|      .=o        |
+----[SHA256]-----+
[root@c5 ~]# ssh-copy-id c2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'c2 (10.1.1.243)' can't be established.
ECDSA key fingerprint is SHA256:dldJTKtxApZyQT/FT6WKQsqKgtf4cPuAxBTiLMFdxSk.
ECDSA key fingerprint is MD5:1a:07:07:69:3f:0e:94:b3:f3:c5:04:dc:73:6b:ba:3e.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@c2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'c2'"
and check to make sure that only the key(s) you wanted were added.

[root@c5 ~]# ssh c2
Last login: Mon May 25 21:40:01 2020 from 192.168.10.45
[root@c2 ~]# 
3.2 安装与配置keepalived
3.2.1 先清除c2上的ipvsadm策略
[root@c2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 rr
  -> 10.1.1.244:80                Route   1      0          0         
  -> 10.1.1.245:80                Route   1      0          0         
[root@c2 ~]# ls
anaconda-ks.cfg  original-ks.cfg  vs.sh
[root@c2 ~]# sh vs.sh stop
[root@c2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

3.2.2 安装keepalived服务，ipvsadm工具
[root@c2 ~]# yum install keepalived.x86_64 -y
[root@c5 ~]# yum install keepalived -y
[root@c5 ~]# yum install ipvsadm -y

[root@c2 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id node1
   vrrp_mcast_group4 224.0.100.100
}

vrrp_instance VI_1 {
    state MASTER
    interface bond0
    virtual_router_id 5
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.100/24 dev bond0 label bond0:0
    }
}

virtual_server 10.0.0.100 80 {
    delay_loop 1
    lb_algo rr
    lb_kind DR
    protocol TCP

    real_server 10.1.1.244 80 {
        weight 1   ###后端服务检测
        HTTP_GET {          
        url {
            path /
            status_code 200
        }
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    real_server 10.1.1.245 80 {
        weight 1
        HTTP_GET {
                url {
                        path /
                        status_code 200
                }       
            } 
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }

    }
}
[root@c2 keepalived]# systemctl start keepalived
[root@c2 keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 00:0c:29:ba:03:94 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 00:0c:29:ba:03:9e brd ff:ff:ff:ff:ff:ff
7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0c:29:ba:03:94 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.243/24 brd 10.1.1.255 scope global noprefixroute bond0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/24 scope global bond0:0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feba:394/64 scope link 
       valid_lft forever preferred_lft forever
[root@c5 keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id node2
   vrrp_mcast_group4 224.0.100.100
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 5
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        10.0.0.100/24 dev eth0 label eth0:0
    }
}

virtual_server 10.0.0.100 80 {
    delay_loop 1
    lb_algo rr
    lb_kind DR
    protocol TCP

    real_server 10.1.1.244 80 {
        weight 1
        HTTP_GET {
        url {
            path /
            status_code 200
        }
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    real_server 10.1.1.245 80 {
        weight 1
        HTTP_GET {
                url {
                        path /
                        status_code 200
                }       
            } 
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }

    }
}
[root@c5 keepalived]# systemctl start keepalived.service

3.3 测试
3.3.1 先测试lvs是否正常工做
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# 
3.3.2 停掉c2上的keepalived服务，再测试是否正常调度
[root@c2 keepalived]# systemctl stop keepalived

[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
rs1
[root@c1 ~]# curl 10.0.0.100
rs2
[root@c1 ~]# curl 10.0.0.100
3.3.3 测试后端服务器的健康性检查
[root@c3 ~]# systemctl stop httpd
[root@c1 ~]# while true;do curl 10.0.0.100;sleep 1;done
rs2
rs1
rs2
rs1
rs2
curl: (7) Failed connect to 10.0.0.100:80; Connection refused
rs2
curl: (7) Failed connect to 10.0.0.100:80; Connection refused
rs2
rs2
rs2
rs2
rs2
rs2
rs2

[root@c3 ~]# systemctl start httpd
[root@c1 ~]# while true;do curl 10.0.0.100;sleep 1;done
rs2
rs2
rs2
rs2
rs1
rs2
rs1
rs2
rs1
rs2
rs1
rs2
rs1