lvs_nat

nat模式环境模拟

1.实验环境概览

192.168.1.186   192.168.75.131          双网卡         dir     调度器(vip)        ipvsadm，双网卡
192.168.1.185                           单网卡         rs1     真实服务器1      nginx，网关指定调度器(vip)
192.168.1.190                           单网卡         rs2     真实服务器2      nginx，网关指定调度器(vip)
192.168.75.150                          单网卡(双网卡)    客户端 (能够不用配置，直接使用vip测试，能够跟调度器同样的配置，方便远程链接)

2.环境准备

• 准备三台虚拟机(网卡为桥接模式，单网卡，直接安装下一步或者克隆；能上网)，客户端能够不用配置,关闭iptables和selinux
• 分别修改hostname(vim /etc/sysconfig/network)；分别为vip，real_server1，real_server2
• 在vip上安装ipvsadm，两台real_server安装nginx
• 在real_server1上执行 echo "1111111111111master" > /usr/share/nginx/html/index.html，修改网卡网关
• 在real_server2上执行 echo "2222222222222slave" > /usr/share/nginx/html/index.html，修改网卡网关

vip服务器上增长网卡：在vmware上关闭vip，

1. "编辑虚拟机">"添加">"网络适配器">"下一步">"NAT模式">"完成"
2. "虚拟网络编辑器">"VMnet8" 点击下方 NAT设置查看网关(个人是192.168.75.2)
3. 查看"DHCP设置"查看开启的网段(设置外网网段的设置范围)
4. 开启vip，执行vim /etc/udev/rules.d/70-persistent-net.rules，记录ATTR{address}，即MAC地址
5. cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1
6. vim /etc/sysconfig/network-scripts/ifcfg-eth1
7. 设置ip为192.168.75.131，网关为192.168.75.2，HWADDR填写刚刚记录的网卡，设置开机启动，重启网卡

进行试验

1.在调度器上新建脚本

vim /usr/local/sbin/lvs_nat.sh

#! /bin/bash
# director 服务器上开启路由转发功能: 
echo 1 > /proc/sys/net/ipv4/ip_forward
# 关闭icmp的重定向
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/eth1/send_redirects

iptables -F
iptables -t nat -F
iptables -t nat -X
#iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE
#这条命令并无关联
ipvsadm -C
#ipvsadm -A -t 192.168.75.131:80 -s rr -p 300   #-p 300   长连接300秒(会致使5分钟不更换服务器)
ipvsadm -A -t 192.168.75.131:80 -s wrr
#wrr表示权重轮询，根据权重来轮询
ipvsadm -a -t 192.168.75.131:80 -r 192.168.1.185:80 -m -w 1
ipvsadm -a -t 192.168.75.131:80 -r 192.168.1.190:80 -m -w 2

2.执行脚本

sh /usr/local/sbin/lvs_nat.sh

3.测试结果

在vip上执行

curl 192.168.75.131

会根据权重来返回结果

抓包结果

[root@real_server1 ~]# tcpdump -nn -i eth0 host 192.168.75.150 and ! port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
01:00:20.379751 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [S], seq 4207417558, win 14600, options [mss 1460,sackOK,TS val 4562638 ecr 0,nop,wscale 6], length 0
01:00:20.379839 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [S.], seq 3392244670, ack 4207417559, win 14480, options [mss 1460,sackOK,TS val 5501088 ecr 4562638,nop,wscale 6], length 0
01:00:20.388310 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 0
01:00:20.388566 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 175
01:00:20.388772 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [.], ack 176, win 243, options [nop,nop,TS val 5501096 ecr 4562642], length 0
01:00:20.389948 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5501098 ecr 4562642], length 245
01:00:20.395679 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4562654 ecr 5501098], length 0
01:00:20.425383 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4562657 ecr 5501098], length 0
01:00:20.426466 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5501134 ecr 4562657], length 0
01:00:20.436678 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4562695 ecr 5501134], length 0
01:00:57.176788 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [S], seq 704010524, win 14600, options [mss 1460,sackOK,TS val 4599425 ecr 0,nop,wscale 6], length 0
01:00:57.178908 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [S.], seq 2021861845, ack 704010525, win 14480, options [mss 1460,sackOK,TS val 5533840 ecr 4599425,nop,wscale 6], length 0
01:00:57.187976 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 0
01:00:57.188021 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 175
01:00:57.188359 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [.], ack 176, win 243, options [nop,nop,TS val 5533851 ecr 4599444], length 0
01:00:57.189733 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5533852 ecr 4599444], length 245
01:00:57.200746 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4599454 ecr 5533852], length 0
01:00:57.200787 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4599456 ecr 5533852], length 0
01:00:57.200820 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5533861 ecr 4599456], length 0
01:00:57.212718 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4599460 ecr 5533861], length 0
01:02:09.403093 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [S], seq 970535959, win 14600, options [mss 1460,sackOK,TS val 4671656 ecr 0,nop,wscale 6], length 0
01:02:09.403203 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [S.], seq 1701915808, ack 970535960, win 14480, options [mss 1460,sackOK,TS val 5610111 ecr 4671656,nop,wscale 6], length 0
01:02:09.412386 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4671667 ecr 5610111], length 0
01:02:09.412460 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4671669 ecr 5610111], length 175
01:02:09.412507 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [.], ack 176, win 243, options [nop,nop,TS val 5610120 ecr 4671669], length 0
01:02:09.423352 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5610131 ecr 4671669], length 245
01:02:09.434206 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4671689 ecr 5610131], length 0
01:02:09.434294 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4671689 ecr 5610131], length 0
01:02:09.435994 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5610143 ecr 4671689], length 0
01:02:09.439232 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4671697 ecr 5610143], length 0

[root@vip ~]# tcpdump -nn -i eth1 host 192.168.75.150 and ! port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
00:58:58.218430 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [S], seq 2818675909, win 14600, options [mss 1460,sackOK,TS val 4481342 ecr 0,nop,wscale 6], length 0
00:58:58.219493 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [S.], seq 4017612900, ack 2818675910, win 14480, options [mss 1460,sackOK,TS val 5419792 ecr 4481342,nop,wscale 6], length 0
00:58:58.223052 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4481347 ecr 5419792], length 0
00:58:58.247015 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4481349 ecr 5419792], length 175
00:58:58.250997 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [.], ack 176, win 243, options [nop,nop,TS val 5419821 ecr 4481349], length 0
00:58:58.251869 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5419824 ecr 4481349], length 245
00:58:58.253316 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4481377 ecr 5419824], length 0
00:58:58.253974 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4481378 ecr 5419824], length 0
00:58:58.263118 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5419836 ecr 4481378], length 0
00:58:58.265457 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4481390 ecr 5419836], length 0
00:59:03.219316 ARP, Request who-has 192.168.75.150 tell 192.168.75.131, length 28
00:59:03.221931 ARP, Reply 192.168.75.150 is-at 00:0c:29:5a:c3:3e, length 46
01:00:19.513727 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [S], seq 4207417558, win 14600, options [mss 1460,sackOK,TS val 4562638 ecr 0,nop,wscale 6], length 0
01:00:19.514865 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [S.], seq 3392244670, ack 4207417559, win 14480, options [mss 1460,sackOK,TS val 5501088 ecr 4562638,nop,wscale 6], length 0
01:00:19.520316 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 0
01:00:19.520645 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 175
01:00:19.525721 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [.], ack 176, win 243, options [nop,nop,TS val 5501096 ecr 4562642], length 0
01:00:19.526217 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5501098 ecr 4562642], length 245
01:00:19.529640 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4562654 ecr 5501098], length 0
01:00:19.559466 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4562657 ecr 5501098], length 0
01:00:19.561567 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5501134 ecr 4562657], length 0
01:00:19.570684 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4562695 ecr 5501134], length 0
01:00:24.515461 ARP, Request who-has 192.168.75.150 tell 192.168.75.131, length 28
01:00:24.526927 ARP, Reply 192.168.75.150 is-at 00:0c:29:5a:c3:3e, length 46
01:00:56.305273 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [S], seq 704010524, win 14600, options [mss 1460,sackOK,TS val 4599425 ecr 0,nop,wscale 6], length 0
01:00:56.317205 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [S.], seq 2021861845, ack 704010525, win 14480, options [mss 1460,sackOK,TS val 5533840 ecr 4599425,nop,wscale 6], length 0
01:00:56.321997 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 0
01:00:56.322218 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 175
01:00:56.323050 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [.], ack 176, win 243, options [nop,nop,TS val 5533851 ecr 4599444], length 0
01:00:56.324052 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5533852 ecr 4599444], length 245
01:00:56.331228 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4599454 ecr 5533852], length 0
01:00:56.331974 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4599456 ecr 5533852], length 0
01:00:56.332944 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5533861 ecr 4599456], length 0
01:00:56.335788 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4599460 ecr 5533861], length 0
01:01:01.317186 ARP, Request who-has 192.168.75.150 tell 192.168.75.131, length 28
01:01:01.318083 ARP, Reply 192.168.75.150 is-at 00:0c:29:5a:c3:3e, length 46

nat模式在收到请求后，客户端包进入调度器后，调度器修改目的地址；

发送给后端real_server处理，real_server处理完后再返回给调度器；

调度器再修改源地址，返回给客户

为何在real_server上能接收到另外的服务器包的信息？相互之间会传播信息（这个不是很理解）

看了抓包后再分析nat与fullnat模式的区别

• 为何nat模式下须要将real_server的网关修改为vip的内网地址？
  nat模式下，vip将请求转发到后端，是经过修改dst来发送给后端real_server。
  
  但real_server返回包却出现问题，由于real_server是单网卡，须要将包发送给另外一个网段的机器
  
  须要通过网关来调度，但默认网关却不能实现这个功能，只有被设为vip的机器能实现。
  
  只有将real_server网关设置成调度器，返回数据包才会发送给调度器
  
  同时，网关会将返回包的源ip换成vip。(最终目的)
  
  因此要将real_server的网关设置为vip的内网地址

• fullnat就没有这个问题
  fullnat模式下，vip的功能是同时改变src跟dst(源地址和目的地址)，那样就不须要将real_server网关设置成vip的内网地址
  
  由于vip与real_server都在同一网段，因此就不须要修改网关

• fullnat缺点
  RealServer没法得到用户IP；淘宝经过叫TOA的方式解决的，
  
  主要原理是：将client address放到了TCP Option里面带给后端RealServer，RealServer收到后保存在socket
  
  的结构体里并经过toa内核模块hook了getname函数，这样当用户调用getname获取远端地址时，返回的是保
  
  存在socket的TCPOption的IP. 百度的BVS是经过叫ttm模块实现的，其实现方式跟toa基本同样，只是没有开源。