记一次bond引发的网络故障

时间 2019-12-14

原文原文链接

本案中3个关键服务器
物理服务器：192.168.6.63，简称P，(Physical server)
KVM-VM：192.168.6.150，是物理服务器P上的一个KVM虚机，简称VM
NAS：外部NAS服务器，用来作ping/arp测试服务器，简称NAShtml

物理服务器P的配置：linux

#uname -a
Linux cz63 4.15.18-11-pve #1 SMP PVE 4.15.18-34 (Mon, 25 Feb 2019 14:51:06 +0100) x86_64 GNU/Linux

#cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp3s0f0
iface enp3s0f0 inet manual

auto enp3s0f1
iface enp3s0f1 inet manual

auto ens1f0
iface ens1f0 inet manual

auto ens1f1
iface ens1f1 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves enp3s0f0 enp3s0f1
    bond-miimon 100
    bond-mode balance-rr

auto bond1
iface bond1 inet manual
    bond-slaves ens1f0 ens1f1
    bond-miimon 100
    bond-mode balance-rr

auto vmbr0
iface vmbr0 inet static
    address  192.168.6.63
    netmask  255.255.255.0
    gateway  192.168.6.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet static
    address  10.1.1.63
    netmask  255.255.255.0
    bridge-ports bond1
    bridge-stp off
    bridge-fd 0
#brctl show
bridge name bridge id       STP enabled interfaces
vmbr0       8000.ac1f6b342094   no      bond0
                            tap401000001i0
vmbr1       8000.74a4b500e768   no      bond1

安装KVM-VM，以后VM与外部服务器的链接很是不稳定，90%状况下会出现ping不可达。这里测试了centos、ubuntu、win7都是同样的不稳定。
在VM内执行ping 192.168.6.40（NAS）
结果ping不通。
此时，在P上执行
tcpdump -leni vmbr0 arp
tcpdump -leni tap401000001i0 arp
发现状况以下：shell

P上
#tcpdump -leni vmbr0 arp | grep 2a:f0:5f:ae:c9:8b
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46

P上
#tcpdump -leni tap401000001i0  arp | grep 2a:f0:5f:ae:c9:8b
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

NAS上
# tcpdump -leni vmx0 arp | grep c9:8b
02:03:50.930907 2a:f0:5f:ae:c9:8b > 00:50:56:87:86:b9, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 tell 192.168.6.150, length 46
02:03:50.930923 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 42: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 28
02:04:02.669823 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:02.670131 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:03.670770 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:03.671059 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:04.672736 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:04.672992 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:06.671878 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:06.672021 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:07.674726 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:07.674773 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:08.676733 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:08.676868 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46
02:04:10.673678 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46
02:04:10.674026 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

vmbr0上发现了2个request包和1个reply包
tap401000001i0上仅发现了request包，并且length不一致，未发现reply包。
NAS上一切正常
看到的现象就是VM在发送icmp以前须要先得到NAS的MAC地址，因此须要先经过ARP广播获取到NAS的MAC地址。
ubuntu

P的vmbr0异常：接收到2个request，1个reply
P的vmbr0异常：接收到reply未发送至tap401000001i0
因为以上2个缘由，致使VM没法收到reply，因此icmp不能进行，ping失败。

问题定位在数据链路层的ARP协议上。
在VM上执行arping -c 10 192.168.6.40
现象依据。再次证实以上2个现象。windows

# brctl show
bridge name bridge id       STP enabled interfaces
vmbr0       8000.ac1f6b342094   no      bond0
                            tap401000001i0
vmbr1       8000.74a4b500e768   no      bond1
# brctl showstp  vmbr0
vmbr0
 bridge id      8000.ac1f6b342094
 designated root    8000.ac1f6b342094
 root port         0            path cost          0
 max age          20.00         bridge max age        20.00
 hello time        2.00         bridge hello time      2.00
 forward delay         0.00         bridge forward delay       0.00
 ageing time         300.00
 hello timer           0.00         tcn timer          0.00
 topology change timer     0.00         gc timer           4.18
 flags


bond0 (1)
 port id        8001            state            forwarding
 designated root    8000.ac1f6b342094   path cost          4
 designated bridge  8000.ac1f6b342094   message age timer      0.00
 designated port    8001            forward delay timer    0.00
 designated cost       0            hold timer         0.00
 flags

tap401000001i0 (2)
 port id        8002            state            forwarding
 designated root    8000.ac1f6b342094   path cost        100
 designated bridge  8000.ac1f6b342094   message age timer      0.00
 designated port    8002            forward delay timer    0.00
 designated cost       0            hold timer         0.00
 flags

# brctl showmacs vmbr0
port no mac addr        is local?   ageing timer
  1 2a:f0:5f:ae:c9:8b   no         1.13
  1 00:50:56:87:e3:6d   no         0.00
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
# ip rule show
0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default
# ip route show table all
default via 192.168.6.1 dev vmbr0 onlink
10.1.1.0/24 dev vmbr1 proto kernel scope link src 10.1.1.63
192.168.6.0/24 dev vmbr0 proto kernel scope link src 192.168.6.63
broadcast 10.1.1.0 dev vmbr1 table local proto kernel scope link src 10.1.1.63
local 10.1.1.63 dev vmbr1 table local proto kernel scope host src 10.1.1.63
broadcast 10.1.1.255 dev vmbr1 table local proto kernel scope link src 10.1.1.63
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.6.0 dev vmbr0 table local proto kernel scope link src 192.168.6.63
local 192.168.6.63 dev vmbr0 table local proto kernel scope host src 192.168.6.63
broadcast 192.168.6.255 dev vmbr0 table local proto kernel scope link src 192.168.6.63
# bridge vlan show
port    vlan ids
bond0    1 PVID Egress Untagged

vmbr0    1 PVID Egress Untagged

bond1    1 PVID Egress Untagged

vmbr1    1 PVID Egress Untagged

tap401000001i0   1 PVID Egress Untagged

以上一切正常，未发现输出问题，思路该怎么展开？centos

google了一下午加一个晚上，各类不解决。
最后看到了linux关于bond的说明：
https://forum.huawei.com/enterprise/zh/thread-282727.html
再看看P的网卡，明明是mode0啊！
这时候再去交换机上一看！哈啊~！服务器

interface GigabitEthernet0/0/5
 port link-type access
 port default vlan 6
#
interface GigabitEthernet0/0/6
 port link-type access
 port default vlan 6
#
interface GigabitEthernet0/0/7
 port link-type access
 port default vlan 6
#
interface GigabitEthernet0/0/8
 port link-type access
 port default vlan 6

感受好像是这个问题。
而后ifdown ens1f1，问题解决。
VM与全部外部网络通信正常，无任何故障现象。
经过阅读华为网站的文章，肯定了balance-rr须要交换机作修改。
不想劳烦网管，因此本身改为mode6。
一切归于平静。
网络

问题只是临时解决了，可是产生现象的原理还没明白。

2009年本身就详细看了windows teaming linux bond的原理及配置，苦于只是知道OS层面，不知道switch侧应该怎么对应，因此知识点一直有欠缺，2017年、2018年都曾经入坑，惋惜哪里有2台物理服务器和物理交换机实践一下呀！
此次总算补齐了OS和switch的配置。

能用mode6就用mode6，再其次就是mode4。

如下给本身看的:tcp

brctl show                          #查看bridge信息
brctl showstp  vmbr0                #查看vnbr0的stp信息，是否转发等
brctl showmacs vmbr0 | grep c9:8b   #查看bridge的MAC地址
tcpdump -leni vmbr0 arp             #debug vmbr0网卡的arp包信息
tcpdump -leni vmbr0 icmp            #debug vmbr0网卡的icmp包信息
iptables -A FORWARD -i vmbr0 -o vmbr0 -j ACCEPT
iptables -L     #查看iptables
ebtables -L     #查看数据链路层的table信息
arping -c 10 192.168.1.1            #arp ping
ip rule show
ip route show table local
ip route show table all
bridge monitor          #monitor fdb update
bridge vlan show        #查看bridge的vlan信息

ping -c 1 -I veth1 192.168.3.1      #指定网卡进行ping测试

/proc/sys/net/bridge/bridge-nf-call-iptables    #干吗的
/proc/sys/net/ipv4/ip_forward                   #多网卡下，数据包转发

sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables中的0或者1表明什么意思？？

echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore        #什么意思？？
echo 8 > /proc/sys/net/ipv4/conf/eth0/arp_announce      #什么意思？？