从零开始学习docker（二十）RoutingMesh--Ingress负载均衡

时间 2019-11-12

标签开始学习 docker 二十 routingmesh ingress 负载均衡栏目 Docker 繁體版

原文原文链接

咱们知道容器之间的通讯，例如上图中的10.0.9.3与10.0.9.5通讯是经过overlay网络，是经过一个VXLAN tannel来实现的。web

可是service和service之间通讯是经过VIP实现的。例如client的service与web 的service进行通讯，而web有一个scale，所以client访问web是经过访问虚拟IP（VIP）来实现的。那么VIP是怎么映射到具体的10.0.9.5或者10.0.9.6呢？这是经过LVS实现的。docker

什么是LVS？

LVS，Linux Virtual Server。能够实如今系统级别的负载均衡。shell

咱们能够在三个节点上任何一个节点访问80端口均可以访问到wordpress，这个实现就是IngressNetWork的做用。任何一台swarm节点上去访问端口服务时，会经过端口服务经过本节点IPVS（IP Virtual Service），并经过LVS给loadbanlance到真正具备service上面，例如上图中咱们经过访问Docker Host3 bash

转发到另外两个节点中。网络

咱们的实验环境跟上一节同样，咱们将whoami 的scale 变为2app

iie4bu@swarm-manager:~$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                   PORTS
h4wlczp85sw5        client              replicated          1/1                 busybox:1.28.3          
9i6wz6cg4koc        whoami              replicated          3/3                 jwilder/whoami:latest   *:8000->8000/tcp
iie4bu@swarm-manager:~$ docker service scale whoami=2
whoami scaled to 2
overall progress: 2 out of 2 tasks 
1/2: running   [==================================================>] 
2/2: running   [==================================================>] 
verify: Service converged

查看whoami的运行状况：负载均衡

iie4bu@swarm-manager:~$ docker service ps whoami
ID                  NAME                IMAGE                   NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
6hhuf528spdw        whoami.1            jwilder/whoami:latest   swarm-manager       Running             Running 17 hours ago                                                                
9idgk9jbrlcm        whoami.3            jwilder/whoami:latest   swarm-worker2       Running             Running 16 hours ago

分别运行在swarm-manager和swarm-worker2节点上。curl

在swarm-manager中访问whoami：tcp

iie4bu@swarm-manager:~$ curl 127.0.0.1:8000
I'm cc9f97cc5056
iie4bu@swarm-manager:~$ curl 127.0.0.1:8000
I'm f47e05019fd9

在swarm-worker1中访问whoami：wordpress

iie4bu@swarm-worker1:~$ curl 127.0.0.1:8000
I'm f47e05019fd9
iie4bu@swarm-worker1:~$ curl 127.0.0.1:8000
I'm cc9f97cc5056

为何对于swarm-worker1来说本地并无whoami的service，确能访问8000端口？

经过iptables能够看到本地的转发规则：

iie4bu@swarm-worker1:~$ sudo iptables -nL -t nat
[sudo] password for iie4bu: 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER-INGRESS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER-INGRESS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  172.19.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  172.18.0.0/16        0.0.0.0/0           
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:443
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:80
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:22

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.17.0.2:443
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:81 to:172.17.0.2:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:23 to:172.17.0.2:22

Chain DOCKER-INGRESS (2 references)
target     prot opt source               destination         
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 to:172.19.0.2:8000
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

咱们看到DOCKER-INGRESS，它的转发规则是若是咱们访问tcp8000duank端口，它会转发到172.19.0.2:8000上，那么这个172.19.0.2:8000是什么呢？

咱们先看一下本地的ip:

iie4bu@swarm-worker1:~$ ifconfig
br-3f2fc691f5da Link encap:Ethernet  HWaddr 02:42:c8:f4:03:ad  
          inet addr:172.18.0.1  Bcast:172.18.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

docker0   Link encap:Ethernet  HWaddr 02:42:43:69:b7:60  
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::42:43ff:fe69:b760/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:83 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:9208 (9.2 KB)

docker_gwbridge Link encap:Ethernet  HWaddr 02:42:41:bf:4d:15  
          inet addr:172.19.0.1  Bcast:172.19.255.255  Mask:255.255.0.0
          inet6 addr: fe80::42:41ff:febf:4d15/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:42 errors:0 dropped:0 overruns:0 frame:0
          TX packets:142 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3556 (3.5 KB)  TX bytes:13857 (13.8 KB)
......
......

咱们能够看到本地有个网络是docker_gwbridge，它的ip是172.19.0.1，这个地址与172.19.0.2在同一个网段。因此说咱们能够猜想172.19.0.2确定是与docker_gwbridge相连的一个网络。能够经过brctl show 查看：

iie4bu@swarm-worker1:~$ brctl show
bridge name	bridge id		STP enabled	interfaces
br-3f2fc691f5da		8000.0242c8f403ad	no		
docker0		8000.02424369b760	no		veth75a496d
docker_gwbridge		8000.024241bf4d15	no		veth400f4b4
							veth44af5f8

能够看到docker_gwbridge有两个interface，这两个interface哪一个是呢？

经过docker network ls：

iie4bu@swarm-worker1:~$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
bdf23298113d        bridge              bridge              local
969e60257ba5        docker_gwbridge     bridge              local
cdcffe1b31cb        host                host                local
uz1kgf9j6m48        ingress             overlay             swarm
cojus8blvkdo        my-demo             overlay             swarm
3f2fc691f5da        network_default     bridge              local
dba4587ee914        none                null                local

查看docker_gwbridge的详细信息：

iie4bu@swarm-worker1:~$ docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "969e60257ba50b070374d31ea43a0550d6cd3ae3e68623746642fe8736dee5a4",
        "Created": "2019-04-08T09:47:59.343371327+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "5559895ccaea972dad4f2fe52c0ec754d2d7c485dceb35083719768f611552e7": {
                "Name": "gateway_bf5031da0049",
                "EndpointID": "6ad54f228134798de719549c0f93c804425beb85dd97f024408c4f7fc393fdf9",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "177757eca7a18630ae91c01b8ac67bada25ce1ea050dad4ac5cc318093062003",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]

能够看到与docker_gwbridge相连的container有两个，分别是gateway_bf5031da0049和gateway_ingress-sbox，而gateway_ingress-sbox的ip正是172.19.0.2。也就是说数据被转发到这个gateway_ingress-sbox network namespace中去了。

进入gateway_ingress-sbox：

iie4bu@swarm-worker1:~$ sudo ls /var/run/docker/netns
1-cojus8blvk  1-uz1kgf9j6m  44e6e70b2177  b1ba5b4dd9f2	bf5031da0049  ingress_sbox
iie4bu@swarm-worker1:~$ sudo nsenter --net=//var/run/docker/netns/ingress_sbox
-bash: /home/iie4bu/anaconda3/etc/profiel.d/conda.sh: No such file or directory

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
If your shell is Bash or a Bourne variant, enable conda for the current user with

    $ echo ". /home/iie4bu/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc

or, for all users, enable conda with

    $ sudo ln -s /home/iie4bu/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh

The options above will permanently enable the 'conda' command, but they do NOT
put conda's base (root) environment on PATH.  To do so, run

    $ conda activate

in your terminal, or to put the base environment on PATH permanently, run

    $ echo "conda activate" >> ~/.bashrc

Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
your ~/.bashrc file.  You should manually remove the line that looks like

    export PATH="/home/iie4bu/anaconda3/bin:$PATH"

^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^


root@swarm-worker1:~#

这样就进入了ingress_sbox的namespace了。查看ip

root@swarm-worker1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:0a:ff:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.255.0.3/16 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.255.0.168/32 brd 10.255.0.168 scope global eth0
       valid_lft forever preferred_lft forever
15: eth1@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:13:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 172.19.0.2/16 brd 172.19.255.255 scope global eth1
       valid_lft forever preferred_lft forever

发现ip是172.19.0.2。

为了掩饰LVS，咱们退出ingress_sbox的namespace。在swarm-worker1中安装ipvsadm，这是lvs的一个管理工具。

iie4bu@swarm-worker1:~$ sudo apt-get install ipvsadm

安装成功以后，在进入到ingress_sbox中，而后输入命令iptables -nL -t mangle

root@swarm-worker1:~# iptables -nL -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
MARK       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 MARK set 0x100

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
MARK       all  --  0.0.0.0/0            10.255.0.168         MARK set 0x100

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

MARK一行就表示负载均衡。

输入命令ipvsadm -l

root@swarm-worker1:~# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  256 rr
  -> 10.255.0.5:0                 Masq    1      0          0         
  -> 10.255.0.7:0                 Masq    1      0          0

10.255.0.5 10.255.0.7就是whoami的service地址。

在swarm-manager上查看whoami：

iie4bu@swarm-manager:~$ docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS               NAMES
cc9f97cc5056        jwilder/whoami:latest   "/app/http"         19 hours ago        Up 19 hours         8000/tcp            whoami.1.6hhuf528spdw9j9pla7l3tv3t
iie4bu@swarm-manager:~$ docker exec cc9 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.255.0.168/32 brd 10.255.0.168 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.0.2.5/32 brd 10.0.2.5 scope global lo
       valid_lft forever preferred_lft forever
18: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:0a:ff:00:05 brd ff:ff:ff:ff:ff:ff
    inet 10.255.0.5/16 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
20: eth1@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
23: eth2@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:0a:00:02:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.7/24 brd 10.0.2.255 scope global eth2
       valid_lft forever preferred_lft forever
iie4bu@swarm-manager:~$

whoami的IP正是10.255.0.5。

在swarm-worker2上查看：

iie4bu@swarm-worker2:~$ docker container ls
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS               NAMES
f47e05019fd9        jwilder/whoami:latest   "/app/http"              20 hours ago        Up 20 hours         8000/tcp            whoami.3.9idgk9jbrlcm3ufvkmbmvv2t8
633ddfc082b9        busybox:1.28.3          "sh -c 'while true; …"   20 hours ago        Up 20 hours                             client.1.3iv3gworpyr5vdo0h9eortlw0
iie4bu@swarm-worker2:~$ docker exec -it f47 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.255.0.168/32 brd 10.255.0.168 scope global lo
       valid_lft forever preferred_lft forever
    inet 10.0.2.5/32 brd 10.0.2.5 scope global lo
       valid_lft forever preferred_lft forever
24: eth2@if25: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:0a:00:02:0d brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.13/24 brd 10.0.2.255 scope global eth2
       valid_lft forever preferred_lft forever
26: eth1@if27: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.4/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
28: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:0a:ff:00:07 brd ff:ff:ff:ff:ff:ff
    inet 10.255.0.7/16 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever

whoami的IP正是10.255.0.7。

所以当咱们的数据包进入到ingress_sbox经过lvs作了一个负载均衡，也就是说咱们访问8000端口，它会把数据包转发到10.255.0.5和10.255.0.7作一个负载。而后就会进入到swarm节点中了。