使用Kubeadm + HAProxy + Keepalived部署高可用Kubernetes集群
#使用Kubeadm + HAProxy + Keepalived部署高可用Kubernetes集群
这两天kubernetes爆出第一个特权升级高危漏洞，波及非常广泛，且没有有效的补丁可以修改此漏洞，只能将kubernetes升级。说巧不巧，与此同时，kubernetes v1.13.0版本发布，kubeadm也升级为GA版本，简单阅读了一下更新日志，先部署一套练练手～
1、环境说明
本次高可用集群基本参照官网步骤进行部署，官网给出了两种拓扑结构：堆叠control plane node和external etcd node，本文基于第一种拓扑结构进行部署，使用Keepalived + HAProxy搭建高可用Load balancer，完整的拓扑图如下：
单个mastre节点将部署keepalived、haproxy、etcd、apiserver、controller-manager、schedule六种服务，load balancer集群和etcd集群仅用来为kubernetes集群集群服务，不对外营业。如果必要，可以将load balancer或者etcd单独部署，为kubernetes集群提供服务的同时，也可以为其他有需要的系统提供服务，比如下面这样的拓扑结构：
这种拓扑结构也对应external etcd node～
本文仅部署master节点，使用kubeadm部署worker节点非常简单，不在赘述，环境清单：
[[email protected] kubernetes]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0 Ready master 4h2m v1.13.0 172.16.7.11 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0 master-1 Ready master 3h53m v1.13.0 172.16.7.12 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0 master-2 Ready master 3h52m v1.13.0 172.16.7.13 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
镜像清单：
[[email protected] kubernetes]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.13.0 8fa56d18961f 34 hours ago 80.2MB k8s.gcr.io/kube-scheduler v1.13.0 9508b7d8008d 34 hours ago 79.6MB k8s.gcr.io/kube-apiserver v1.13.0 f1ff9b7e3d6e 34 hours ago 181MB k8s.gcr.io/kube-controller-manager v1.13.0 d82530ead066 34 hours ago 146MB quay.io/calico/node v3.3.1 427a0694c75c 3 weeks ago 75.3MB quay.io/calico/cni v3.3.1 fa6f35a1c14d 3 weeks ago 75.4MB k8s.gcr.io/coredns 1.2.6 f59dcacceff4 4 weeks ago 40MB k8s.gcr.io/etcd 3.2.24 3cab8e1b9802 2 months ago 220MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 11 months ago 742kB
主要软件清单：
- keepalived-1.3.5-6.el7.x86_64.rpm
- haproxy-1.5.18-8.el7.x86_64.rpm
- docker-ce-18.09.0-3.el7.x86_64.rpm
- kubeadm-1.13.0-0.x86_64.rpm
- kubectl-1.13.0-0.x86_64.rpm
- kubelet-1.13.0-0.x86_64.rpm
网盘下载地址：https://pan.baidu.com/s/1PwS5kiUd9G-oY2i9TgJVyA（包括rpm依赖）
2、部署步骤
2.1、部署keepalived
此处的keeplived的主要作用是为haproxy提供vip（172.16.7.10），在三个haproxy实例之间提供主备，降低当其中一个haproxy失效的时对服务的影响。
- 系统配置
[[email protected] ~]# cat >> /etc/sysctl.conf << EOF net.ipv4.ip_forward = 1 EOF [[email protected] ~]# sysctl -p net.ipv4.ip_forward = 1
In order for the Keepalived service to forward network packets properly to the real servers, each router node must have IP forwarding turned on in the kernel.
- 安装keepalived
[[email protected] ~]# yum install -y keepalived
- 配置keepalived
[[email protected] ~]# cat > /etc/keepalived/keepalived.conf << EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script check_haproxy { script "killall -0 haproxy" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 250 advert_int 1 authentication { auth_type PASS auth_pass 35f18af7190d51c9f7f78f37300a0cbd } virtual_ipaddress { 172.16.7.10 } track_script { check_haproxy } } EOF
killall -0根据进程名称检测进程是否存活- master-0节点为***MASTER***，其余节点为***BACKUP***
- priority各个几点到优先级相差50，范围：0～250（非强制要求）
- 启动并检测服务
[[email protected] ~]# systemctl enable keepalived.service [[email protected] ~]# systemctl start keepalived.service [[email protected] ~]# systemctl status keepalived.service [[email protected] ~]# ip address show ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:50:56:25:99:6e brd ff:ff:ff:ff:ff:ff inet 172.16.7.11/24 brd 172.16.7.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 172.16.7.10/32 scope global ens33 valid_lft forever preferred_lft forever
2.2、部署haproxy
此处的haproxy为apiserver提供反向代理，haproxy将所有请求轮询转发到每个master节点上。相对于仅仅使用keepalived主备模式仅单个master节点承载流量，这种方式更加合理、健壮。
- 系统配置
[[email protected] ~]# cat >> /etc/sysctl.conf << EOF net.ipv4.ip_nonlocal_bind = 1 EOF [[email protected] ~]# sysctl -p net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1
Load balancing in HAProxy and Keepalived at the same time also requires the ability to bind to an IP address that are nonlocal, meaning that it is not assigned to a device on the local system. This allows a running load balancer instance to bind to an IP that is not local for failover.
- 安装haproxy
[[email protected] ~]# yum install -y haproxy
- 配置haproxy
[[email protected] ~]# cat > /etc/haproxy/haproxy.cfg << EOF #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # kubernetes apiserver frontend which proxys to the backends #--------------------------------------------------------------------- frontend kubernetes-apiserver mode tcp bind *:16443 option tcplog default_backend kubernetes-apiserver #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend kubernetes-apiserver mode tcp balance roundrobin server master-0 172.16.7.11:6443 check server master-1 172.16.7.12:6443 check server master-2 172.16.7.13:6443 check #--------------------------------------------------------------------- # collection haproxy statistics message #--------------------------------------------------------------------- listen stats bind *:1080 stats auth admin:awesomePassword stats refresh 5s stats realm HAProxy\ Statistics stats uri /admin?stats EOF
所以master节点上的配置完全相同～
- 启动并检测服务
[[email protected] ~]# systemctl enable haproxy.service [[email protected] ~]# systemctl start haproxy.service [[email protected] ~]# systemctl status haproxy.service [[email protected] ~]# ss -lnt | grep -E "16443|1080" LISTEN 0 128 *:1080 *:* LISTEN 0 128 *:16443 *:*
2.3、安装kubeadm、kubectl、kubelet、docker
- 系统配置
[[email protected] ~]# systemctl disable firewalld.service [[email protected] ~]# systemctl stop firewalld.service [[email protected] ~]# systemctl status firewalld.service [[email protected] ~]# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config [[email protected] ~]# setenforce 0 [[email protected] ~]# sed -i 's/\(.*swap.*\)/# \1/g' /etc/fstab [[email protected] ~]# swapoff -a [[email protected] ~]# cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF [[email protected] ~]# sysctl --system
- 安装软件
### 设置docker-ce的yum源 ### [[email protected] ~]# cat <<EOF > /etc/yum.repos.d/docker-ce.repo [docker-ce-stable] name=Docker CE Stable - $basearch baseurl=https://download.docker.com/linux/centos/7/$basearch/stable enabled=1 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg EOF ### 安装docker-ce ### [[email protected] ~]# yum install -y docker-ce --disableexcludes=docker-ce-stable ### 设置kubernetes的yum源 ### [[email protected] ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kube* EOF ### 安装 ### [[email protected] ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
- 设置开机启动
[[email protected] ~]# systemctl enable docker.service [[email protected] ~]# systemctl start docker.service [[email protected] ~]# systemctl status docker.service [[email protected] ~]# systemctl enable kubelet.service
2.4、部署master-0
- 编辑hosts文件
[[email protected] ~]# cat >> /etc/hosts << EOF 172.16.7.10 cluster.kube.com 172.16.7.11 master-0 172.16.7.12 master-1 172.16.7.13 master-2 EOF
- 编辑kubeadm配置文件
[[email protected] ~]# cat > kubeadm-config.yaml << EOF apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: v1.13.0 apiServer: certSANs: - "cluster.kube.com" controlPlaneEndpoint: "cluster.kube.com:16443" networking: podSubnet: "192.168.0.0/16" EOF
CNI使用Calico，设置podSubnet: “192.168.0.0/16”
- 初始化第一个master节点
[[email protected] ~]# kubeadm init --config kubeadm-config.yaml [init] Using Kubernetes version: v1.13.0 [preflight] Running pre-flight checks [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06 [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master-0 localhost] and IPs [172.16.7.11 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master-0 localhost] and IPs [172.16.7.11 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "ca" certificate and key [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local cluster.kube.com cluster.kube.com] and IPs [10.96.0.1 172.16.7.11] [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 19.503612 seconds [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-0" as an annotation [mark-control-plane] Marking the node master-0 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master-0 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: 29kttk.v6a5ts4021pkc4zr [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join cluster.kube.com:16443 --token 29kttk.v6a5ts4021pkc4zr --discovery-token-ca-cert-hash sha256:d0bdfd403cd9822177f66c6f3fa28735a67c0747f52511f253ba82843b70f99a [[email protected] ~]# mkdir -p $HOME/.kube [[email protected] ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [[email protected] ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 安装网络插件
[[email protected] ~]# kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created [[email protected] ~]# kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml configmap/calico-config created service/calico-typha created deployment.apps/calico-typha created poddisruptionbudget.policy/calico-typha created daemonset.extensions/calico-node created serviceaccount/calico-node created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
- 复制相关文件到其他master节点上
[[email protected] ~]# ssh [email protected] mkdir -p /etc/kubernetes/pki/etcd [[email protected] ~]# scp /etc/kubernetes/admin.conf [email protected]:/etc/kubernetes [[email protected] ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} [email protected]:/etc/kubernetes/pki [[email protected] ~]# scp /etc/kubernetes/pki/etcd/ca.* [email protected]:/etc/kubernetes/pki/etcd
2.5、部署master-other
[[email protected] ~]# kubeadm join cluster.kube.com:16443 --token 29kttk.v6a5ts4021pkc4zr --discovery-token-ca-cert-hash sha256:d0bdfd403cd9822177f66c6f3fa28735a67c0747f52511f253ba82843b70f99a --experimental-control-plane [preflight] Running pre-flight checks [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06 [discovery] Trying to connect to API Server "cluster.kube.com:16443" [discovery] Created cluster-info discovery client, requesting info from "https://cluster.kube.com:16443" [discovery] Requesting info from "https://cluster.kube.com:16443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "cluster.kube.com:16443" [discovery] Successfully established connection with API Server "cluster.kube.com:16443" [join] Reading configuration from the cluster... [join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [join] Running pre-flight checks before initializing the new control plane instance [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06 [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local cluster.kube.com cluster.kube.com] and IPs [10.96.0.1 172.16.7.12] [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master-1 localhost] and IPs [172.16.7.12 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master-1 localhost] and IPs [172.16.7.12 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] valid certificates and keys now exist in "/etc/kubernetes/pki" [certs] Using the existing "sa" key [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Using existing up-to-date kubeconfig file: "/etc/kubernetes/admin.conf" [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Checking Etcd cluster health [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-1" as an annotation [etcd] Announced new etcd member joining to the existing etcd cluster [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [mark-control-plane] Marking the node master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Master label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster.
3、检查测试
- 查看kubernetes集群状态
[[email protected] etcd]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0 Ready master 11m v1.13.0 172.16.7.11 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0 master-1 Ready master 2m16s v1.13.0 172.16.7.12 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0 master-2 Ready master 45s v1.13.0 172.16.7.13 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0 [[email protected] etcd]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health": "true"} [[email protected] etcd]# kubectl get csr NAME AGE REQUESTOR CONDITION csr-6jgcr 11m system:node:master-0 Approved,Issued node-csr-8wOc8SoFxlkm8Ep5QEr9erUexnYJ-lrilft4QUJ7RfI 2m26s system:bootstrap:29kttk Approved,Issued node-csr-l-ijTesiUBRFmaAXIHYj3J4vYEA7qbJLolIsKv5FZuM 55s system:bootstrap:29kttk Approved,Issued
- 查看etcd集群状态
[[email protected] ~]# kubectl exec -ti -n kube-system etcd-master-0 sh / # export ETCDCTL_API=3 / # etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/ku bernetes/pki/etcd/healthcheck-client.key member list 25e9e768b2abcbb6, started, master-2, https://172.16.7.13:2380, https://172.16.7.13:2379 28e1924984642da1, started, master-1, https://172.16.7.12:2380, https://172.16.7.12:2379 e19f37b34b9497ee, started, master-0, https://172.16.7.11:2380, https://172.16.7.11:2379