Kubeadm部署基于Calico K8s集群

1 环境

Host Name	Role	IP
master1	k8s-1001	172.31.135.239
node1	k8s-1002	172.31.135.238
node2	k8s-1003	172.31.135.237

 2 内核调优

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

3 修改文件描述符

echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf


hard limits自AIX 4.1版本开始引入。hard limits 应由AIX系统管理员设置，只有security组的成员能够将此值增大，用户自己能够减少此限定值,可是其更改将随着该用户从系统退出而失效

soft limits 是AIX核心使用的限制进程对系统资源的使用的上限值。此值可由任何人更改，但不能超出hard limits值。这里要注意的是只有security组的成员可以使更改永久生效普通用户的更改在其退出系统后将失效

1)soft nofile和hard nofile示，单个用用户的软限制为1000，硬限制为1200，即表示单用户能打开的最大文件数量为1000，无论它开启多少个shell。

2）soft nproc和hard nproc 单个用户可用的最大进程数量,软限制和硬限制

3）memlock 一个任务锁住的物理内存的最大值（这里设置成无限制）

4  配置k8s yum 源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

5 配置docker yum源

cd /etc/yum.repos.d
wget https://download.docker.com/linux/centos/docker-ce.repo

6 时间同步 

一个集群内的时间同步必不可少

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service

ntpdate -u ntp.api.bz

7  关闭SELinux、防火墙

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

8 关闭系统的Swap

swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

9 配置hosts 解析

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.135.239	k8s-1001	 k8s-1001
172.31.135.237  k8s-1003     k8s-1003 
172.31.135.238  k8s-1002     k8s-1002

10 配置节点免密登陆

ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub  用户名字@192.168.x.xxx

11 安装依赖等

yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl lrzsz wget

12 配置ipvs模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4




yum install ipset ipvsadm -y

13 安装docker 

yum list docker-ce --showduplicates | sort -r 

yum install -y docker-ce-18.06.1.ce-3.el7

最号先不要安装最新版本
systemctl daemon-reload
systemctl enable docker
systemctl start docker

14 master 和node 节点安装 kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl  master node
systemctl enable kubelet
暂不启动 kubelet

15 若是在国内请按照下面步骤进行

生成默认配置 

kubeadm config print init-defaults > /root/kubeadm.conf

修改 /root/kubeadm.conf，使用国内阿里的imageRepository: registry.aliyuncs.com/google_containers

下载镜像

kubeadm config images pull --config /root/kubeadm.conf

16 如过网络容许能够直接初始化集群

kubeadm init --kubernetes-version=v1.14.1 --pod-network-cidr=10.244.0.0/16


这里没有设置--service-cidr 是由于咱们下面须要部署calico网络，calico会帮助咱们设置service网络，若是此地设置了service网络会致使calico部署不成功

保存这段内容

mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
    --discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1

16 建立相关文件夹

mkdir -p $HOME/.kube
 cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

18 若是之后还有机器加入集群如何获取token 和 hash值？

得到token和hash值
1）获取token
kubeadm token list
默认状况下 Token 过时是时间是24小时，若是 Token 过时之后，能够输入如下命令，生成新的 Token
kubeadm token create

2）获取hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

19 验证

kubectl get pods --all-namespaces

NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-pdf9r            0/1     Pending   0          6m50s
kube-system   coredns-fb8b8dccf-rngcz            0/1     Pending   0          6m50s
kube-system   etcd-k8s-1001                      1/1     Running   0          5m52s
kube-system   kube-apiserver-k8s-1001            1/1     Running   0          6m4s
kube-system   kube-controller-manager-k8s-1001   1/1     Running   0          5m51s
kube-system   kube-proxy-b8dhg                   1/1     Running   0          6m50s
kube-system   kube-scheduler-k8s-1001            1/1     Running   0          5m47s

coredns 是pending状态 先不用管它，由于这个没有网络插件的致使的

20 部署calico网络插件

官方文档

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/


kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml

咱们须要修改calico文件

https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

vim calico.yaml

1）修改ipip模式关闭 和typha_service_name

- name: CALICO_IPV4POOL_IPIP
value: "off"


typha_service_name: "calico-typha"




calico网络，默认是ipip模式（在每台node主机建立一个tunl0网口，这个隧道连接全部的node容器网络，官网推荐不一样的ip网段适合，好比aws的不一样区域主机），

修改为BGP模式，它会以daemonset方式安装在全部node主机，每台主机启动一个bird(BGP client)，它会将calico网络内的全部node分配的ip段告知集群内的主机，并经过本机的网卡eth0或者ens33转发数据；

2）修改replicas

  replicas: 1
  revisionHistoryLimit: 2

3）修改pod的网段CALICO_IPV4POOL_CIDR

- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
4）若是手动下载镜像请查看calico.yaml 文件里面标注的镜像版本 不然能够直接执行会自动下载
5）部署calico
kubectl apply -f calico.yaml

6）查看
kubectl get po --all-namespaces
此时你会发现是pending状态是由于node节点尚未相关组件
7） 验证是否为bgp模式
# ip route show
default via 172.31.143.253 dev eth0 
blackhole 10.244.0.0/24 proto bird 
10.244.0.2 dev caliac6de7553e8 scope link 
10.244.0.3 dev cali1591fcccf0f scope link 
10.244.1.0/24 via 172.31.135.237 dev eth0 proto bird 
10.244.2.0/24 via 172.31.135.238 dev eth0 proto bird 
169.254.0.0/16 dev eth0 scope link metric 1002 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.31.128.0/20 dev eth0 proto kernel scope link src 172.31.135.239

21 将node加入到节点

kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
    --discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1 


master上查看集群
[root@k8s-1001 ~]# kubectl get node
NAME       STATUS   ROLES    AGE    VERSION
k8s-1001   Ready    master   37m    v1.14.1
k8s-1002   Ready    <none>   99s    v1.14.1
k8s-1003   Ready    <none>   115s   v1.14.1
[root@k8s-1001 ~]# kubectl get node -o wide
NAME       STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-1001   Ready    master   37m    v1.14.1   172.31.135.239   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
k8s-1002   Ready    <none>   103s   v1.14.1   172.31.135.238   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
k8s-1003   Ready    <none>   119s   v1.14.1   172.31.135.237   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1

[root@k8s-1001 ~]# kubectl get pod -o wide -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
calico-node-8z92v                  2/2     Running   0          2m1s    172.31.135.238   k8s-1002   <none>           <none>
calico-node-k542k                  2/2     Running   0          7m32s   172.31.135.239   k8s-1001   <none>           <none>
calico-node-n4jgf                  2/2     Running   0          2m17s   172.31.135.237   k8s-1003   <none>           <none>
calico-typha-55968bfd7b-c5r4z      1/1     Running   0          7m33s   172.31.135.237   k8s-1003   <none>           <none>
coredns-fb8b8dccf-pdf9r            1/1     Running   0          37m     10.244.0.3       k8s-1001   <none>           <none>
coredns-fb8b8dccf-rngcz            1/1     Running   0          37m     10.244.0.2       k8s-1001   <none>           <none>
etcd-k8s-1001                      1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
kube-apiserver-k8s-1001            1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
kube-controller-manager-k8s-1001   1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
kube-proxy-b8dhg                   1/1     Running   0          37m     172.31.135.239   k8s-1001   <none>           <none>
kube-proxy-nvlmz                   1/1     Running   0          2m17s   172.31.135.237   k8s-1003   <none>           <none>
kube-proxy-rfb77                   1/1     Running   0          2m1s    172.31.135.238   k8s-1002   <none>           <none>
kube-scheduler-k8s-1001            1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>

22 开启kube-proxy  lvs

修改ConfigMap的kube-system/kube-proxy中的config.conf，`mode: "ipvs"`：

kubectl edit cm kube-proxy -n kube-system
 kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr
  -> 172.31.135.239:6443          Masq    1      0          0         
TCP  10.96.0.10:53 rr
  -> 10.244.0.2:53                Masq    1      0          0         
  -> 10.244.0.3:53                Masq    1      0          0         
TCP  10.96.0.10:9153 rr
  -> 10.244.0.2:9153              Masq    1      0          0         
  -> 10.244.0.3:9153              Masq    1      0          0         
TCP  10.111.3.127:5473 rr
  -> 172.31.135.237:5473          Masq    1      0          0         
UDP  10.96.0.10:53 rr
  -> 10.244.0.2:53                Masq    1      0          0         
  -> 10.244.0.3:53                Masq    1      0          0

23 master默认是污点，pod是不会调度到master

1）开放master可被调度
kubectl taint node k8s-1001 node-role.kubernetes.io/master-
2）
若是要恢复 Master Only 状态
kubectl taint node k8s-1001 node-role.kubernetes.io/master="":NoSchedule

24 建立一个测试pod 验证集群

kubectl run net-test --image=alpine --replicas=2 sleep 360

# kubectl get pod -o wide 
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
net-test-7d6d58cc8-78wd6   1/1     Running   0          6s    10.244.1.2   k8s-1003   <none>           <none>
net-test-7d6d58cc8-hjdhw   1/1     Running   0          6s    10.244.2.2   k8s-1002   <none>           <none>