Ubuntu 18 Kubernetes集群的安装和部署以及Helm的安装

时间 2020-05-12

标签 ubuntu kubernetes 集群安装部署以及 helm 栏目 Ubuntu 繁體版

原文原文链接

首先说一下个人环境，我是在windows 10 上面建了一个ubuntu18的虚拟机，同时因为某些缘由不受网络限制，因此安装比较顺利。html

Install

1.安装并启用 Docker node

sudo apt install docker.io
sudo systemctl enable docker
docker --versionlinux

2.添加 Kubernetes signing key 和Repositorynginx

sudo apt install curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"git

3.安装Kubeadmgithub

sudo apt install kubeadm kubeadm version
 #经常使用命令 重启kubelet服务： systemctl daemon-reload systemctl restart kubelet sudo systemctl restart kubelet.service sudo systemctl daemon-reload sudo systemctl stop kubelet sudo systemctl enable kubelet sudo systemctl start kubelet

4.禁用 swapoffdocker

sudo swapoff -ajson

sudo sed -i '/ swap / s/^/#/' /etc/fstabubuntu

以上的指令我只在一台Ubuntu上执行的（若是你有多台计算机，须要在全部的计算机上执行以上指令，我这里是经过拷贝虚拟机来实现的）windows

5.准备2台虚拟机k8s-master和k8s-node(我这里把上面的计算机命名为 k8s_master ，copy它并命名为k8s_node)

sudo hostnamectl set-hostname k8s-master #在k8s-master 上执行 IP：192.168.255.229
sudo hostnamectl set-hostname k8s-node #k8s-node 上执行 IP：192.168.255.230

Deploy

1.在master上初始化 Kubernetes

使用kubeadm config print init-defaults能够打印集群初始化默认的使用的配置,使用kubeadm默认配置初始化的集群，会在master节点打上node-role.kubernetes.io/master:NoSchedule的污点，阻止master节点接受调度运行工做负载。这里测试环境只有两个节点，因此将这个taint的effect从NoSchedule改成PreferNoSchedule 还有就是修订kubernet版本1.15.2

apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.100.7 bindPort: 6443 nodeRegistration: taints: - effect: PreferNoSchedule key: node-role.kubernetes.io/master --- apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.15.2 networking: podSubnet: 10.244.0.0/16

#sudo kubeadm init --pod-network-cidr=192.168.255.229/2 kubeadm init --config kubeadm.yaml #--ignore-preflight-errors=Swap

#sudo kubeadm init --pod-network-cidr=192.168.100.0/2 这里不该该用现有计算机的ip 这里有解决方案 Kubernetes-dashboard pod is crashing again and again

若是执行中遇到detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd"

请参考Container runtimes执行

# Setup daemon. cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "insecure-registries":["192.168.100.20:80"] } EOF mkdir -p /etc/systemd/system/docker.service.d #我顺便吧docker的私有仓库也加在里面 # Restart docker. systemctl daemon-reload systemctl restart docker

若是遇到port 10251 and 10252 are in use 错误请执行 netstat -lnp | grep 1025 而后kill 进程ID

2在master节点上执行以下：

mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config #检查 master kubectl get nodes

3.Deploy a Pod Network & view the status

sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl get pods --all-namespaces
sudo kubectl get nodes

4.add slave node

在k8s-node上执行

kubeadm join 192.168.254.229:6443 --token ewlb93.v0ohocpvncaxgl16 --discovery-token-ca-cert-hash sha256:2522834081168fbe4b5b05854b964e76f1ea8bac6f8fc5e2be21c93c6a27c427

在k8s-master上检查节点信息：

5.安装 Dashboard 插件 &检查状态

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml 
kubectl get deployment kubernetes-dashboard -n kube-system kubectl get svc kubernetes-dashboard -n kube-system
#kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

为了简单，我这里直接为 Dashboard 赋予 Admin 的权限(否者会禁止访问)，参考https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard labels: k8s-app: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system

启动代理

kubectl proxy #或者 若是须要暴露给其余机器，须要指定address: kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='^*$'

访问http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

不过我在访问这个地址遇到错误以下：

root@k8s-master:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-5c98db65d4-97xrf 1/1 Running 0 83m kube-system coredns-5c98db65d4-vvtfc 1/1 Running 0 83m kube-system etcd-k8s-master 1/1 Running 0 83m kube-system kube-apiserver-k8s-master 1/1 Running 0 83m kube-system kube-controller-manager-k8s-master 1/1 Running 0 83m kube-system kube-flannel-ds-amd64-gbg49 1/1 Running 1 80m kube-system kube-flannel-ds-amd64-hmrcp 1/1 Running 0 82m kube-system kube-proxy-lbp5k 1/1 Running 0 80m kube-system kube-proxy-szkb8 1/1 Running 0 83m kube-system kube-scheduler-k8s-master 1/1 Running 0 83m kube-system kubernetes-dashboard-7d75c474bb-p5nz5 0/1 CrashLoopBackOff 6 10m root@k8s-master:~# kubectl logs kubernetes-dashboard-7d75c474bb-p5nz5 -n kube-system 2019/08/07 11:34:33 Using in-cluster config to connect to apiserver 2019/08/07 11:34:33 Using service account token for csrf signing 2019/08/07 11:34:33 Starting overwatch 2019/08/07 11:35:03 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
root@k8s-master:~#

后来在https://stackoverflow.com/questions/52273029/kubernetes-dashboard-pod-is-crashing-again-and-again 找到解决方案，

Make sure you understand the difference between Host network, Pod network and Service network. These 3 networks can not overlap. For example --pod-network-cidr=192.168.0.0/16 must notinclude the IP address of your host, change it to 10.0.0.0/16 or something smaller if necessary.

因此初始化该为 sudo kubeadm init --pod-network-cidr=192.168.100.0/2 就okay 了

建立帐号

kubectl create serviceaccount dashboard -n default kubectl create clusterrolebinding dashboard-admin -n default --clusterrole=cluster-admin --serviceaccount=default:dashboard kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode

Copy the decoded password and login to dashboard.

在Kubernetes的安装过程当中会默认安装不少Service帐户，分别有不一样的访问权限，要找到对应的token，你可使用下面的方式：

kubectl -n kube-system get secret
kubectl -n kube-system describe secret certificate-controller-token-4xr9x

上面咱们使用token来登陆，这里咱们仍是用上面的dashboard帐号来生成kubeconfig文件，之后用生成kubeconfig文件来登陆

#配置集群信息 kubectl config set-cluster kubernetes --server=192.168.100.11:6443 --kubeconfig=/root/dashbord.conf #使用token写入集群验证 #kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode #查看token kubectl config set-credentials dashboard --token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRhc2hib2FyZC10b2tlbi12dDl4OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxZmM1ZDc1ZS0xMjE2LTQwMDgtYThhOS03ZjEwZGQ1NWJjNWEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkYXNoYm9hcmQifQ.Neb5_blRSig6IU5oPtRIQlQhcELaeI8uu7jeiVEdiR3CLiCZYyiI7X6uNsrpGKAkR-OkGM1gOp09-pmxFI6m4lKHYu9S7R1MNigmQrxfZB4RJ-iYZCNp3Rra7mFrluwY_yMbzuZ__XeYShSOiO1VAS2ezWFGk9adgtbiWZkef_NxmYdwEmTAGkmazhatK9SGDWBea-1seoJx-SGFyA9j0gNcWqNrX93ozFmuNtYrPZSwhYkul-q4NHOz4Dp4Ux1C7gZzTIgBySaYZd5tiJIAmZ-6CV-ukmPtFn7tVlNaDkK4K5N6jzyDttlvHZJtWqBR7iWTyamAKAbycm_BmaQR4Q --kubeconfig=/root/dashboard.conf 配置上下文和当前上下文 kubectl config set-context dashboard@kubernetes --cluster=kubernetes --user=dashboard --kubeconfig=/root/dashboard.conf kubectl config use-context dashboard@kubernetes --kubeconfig=/root/dashboard.conf kubectl config view --kubeconfig=/root/dashboard.conf

将/root/dashboard.conf 文件发送到宿主机，浏览器访问时选择Kubeconfig认证，载入该配置文件，点击登录，便可实现访问，如图：

卸载集群

想要撤销kubeadm作的事，首先要排除节点，并确保在关闭节点以前要清空节点。

在主节点上运行：

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
而后在须要移除的节点上，重置kubeadm的安装状态：

kubeadm reset

重启master节点若是遇到以下错误

The connection to the server 10.2.0.165:6443 was refused - did you specify the right host or port? 解决方案The connection to the server <host>:6443 was refused - did you specify the right host or port?

sudo -i swapoff -a exit strace -eopenat kubectl version

#若是以上方法没有搞定能够执行如下命令
systemctl status docker #查看docker状态
systemctl status kubelet #查看kubelet的状态
netstat -pnlt | grep 6443 #查看6443的端口应该启动起来了
journalctl -xeu kubelet #查看kubelet的日志解决问题

systemctl restart kubelet #最后重启

-------------------------2019-08-20--------------------------

Kubernetes经常使用组件部署

Helm的安装

Helm由客户端命helm令行工具和服务端tiller组成，Helm的安装十分简单。下载helm命令行工具到master节点node1的/usr/local/bin下，这里下载的2.14.1版本：

curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz
tar -zxvf helm-v2.14.1-linux-amd64.tar.gz cd linux-amd64/ cp helm /usr/local/bin/

为了安装服务端tiller，还须要在这台机器上配置好kubectl工具和kubeconfig文件，确保kubectl工具能够在这台机器上访问apiserver且正常使用。这里的node1节点已经配置好了kubectl。

由于Kubernetes APIServer开启了RBAC访问控制，因此须要建立tiller使用的service account: tiller并分配合适的角色给它。详细内容能够查看helm文档中的Role-based Access Control。这里简单起见直接分配cluster-admin这个集群内置的ClusterRole给它。建立helm-rbac.yaml文件

apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system

接下来使用helm部署tiller，tiller默认被部署在k8s集群中的kube-system这个namespace下

kubectl create -f helm-rbac.yaml helm init --service-account tiller --skip-refresh kubectl get pod -n kube-system -l app=helm helm version

最后在k8s-master上修改helm chart仓库的地址为azure提供的镜像地址：

helm repo add stable http://mirror.azure.cn/kubernetes/charts

使用Helm部署Nginx Ingress

为了便于将集群中的服务暴露到集群外部，须要使用Ingress。接下来使用Helm将Nginx Ingress部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的边缘节点上，关于Kubernetes边缘节点的高可用相关的内容能够查看以前整理的Bare metal环境下Kubernetes Ingress边缘节点的高可用,Ingress Controller使用hostNetwork。

咱们将node1(192.168.100.7)作为边缘节点，打上Label：

root@k8s-master:~/linux-amd64# kubectl label node k8s-master node-role.kubernetes.io/edge= node/k8s-master labeled root@k8s-master:~/linux-amd64# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready edge,master 34m v1.15.2 k8s-node Ready <none> 28m v1.15.2 root@k8s-master:~/linux-amd64#

stable/nginx-ingress chart的值文件ingress-nginx.yaml以下：

controller: replicaCount: 1 hostNetwork: true nodeSelector: node-role.kubernetes.io/edge: '' affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-ingress - key: component operator: In values: - controller topologyKey: kubernetes.io/hostname tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule defaultBackend: nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule

nginx ingress controller的副本数replicaCount为1，将被调度到k8s-master这个边缘节点上。这里并无指定nginx ingress controller service的externalIPs，而是经过hostNetwork: true设置nginx ingress controller使用宿主机网络。

helm repo update helm install stable/nginx-ingress -n nginx-ingress --namespace ingress-nginx -f ingress-nginx.yaml helm repo add stable http://mirror.azure.cn/kubernetes/charts

若是访问http://192.168.100.7返回default backend，则部署完成。

kubectl -n ingress-nginx exec nginx-ingress-controller-xxxxx -- cat /etc/nginx/nginx.confkubectl get pod -n ingress-nginx -o wide

使用Helm部署dashboard

kubernetes-dashboard.yaml

image: repository: k8s.gcr.io/kubernetes-dashboard-amd64 tag: v1.10.1 ingress: enabled: true hosts: - k8s.frognew.com annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" tls: - secretName: frognew-com-tls-secret hosts: - k8s.frognew.com nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule rbac: clusterAdminRole: true

安装：

helm install stable/kubernetes-dashboard -n kubernetes-dashboard --namespace kube-system -f kubernetes-dashboard.yaml
kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubectl describe -n kube-system secret/kubernetes-dashboard-token-xxx
helm search kubernetes-dashboard
kubectl get pods -n kube-system -o wide

修改本机的hosts文件：

192.168.100.11 k8s.frognew.com