kubernetes 1.13.3版本升级至1.14.1版本

时间 2019-11-07

标签 kubernetes 1.13.3 版本升级 1.14.1 繁體版

原文原文链接

1、说明

本文章介绍如何将使用kubeadm建立的Kubernetes集群从版本1.13.x升级到版本1.14.x。bootstrap

只能从一个MINOR版本升级到下一个MINOR版本，或者在同一个MINOR的PATCH版本之间升级。也就是说，升级时不能跳过MINOR版本。例如，能够从1.y升级到1.y + 1，但不能从1.y升级到1.y + 2。api

升级工做流程以下：服务器

升级主master节点。
升级其余master节点。
升级work节点。

当前kubernetes版本信息：网络

[root@node-01 ~]# kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
node-01   Ready    master   99d   v1.13.3
node-02   Ready    master   99d   v1.13.3
node-03   Ready    master   99d   v1.13.3
node-04   Ready    <none>   99d   v1.13.3
node-05   Ready    <none>   99d   v1.13.3
node-06   Ready    <none>   99d   v1.13.3

2、升级主master节

1. 在第一个master节点上，升级kubeadm

yum install kubeadm-1.14.1  -y

2. 验证下载是否有效并具备预期版本：

kubeadm version

3. 运行kubeadm upgrade plan

此命令检查您的群集是否能够升级，并获取能够升级到的版本
,您应该看到与此相似的输出：app

[root@node-01 ~]#  kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.13.3
[upgrade/versions] kubeadm version: v1.14.1
I0505 13:55:58.449783   12871 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0505 13:55:58.449867   12871 version.go:97] falling back to the local client version: v1.14.1
[upgrade/versions] Latest stable version: v1.14.1
I0505 13:56:08.645796   12871 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.13.txt": Get https://dl.k8s.io/release/stable-1.13.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0505 13:56:08.645861   12871 version.go:97] falling back to the local client version: v1.14.1
[upgrade/versions] Latest version in the v1.13 series: v1.14.1

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     6 x v1.13.3   v1.14.1

Upgrade to the latest version in the v1.13 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.13.3   v1.14.1
Controller Manager   v1.13.3   v1.14.1
Scheduler            v1.13.3   v1.14.1
Kube Proxy           v1.13.3   v1.14.1
CoreDNS              1.2.6     1.3.1
Etcd                 3.2.24    3.3.10

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.14.1

_____________________________________________________________________

4. 运行升级命令

kubeadm upgrade apply v1.14.1

你应该看到与此相似的输出ide

[root@node-01 ~]# kubeadm upgrade apply v1.14.1
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/version] You have chosen to change the cluster version to "v1.14.1"
[upgrade/versions] Cluster version: v1.13.3
[upgrade/versions] kubeadm version: v1.14.1
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.14.1"...
Static pod: kube-apiserver-node-01 hash: 26d86add2bfd0fd6825f5507fff1fb5e
Static pod: kube-controller-manager-node-01 hash: 21ea3d3ccb8d8dc00056209ca3da698b
Static pod: kube-scheduler-node-01 hash: a8d928943d47ec793a700ef95c4b6b4a
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-node-01 hash: 35015766b0b1714f398bed77aae5be95
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-node-01 hash: 35015766b0b1714f398bed77aae5be95
Static pod: etcd-node-01 hash: 35015766b0b1714f398bed77aae5be95
Static pod: etcd-node-01 hash: 17ddbcfb2ddf1d447ceec2b52c9faa96
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests940835611"
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-node-01 hash: 26d86add2bfd0fd6825f5507fff1fb5e
Static pod: kube-apiserver-node-01 hash: ff2267bcddb83b815efb49ff766ad897
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-node-01 hash: 21ea3d3ccb8d8dc00056209ca3da698b
Static pod: kube-controller-manager-node-01 hash: ff8be061048a4660a1fbbf72db229d0d
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-00-02/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-node-01 hash: a8d928943d47ec793a700ef95c4b6b4a
Static pod: kube-scheduler-node-01 hash: 959a5cdf1468825401daa8d35329351e
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.14.1". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

5. 手动升级您的CNI提供程序插件。

你的容器网络接口（CNI）提供商可能有本身的升级说明。检查插件页面以查找你的CNI提供商，并查看是否须要其余升级步骤。post

6. 升级第一个master节点上的kubelet和kubectl

yum install kubectl-1.14.1 kebulet-1.14.1 -y

重启kubeletfetch

systemctl daemon-reload
systemctl restart kubelet

3、升级其余master节点

1. 升级kubeadm程序

yum install kubeadm-1.14.1 -y

2. 升级静态pod

kubeadm upgrade node experimental-control-plane

你能够看到相似信息：this

[root@node-02 ~]# kubeadm upgrade node experimental-control-plane
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.14.1"...
Static pod: kube-apiserver-node-02 hash: 26d86add2bfd0fd6825f5507fff1fb5e
Static pod: kube-controller-manager-node-02 hash: 21ea3d3ccb8d8dc00056209ca3da698b
Static pod: kube-scheduler-node-02 hash: a8d928943d47ec793a700ef95c4b6b4a
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-node-02 hash: 1aa55e50528bd0621c5734515b3265fd
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-node-02 hash: 1aa55e50528bd0621c5734515b3265fd
Static pod: etcd-node-02 hash: 1aa55e50528bd0621c5734515b3265fd
Static pod: etcd-node-02 hash: 4710a34897e7838519a1bf8fe4dccf07
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests483113569"
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-node-02 hash: 26d86add2bfd0fd6825f5507fff1fb5e
Static pod: kube-apiserver-node-02 hash: fe1005f40c3f390280358921c3073223
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-node-02 hash: 21ea3d3ccb8d8dc00056209ca3da698b
Static pod: kube-controller-manager-node-02 hash: ff8be061048a4660a1fbbf72db229d0d
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-05-05-14-10-46/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-node-02 hash: a8d928943d47ec793a700ef95c4b6b4a
Static pod: kube-scheduler-node-02 hash: 959a5cdf1468825401daa8d35329351e
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade] The control plane instance for this node was successfully updated!

3. 更新kubelet和kubectl

yum install kubectl-1.14.1 kebulet-1.14.1 -y

4. 重启kubelet

systemctl daemon-reload
systemctl restart kubelet

4、升级work节点

1. 在work节点上升级kubeadm：

yum install -y kubeadm-1.14.1

2.调整调度策略

经过将节点标记为不可调度并逐出pod来准备节点以进行维护(此步骤在master上执行)。

kubectl drain $NODE --ignore-daemonsets

[root@node-01 ~]# kubectl drain node-04 --ignore-daemonsets
node/node-04 already cordoned
WARNING: ignoring DaemonSet-managed Pods: cattle-system/cattle-node-agent-h555m, default/glusterfs-vhdqv, kube-system/canal-mbwvf, kube-system/kube-flannel-ds-amd64-zdfn8, kube-system/kube-proxy-5d64l
evicting pod "coredns-55696d4b79-kfcrh"
evicting pod "cattle-cluster-agent-66bd75c65f-k7p6n"
pod/cattle-cluster-agent-66bd75c65f-k7p6n evicted
pod/coredns-55696d4b79-kfcrh evicted
node/node-04 evicted

3. 更新node

[root@node-04 ~]# kubeadm upgrade node config --kubelet-version v1.14.1
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

4. 更新kubelet和kubectl

yum install kubectl-1.14.1 kebulet-1.14.1 -y

5.重启kubelet

systemctl daemon-reload
systemctl restart kubelet

6. 恢复调度策略（master执行）

kubectl uncordon $NODE

其余work节点升级根据以上的步骤执行一遍便可。

4、验证群集的状态

在全部节点上升级kubelet以后，经过从kubectl能够访问群集的任何位置运行如下命令来验证全部节点是否可用：

[root@node-01 ~]# kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
node-01   Ready    master   99d   v1.14.1
node-02   Ready    master   99d   v1.14.1
node-03   Ready    master   99d   v1.14.1
node-04   Ready    <none>   99d   v1.14.1
node-05   Ready    <none>   99d   v1.14.1
node-06   Ready    <none>   99d   v1.14.1

该STATUS列全部节点应显示Ready，而且应更新版本号。

5、工做原理

kubeadm upgrade apply 执行如下操做：

检查您的群集是否处于可升级状态：
API服务器是可访问的
全部节点都处于该Ready状态
master是否健康
执行版本控制策略。
确保master镜像可用或可用于拉到机器。
升级master组件或回滚（若是其中任何一个组件没法启动）。
应用新的kube-dns和kube-proxy清单，并确保建立全部必需的RBAC规则。
建立API服务器的新证书和密钥文件，若是旧文件即将在180天后过时，则备份旧文件。

kubeadm upgrade node experimental-control-plane在其余控制平面节点上执行如下操做：

ClusterConfiguration从群集中获取kubeadm 。
可选择备份kube-apiserver证书。
升级master组件的静态Pod清单。

本次的分享就到此，若有问题欢迎在下面留言交流，但愿你们多多关注和点赞，谢谢！