K8s手动方式搭建平台及问题汇总

时间 2021-03-11

标签 node linux nginx git github golang docker json bootstrap vim 栏目 Linux 繁體版

原文原文链接

如今换了份工做，新环境使用k8s容器环境管理，心思着把之前的相关文档汇总起来编写成博文方便本身复习node

k8s概要

因为图片有上一家信息在上面，相关图片就不放上面了。linux

kubernetes，简称K8s，是用8代替8个字符“ubernete”而成的缩写。是一个开源的，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单而且高效（powerful）,Kubernetes提供了应用部署，规划，更新，维护的一种机制nginx

Kubernetes是Google开源的一个容器编排引擎，它支持自动化部署、大规模可伸缩、应用容器化管理。在生产环境中部署一个应用程序时，一般要部署该应用的多个实例以便对应用请求进行负载均衡。git

在Kubernetes中，咱们能够建立多个容器，每一个容器里面运行一个应用实例，而后经过内置的负载均衡策略，实现对这一组应用实例的管理、发现、访问，而这些细节都不须要运维人员去进行复杂的手工配置和处理。github

kubernetes的特色：

可移植: 支持公有云，私有云，混合云，多重云（multi-cloud）
可扩展: 模块化，插件化，可挂载，可组合
自动化: 自动部署，自动重启，自动复制，自动伸缩/扩展golang

kubernetes的体系架构

Kubernetes集群包含有节点代理kubelet和Master组件(APIs, scheduler, etc)，一切都基于分布式的存储系统。
docker

kubernetes 组件特色

咱们把服务分为运行在工做节点上的服务和组成集群级别控制板的服务。json

Kubernetes节点有运行应用容器必备的服务，而这些都是受Master的控制。bootstrap

每次个节点上固然都要运行Docker。Docker来负责全部具体的映像下载和容器运行。vim

Kubernetes主要由如下几个核心组件组成：

etcd保存了整个集群的状态；
apiserver提供了资源操做的惟一入口，并提供认证、受权、访问控制、API注册和发现等机制；
controller manager负责维护集群的状态，好比故障检测、自动扩展、滚动更新等；
scheduler负责资源的调度，按照预约的调度策略将Pod调度到相应的机器上；
kubelet负责维护容器的生命周期，同时也负责Volume（CVI）和网络（CNI）的管理；
Container runtime负责镜像管理以及Pod和容器的真正运行（CRI）；
kube-proxy负责为Service提供cluster内部的服务发现和负载均衡；
除了核心组件，还有一些推荐的Add-ons：

kube-dns负责为整个集群提供DNS服务
Ingress Controller为服务提供外网入口
Heapster提供资源监控
Dashboard提供GUI
Federation提供跨可用区的集群
Fluentd-elasticsearch提供集群日志采集、存储与查询
这些官网或者中文网站均可以查阅到资料，后面主要介绍我本身手动部署安装k8s

手动部署安装

etcd:3.3.11
kubectl:
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
kubernetes-dashboard:v1.6.3
nginx-ingress-controller:0.9.0
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
calico:v3.2.6

初始化环境

配置hosts地址

这里我只在test环境进行测试，生产环境同样配置只是环境地址不一样。
在每台服务器上执行
#编辑每台服务器的 /etc/hosts 文件，配置hostname 通讯

vi /etc/hosts
172.16.16.86 incubator-dc-016
172.16.16.246 incubator-dc-002
172.16.16.250 incubator-dc-003

关闭防火墙

在每台服务器上执行

systemctl stop firewalld.service               
#中止firewall
systemctl disable firewalld.service            
#禁止firewall开机启动
firewall-cmd --state                          
#查看默认防火墙状态（关闭后显示notrunning，开启后显示running）

关闭selinux

在每台服务器上执行

$ setenforce 0
$ vim /etc/selinux/config
SELINUX=disabled

关闭swap

在每台服务器上执行
K8s需使用内存，而不用swap

$ swapoff -a   
$ vim /etc/fstab
注释掉SWAP分区项，便可

安装go 语言环境（按需）

https://golang.org/dl/
下载 linux版本go，解压后配置环境变量便可

vi /etc/profile
export GOROOT=/usr/local/go
export PATH=$GOROOT/bin:$PATH

$ source profile

建立K8s集群验证

安装cfssl

这里使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件。

mkdir /opt/k8s

cd /opt/k8s

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 cfssl-certinfo

chmod +x *

建立CA证书配置

cd /opt/k8s
config.json 文件

vi  config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

csr.json 文件
vi csr.json

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成CA证书和私钥

master

cd /opt/k8s

./cfssl gencert -initca csr.json | ./cfssljson -bare ca

会生成3个文件ca.csr、ca-key.pem、ca.pem

分发证书

建立证书目录master
mkdir -p /etc/kubernetes/ssl

拷贝全部文件到目录下

cp *.pem /etc/kubernetes/ssl

这里要将文件拷贝到全部的k8s 机器上

scp -P53742 -P53742 *.pem 172.16.16.246:/etc/kubernetes/ssl/

scp -P53742 -P53742 *.pem 172.16.16.250:/etc/kubernetes/ssl/
chmod 777 /etc/kubernetes/ssl/*.pem

安装docker

安装 yum-config-manager
yum -y install yum-utils

导入

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

更新 repo

yum makecache

删除已安装的docker

[root@incubator-dc-016 k8s]# yum remove -y doceker
Loaded plugins: fastestmirror
No Match for argument: doceker
No Packages marked for removal
[root@incubator-dc-016 k8s]# rpm -qa | grep docker
docker-ce-19.03.4-3.el7.x86_64
docker-ce-cli-19.03.4-3.el7.x86_64
[root@incubator-dc-016 k8s]# rpm -e --nodeps docker-ce-19.03.4-3.el7.x86_64
[root@incubator-dc-016 k8s]# rpm -e --nodeps docker-ce-cli-19.03.4-3.el7.x86_64
[root@incubator-dc-016 k8s]# rpm -qa | grep docker

安装docker

yum install docker-ce –y

更改docker配置

修改配置

前面文件先备份。
mv /usr/lib/systemd/system/docker.service /usr/lib/systemd/system/docker.service.bak

#修改
vi /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS $DOCKER_DNS_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

修改其余配置

mkdir -p /usr/lib/systemd/system/docker.service.d/
/*注释
#添加以下 : (注意 environment 必须在同一行，若是出现换行会没法加载)
#iptables=false 会使 docker run 的容器没法连网，false 是由于 calico 有一些高级的应用，须要限制容器互通。
#建议通常状况不添加 --iptables=false,calico须要添加

vi /usr/lib/systemd/system/docker.service.d/docker-options.conf
暂时未加
[Service]
Environment="DOCKER_OPTS=--insecure-registry=10.254.0.0/16 --graph=/opt/docker --registry-mirror=http://b438f72b.m.daocloud.io --disable-legacy-registry --iptables=false"

从新读取配置，启动 docker

systemctl daemon-reload
systemctl start docker
systemctl enable docker

安装etcd集群

在每台上服务器上执行
yum -y install etcd

cd /opt/k8s

vi etcd-csr.json

{
  "CN": "etcd",
  "hosts": [
    "172.16.16.86",
    "172.16.16.246",
    "172.16.16.250"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成 etcd 密钥

/opt/k8s/cfssl gencert -ca=/opt/k8s/ca.pem \
  -ca-key=/opt/k8s/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes etcd-csr.json | /opt/k8s/cfssljson -bare etcd

查看生成

[root@k8s-master ssl]# ls etcd*
etcd.csr etcd-csr.json etcd-key.pem etcd.pem

拷贝到etcd服务器

etcd-1 
cp etcd*.pem /etc/kubernetes/ssl/

etcd-2
scp -P53742 etcd*.pem 172.16.16.246:/etc/kubernetes/ssl/

etcd-3
scp -P53742 etcd*.pem 172.16.16.250:/etc/kubernetes/ssl/

若是 etcd 非 root 用户，读取证书会提示没权限
chmod 644 /etc/kubernetes/ssl/etcd-key.pem

修改etcd配置

#etcd-1
mv /usr/lib/systemd/system/etcd.service /usr/lib/systemd/system/etcd.service.bak
vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
User=etcd
#set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=etcd1 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://172.16.16.86:2380 \
  --listen-peer-urls=https://172.16.16.86:2380 \
  --listen-client-urls=https://172.16.16.86:2379 \
  --advertise-client-urls=https://172.16.16.86:2379 \
  --initial-cluster-token=k8s-etcd-cluster \
  --initial-cluster=etcd1=https://172.16.16.86:2380,etcd2=https://172.16.16.246:2380,etcd3=https://172.16.16.250:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#etcd-2

vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
User=etcd
#set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=etcd2 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://172.16.16.246:2380 \
  --listen-peer-urls=https://172.16.16.246:2380 \
  --listen-client-urls=https://172.16.16.246:2379 \
  --advertise-client-urls=https://172.16.16.246:2379 \
  --initial-cluster-token=k8s-etcd-cluster \
  --initial-cluster=etcd1=https://172.16.16.86:2380,etcd2=https://172.16.16.246:2380,etcd3=https://172.16.16.250:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#etcd-3

vi /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
User=etcd
#set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=etcd3 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://172.16.16.250:2380 \
  --listen-peer-urls=https://172.16.16.250:2380 \
  --listen-client-urls=https://172.16.16.250:2379 \
  --advertise-client-urls=https://172.16.16.250:2379 \
  --initial-cluster-token=k8s-etcd-cluster \
  --initial-cluster=etcd1=https://172.16.16.86:2380,etcd2=https://172.16.16.246:2380,etcd3=https://172.16.16.250:2380 \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动etcd

分别启动全部节点的 etcd 服务

systemctl enable etcd
systemctl start etcd
systemctl status etcd

#若是报错请使用
journalctl -f -t etcd 和 journalctl -u etcd 来定位问题

验证etcd集群状态

查看 etcd 集群状态：

etcdctl --endpoints=https://172.16.16.86:2379 \
        --cert-file=/etc/kubernetes/ssl/etcd.pem \
        --ca-file=/etc/kubernetes/ssl/ca.pem \
        --key-file=/etc/kubernetes/ssl/etcd-key.pem \
        cluster-health

查看 etcd 集群成员：

etcdctl --endpoints=https://172.16.16.86:2379 \
        --cert-file=/etc/kubernetes/ssl/etcd.pem \
        --ca-file=/etc/kubernetes/ssl/ca.pem \
        --key-file=/etc/kubernetes/ssl/etcd-key.pem \
        member list

查看 etcd 集群成员：
etcdctl --endpoints=https://172.16.16.86:2379 \
--cert-file=/etc/kubernetes/ssl/etcd.pem \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--key-file=/etc/kubernetes/ssl/etcd-key.pem \
member list

安装kubectl 工具

Master节点 172.16.16.86
#首先安装 kubectl

wget https://dl.k8s.io/v1.8.0/kubernetes-client-linux-amd64.tar.gz
（若是链接不上，直接去git上下载二进制文件）

tar -xzvf kubernetes-client-linux-amd64.tar.gz
cp kubernetes/client/bin/* /usr/local/bin/
cp kubernetes/client/bin/* /usr/bin/
chmod a+x /usr/local/bin/kube*

验证安装

$ kubectl version

[root@incubator-dc-016 k8s]# kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

建立 admin 证书

kubectl 与 kube-apiserver 的安全端口通讯，须要为安全通讯提供 TLS 证书和秘钥。
cd /opt/k8s
vi admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

生成 admin 证书和私钥

cd /opt/k8s

./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes admin-csr.json | ./cfssljson -bare admin

查看生成

#ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem
cp admin*.pem /etc/kubernetes/ssl/
scp -P53742 admin*.pem 172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 admin*.pem 172.16.16.250:/etc/kubernetes/ssl/

配置 kubectl kubeconfig 文件

server 配置为本机IP 各自链接本机的 Api
#配置 kubernetes 集群

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443

配置客户端认证

kubectl config set-credentials admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin

kubectl config use-context kubernetes

kubectl config文件

#kubeconfig 文件在以下位置:

/root/.kube

部署 Kubernetes Master 节点

Master 须要部署 kube-apiserver , kube-scheduler , kube-controller-manager 这三个组件。 kube-scheduler 做用是调度pods分配到那个node里，简单来讲就是资源调度。 kube-controller-manager 做用是对 deployment controller , replication controller, endpoints controller, namespace controller, and serviceaccounts controller等等的循环控制，与kube-apiserver交互。

安装Master节点组件

#从github 上下载版本
cd /opt/k8s

wget https://dl.k8s.io/v1.8.3/kubernetes-server-linux-amd64.tar.gz

tar -xzvf kubernetes-server-linux-amd64.tar.gz && cd kubernetes

cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/
cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/bin/

建立kubernetes 证书

cd /opt/k8s

vi kubernetes-csr.json

{
  "CN": "kubernetes",
  "hosts": [
    "172.16.16.86",
    "10.254.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

这里 hosts 字段中三个 IP 分别为 127.0.0.1 本机， 172.16.16.86为 Master 的IP，多个Master须要写多个 10.254.0.1 为 kubernetes SVC 的 IP，通常是部署网络的第一个IP , 如: 10.254.0.1 ，在启动完成后，咱们使用 kubectl get svc ，就能够查看到。

生成 kubernetes 证书和私钥

./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes kubernetes-csr.json | ./cfssljson -bare kubernetes

ls -l kubernetes*

拷贝到目录

cp -r kubernetes*.pem /etc/kubernetes/ssl/
scp -P53742 -r kubernetes*.pem 172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 -r kubernetes*.pem 172.16.16.250:/etc/kubernetes/ssl/

配置 kube-apiserver

kubelet 首次启动时向 kube-apiserver 发送 TLS Bootstrapping 请求，kube-apiserver 验证 kubelet 请求中的 token 是否与它配置的 token 一致，若是一致则自动为 kubelet生成证书和秘钥。

生成 token

[root@incubator-dc-016 k8s]# head -c 16 /dev/urandom | od -An -t x | tr -d ''
 49d1b983 9aafea9c 90300962 60d51a3d
49d1b9839aafea9c9030096260d51a3d     需记录下来

建立 token.csv 文件

cd /opt/k8s

vi token.csv
49d1b9839aafea9c9030096260d51a3d,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

#拷贝

cp token.csv /etc/kubernetes/
scp -P53742 token.csv 172.16.16.246:/etc/kubernetes/
scp -P53742 token.csv 172.16.16.250:/etc/kubernetes/

建立 kube-apiserver.service 文件

# 1.8 新增 (Node) --authorization-mode=Node,RBAC
# 自定义 系统 service 文件通常存于 /etc/systemd/system/ 下
# 配置为 各自的本地 IP
vi /etc/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --advertise-address=172.16.16.86 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/lib/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=172.16.16.86 \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --enable-swagger-ui=true \
  --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
  --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \
  --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \
  --etcd-servers=https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379 \
  --event-ttl=1h \
  --kubelet-https=true \
  --insecure-bind-address=172.16.16.86 \
  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-cluster-ip-range=10.254.0.0/16 \
  --service-node-port-range=30000-32000 \
  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --enable-bootstrap-token-auth \
  --token-auth-file=/etc/kubernetes/token.csv \
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#这里面要注意的是 --service-node-port-range=30000-32000
#这个地方是映射外部端口时的端口范围，随机映射也在这个范围内映射，指定映射端口必须也在这个范围内。

启动 kube-apiserver

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

配置 kube-controller-manager

master 配置为各自本地 IP
建立 kube-controller-manager.service 文件

vi /etc/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://172.16.16.86:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-cidr=10.233.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --leader-elect=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

启动 kube-controller-manager

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manager

配置 kube-scheduler

master 配置为各自本地 IP
建立 kube-cheduler.service 文件

vi /etc/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --address=127.0.0.1 \
  --master=http://172.16.16.86:8080 \
  --leader-elect=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

启动 kube-scheduler

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler
systemctl status kube-scheduler

验证 Master 节点

[root@incubator-dc-016 k8s]# kubectl get componentstatuses
NAME                 STATUS      MESSAGE                                                                                  
scheduler            Healthy     ok                                                                                       
controller-manager   Healthy     ok                                                                                       
etcd-0               Healthy     {"health":"true"}                                                                        
etcd-1               Healthy     {"health":"true"}                                                                        
etcd-2               Healthy     {"health":"true"}

部署 Master节点的 Node 部分
Node 部分须要部署的组件有 docker calico kubectl kubelet kube-proxy 这几个组件。

配置 kubelet

kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求，须要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper 角色，而后 kubelet 才有权限建立认证请求(certificatesigningrequests)。

先建立认证请求

#user 为 master 中 token.csv 文件里配置的用户
#只需建立一次就能够
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

建立 kubelet kubeconfig 文件

server 配置为 master 本机 IP

#配置集群

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443 \
  --kubeconfig=bootstrap.kubeconfig

#配置客户端认证

kubectl config set-credentials kubelet-bootstrap \
  --token=49d1b9839aafea9c9030096260d51a3d \
  --kubeconfig=bootstrap.kubeconfig

#配置关联

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

#配置默认关联
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#拷贝生成的 bootstrap.kubeconfig 文件
mv bootstrap.kubeconfig /etc/kubernetes

建立 kubelet.service 文件

建立 kubelet 目录

> 配置为 node 本机 IP

mkdir /var/lib/kubelet

vi /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --address=172.16.16.86 \
  --hostname-override=172.16.16.86 \
  --pod-infra-container-image=jicki/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --cluster_dns=10.254.0.2 \
  --cluster_domain=doone.com. \
  --hairpin-mode promiscuous-bridge \
  --allow-privileged=true \
  --fail-swap-on=false \
  --serialize-image-pulls=false \
  --logtostderr=true \
  --max-pods=512 \
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

#如上配置:
172.16.16.86 为本机的IP
10.254.0.2 预分配的 dns 地址
cluster.local. 为 kubernetes 集群的 domain
jicki/pause-amd64:3.0 这个是 pod 的基础镜像，既 gcr 的 gcr.io/googlecontainers/pause-amd64:3.0 镜像，下载下来修改成本身的仓库中的比较快。

启动 kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

#若是报错请使用
journalctl -f -t kubelet 和 journalctl -u kubelet 来定位问题

配置 TLS 认证

#查看 csr 的名称

[root@incubator-dc-016 kubelet]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-EBjoD_bmtunjaDMTUmlph04kLO9Kz8-jdUhh6GDhb7w   12s       kubelet-bootstrap   Pending

增长认证

kubectl certificate approve node-csr-Sg6CRaxXhdIEJP0hxMHtE2Xoh9fpeFl6OVtocqGeV34

验证 nodes

[root@incubator-dc-016 kubelet]# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
172.16.16.86   Ready     <none>    30s       v1.8.3

#成功之后会自动生成配置文件与密钥

配置文件

ls /etc/kubernetes/kubelet.kubeconfig   
/etc/kubernetes/kubelet.kubeconfig

秘钥文件

ls /etc/kubernetes/ssl/kubelet*
/etc/kubernetes/ssl/kubelet-client.crt  /etc/kubernetes/ssl/kubelet.crt
/etc/kubernetes/ssl/kubelet-client.key  /etc/kubernetes/ssl/kubelet.key

配置 kube-proxy

建立 kube-proxy 证书

#证书方面因为咱们node端没有装 cfssl
#咱们回到 master 端机器去配置证书，而后拷贝过来

cd /opt/k8s
vi kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Nanjing",
      "L": "Nanjing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成 kube-proxy 证书和私钥

./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/opt/k8s/config.json \
  -profile=kubernetes  kube-proxy-csr.json | ./cfssljson -bare kube-proxy

查看生成

ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

拷贝到目录

cp kube-proxy*.pem /etc/kubernetes/ssl/
scp -P53742 kube-proxy*.pem 172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 kube-proxy*.pem 172.16.16.250:/etc/kubernetes/ssl/

建立 kube-proxy kubeconfig 文件

server 配置为各自本机IP

#配置集群

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443 \
  --kubeconfig=kube-proxy.kubeconfig

配置客户端认证

kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

配置关联

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

配置默认关联

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

拷贝到目录

mv kube-proxy.kubeconfig /etc/kubernetes/

建立 kube-proxy.service 文件

配置为各自的 IP
#建立 kube-proxy 目录

mkdir -p /var/lib/kube-proxy

vi /etc/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --bind-address=172.16.16.86 \
  --hostname-override=172.16.16.86 \
  --cluster-cidr=10.254.0.0/16 \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动 kube-proxy

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

#若是报错请使用
journalctl -f -t kube-proxy 和 journalctl -u kube-proxy 来定位问题

部署Kubernetes Node节点

Node 节点基于 Nginx 负载 API 作 Master HA。172.16.16.246，172.16.16.250。
#master 之间除 api server 之外其余组件经过 etcd 选举，api server 默认不做处理；在每一个 node 上启动一个 nginx，每一个 nginx 反向代理全部 api server，node 上 kubelet、kube-proxy 链接本地的 nginx 代理端口，当 nginx 发现没法链接后端时会自动踢掉出问题的 api server，从而实现 api server 的 HA。

安装Node节点组件

tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes
cp -r server/bin/{kube-proxy,kubelet,kubectl} /usr/local/bin/
cp -r server/bin/{kube-proxy,kubelet,kubectl} /usr/bin/

#ALL node

mkdir -p /etc/kubernetes/ssl/

scp -P53742 ca.pem kube-proxy.pem kube-proxy-key.pem  172.16.16.246:/etc/kubernetes/ssl/
scp -P53742 ca.pem kube-proxy.pem kube-proxy-key.pem  172.16.16.250:/etc/kubernetes/ssl/

建立 kubelet kubeconfig 文件

kubectl config set-cluster kubernetes   --certificate-authority=/etc/kubernetes/ssl/ca.pem   --embed-certs=true   --server=https://172.16.16.86:6443 --kubeconfig=bootstrap.kubeconfig
#必须是server-master节点

配置客户端认证

kubectl config set-credentials kubelet-bootstrap \
  --token=49d1b9839aafea9c9030096260d51a3d \
  --kubeconfig=bootstrap.kubeconfig

配置关联

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

配置默认关联

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

拷贝生成的 bootstrap.kubeconfig 文件

mv bootstrap.kubeconfig /etc/kubernetes/

建立 kubelet.service 文件

mkdir -p /var/lib/kubelet
vi /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --address=172.16.16.246 \
  --hostname-override=172.16.16.246 \
  --pod-infra-container-image=jicki/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --cluster_dns=10.254.0.2 \
  --cluster_domain=doone.com. \
  --hairpin-mode promiscuous-bridge \
  --allow-privileged=true \
  --fail-swap-on=false \
  --serialize-image-pulls=false \
  --logtostderr=true \
  --max-pods=512 \
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

启动 kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

建立 kube-proxy kubeconfig 文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://172.16.16.86:6443 \
  --kubeconfig=kube-proxy.kubeconfig

配置客户端认证

kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

配置关联

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

配置默认关联

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

拷贝到目录

mv kube-proxy.kubeconfig /etc/kubernetes/

建立 kube-proxy.service 文件

mkdir -p /var/lib/kube-proxy
vi /etc/systemd/system/kube-proxy.service
#node节点ip地址要改
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --bind-address=172.16.16.246 \
  --hostname-override=172.16.16.246 \
  --cluster-cidr=10.254.0.0/16 \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
  --logtostderr=true \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动 kube-proxy

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

建立Nginx 代理

在每一个 node 都必须建立一个 Nginx 代理，这里特别注意，当 Master 也作为 Node 的时候不须要配置 Nginx-proxy

在node上建立

建立配置目录

mkdir -p /etc/nginx

####写入代理配置

cat << EOF > /etc/nginx/nginx.conf
error_log stderr notice;

worker_processes auto;
events {
  multi_accept on;
  use epoll;
  worker_connections 1024;
}

stream {
    upstream kube_apiserver {
        least_conn;
        server 172.16.16.86:6443;       
    }

    server {
        listen        0.0.0.0:6443;
        proxy_pass    kube_apiserver;
        proxy_timeout 10m;
        proxy_connect_timeout 1s;
    }
}
EOF

配置 Nginx 基于 docker 进程，而后配置 systemd 来启动

cat << EOF > /etc/systemd/system/nginx-proxy.service

[Unit]
Description=kubernetes apiserver docker wrapper
Wants=docker.socket
After=docker.service

[Service]
User=root
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run -p 127.0.0.1:6443:6443 \\
                              -v /etc/nginx:/etc/nginx \\
                              --name nginx-proxy \\
                              --net=host \\
                              --restart=on-failure:5 \\
                              --memory=512M \\
                              nginx:1.13.5-alpine
ExecStartPre=-/usr/bin/docker rm -f nginx-proxy
ExecStop=/usr/bin/docker stop nginx-proxy
Restart=always
RestartSec=15s
TimeoutStartSec=30s

[Install]
WantedBy=multi-user.target
EOF

启动 Nginx

systemctl daemon-reload
systemctl start nginx-proxy
systemctl enable nginx-proxy
systemctl status nginx-proxy

重启 Node 的 kubelet 与 kube-proxy

systemctl restart kubelet
systemctl status kubelet
systemctl restart kube-proxy
systemctl status kube-proxy

在Master 配置经过 TLS 认证

#查看 csr 的名称

kubectl get csr

增长认证

kubectl certificate approve NAME

[root@incubator-dc-016 cx]# kubectl certificate approve node-csr-EBjoD_bmtunjaDMTUmlph04kLO9Kz8-jdUhh6GDhb7w
certificatesigningrequest "node-csr-EBjoD_bmtunjaDMTUmlph04kLO9Kz8-jdUhh6GDhb7w" approved
[root@incubator-dc-016 cx]# kubectl certificate approve node-csr-v-UvG2zhPQRMf3hDTMUqSq_wvsurSlNFc7CHjl1v3ss
certificatesigningrequest "node-csr-v-UvG2zhPQRMf3hDTMUqSq_wvsurSlNFc7CHjl1v3ss" approved
[root@incubator-dc-016 cx]# kubectl certificate approve node-csr-Sg6CRaxXhdIEJP0hxMHtE2Xoh9fpeFl6OVtocqGeV34 "node-csr-z2sRlOk0UKbsaB_8J9ZhjtnS7gt886GVZBAYESiuf10" approved
[root@incubator-dc-016 cx]#

部署Calico网络

修改 kubelet.service

在每一个节点
增长以下配置

vi /etc/systemd/system/kubelet.service
--network-plugin=cni \

从新加载配置

systemctl daemon-reload
systemctl restart kubelet.service
systemctl status kubelet.service

获取Calico 配置

Calico 部署仍然采用 “混搭” 方式，即 Systemd 控制 calico node，cni 等由 kubernetes daemonset 安装。

#获取 calico.yaml master机器上执行

cat <<EOF > calico-controller.yml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-kube-controllers
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    - extensions
    resources:
      - pods
      - namespaces
      - networkpolicies
    verbs:
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: calico-policy-controller
  namespace: kube-system
  labels:
    k8s-app: calico-policy
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-policy-controller
      namespace: kube-system
      labels:
        k8s-app: calico-policy
    spec:
      hostNetwork: true
      serviceAccountName: calico-kube-controllers
      containers:
      - name: calico-policy-controller
        image: quay.io/calico/kube-controllers:v1.0.0
        env:
          - name: ETCD_ENDPOINTS
            value: "https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379"
          - name: ETCD_CA_CERT_FILE
            value: "/etc/kubernetes/ssl/ca.pem"
          - name: ETCD_CERT_FILE
            value: "/etc/kubernetes/ssl/etcd.pem"
          - name: ETCD_KEY_FILE
            value: "/etc/kubernetes/ssl/etcd-key.pem"
        volumeMounts:
          - mountPath: /etc/kubernetes/ssl/
            name: etcd-ca-certs
            readOnly: true
      volumes:
        - hostPath:
            path: /etc/kubernetes/ssl/
            type: DirectoryOrCreate
          name: etcd-ca-certs
EOF
kubectl apply -f  calico-controller.yml

kubectl -n kube-system get po -l k8s-app=calico-policy
需修改yaml文件内ETCD集群的IP地址

在全部节点下载 Calico

cd /usr/local/bin/
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.2.6/calicoctl

如下操做在三个节点上都须要实现
cd /usr/local/bin/
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.2.6/calicoctl
chmod +x calicoctl
scp -P53742 calicoctl root@172.16.16.246:/usr/local/bin/
scp -P53742 calicoctl root@172.16.16.250:/usr/local/bin/

下载calico、calico-ipam

wget -N -P /opt/cni/bin/ https://github.com/projectcalico/calico-cni/releases/download/v3.1.6/calico
wget -N -P /opt/cni/bin/ https://github.com/projectcalico/calico-cni/releases/download/v3.1.6/calico-ipam
mkdir -p /opt/cni/bin/
cp -rf /opt/k8s/calico /opt/cni/bin/
cp -rf /opt/k8s/calico-ipam /opt/cni/bin/
scp -P53742 calico root@172.16.16.246:/opt/cni/bin/
scp -P53742 calico root@172.16.16.250:/opt/cni/bin/
scp -P53742 calico-ipam root@172.16.16.246:/opt/cni/bin/
scp -P53742 calico-ipam root@172.16.16.250:/opt/cni/bin/

chmod +x /opt/cni/bin/calico /opt/cni/bin/calico-ipam

在全部节点下载 CNI plugins配置文件

vi /etc/cni/net.d/10-calico.conf

{
    "name": "calico-k8s-network",
    "cniVersion": "0.1.0",
    "type": "calico",
    "etcd_endpoints": "https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379",
    "etcd_ca_cert_file": "/etc/kubernetes/ssl/ca.pem",
    "etcd_cert_file": "/etc/kubernetes/ssl/etcd.pem",
    "etcd_key_file": "/etc/kubernetes/ssl/etcd-key.pem",
    "log_level": "info",
    "ipam": {
        "type": "calico-ipam"
    },
    "policy": {
        "type": "k8s"
    },
    "kubernetes": {
        "kubeconfig": "/etc/kubernetes/kubelet.kubeconfig"
    }
}

部署 KubeDNS

下载kubeDNS镜像

#官方镜像

gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7

#国内镜像

jicki/k8s-dns-sidecar-amd64:1.14.7
jicki/k8s-dns-kube-dns-amd64:1.14.7
jicki/k8s-dns-dnsmasq-nanny-amd64:1.14.7

下载yaml文件

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kube-dns.yaml.base

#修改后缀

mv kube-dns.yaml.base kube-dns.yaml

系统预约义的 RoleBinding

预约义的 RoleBinding system:kube-dns 将 kube-system 命名空间的 kube-dns ServiceAccount 与 system:kube-dns Role 绑定，该 Role 具备访问 kube-apiserver DNS 相关 API 的权限；

[root@k8s-master kubedns]# kubectl get clusterrolebindings system:kube-dns -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2017-09-29T04:12:29Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-dns
  resourceVersion: "78"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system%3Akube-dns
  uid: 688627eb-a4cc-11e7-9f6b-44a8420b9988
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-dns
subjects:
- kind: ServiceAccount
  name: kube-dns
  namespace: kube-system

Kube-dns yaml文件

cat <<EOF > kube-dns.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.254.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      dnsPolicy: Default
      serviceAccountName: kube-dns
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-dns-kube-dns-amd64:1.14.7
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - "--domain=cluster.local"
        - --dns-port=10053
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
      - name: dnsmasq
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - "-v=2"
        - "-logtostderr"
        - "-configDir=/etc/k8s/dns/dnsmasq-nanny"
        - "-restartDnsmasq=true"
        - "--"
        - "-k"
        - "--cache-size=1000"
        - "--log-facility=-"
        - "--server=/cluster.local/127.0.0.1#10053"
        - "--server=/in-addr.arpa/127.0.0.1#10053"
        - "--server=/ip6.arpa/127.0.0.1#10053"
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-dns-sidecar-amd64:1.14.7
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - "--v=2"
        - "--logtostderr"
        - "--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A"
        - "--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A"
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
EOF

导入yaml文件

[root@incubator-dc-016 k8s]# kubectl create -f kube-dns.yml
serviceaccount "kube-dns" created
service "kube-dns" created
deployment "kube-dns" created
[root@incubator-dc-016 k8s]#

查看kubedns服务

[root@incubator-dc-016 k8s]# kubectl get all --namespace=kube-system
NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/calico-policy-controller   1         1         1            1           4h
deploy/kube-dns                   1         1         1            0           19s

NAME                                     DESIRED   CURRENT   READY     AGE
rs/calico-policy-controller-5586b678b5   0         0         0         1h
rs/calico-policy-controller-57dd959cc9   0         0         0         4h
rs/calico-policy-controller-6d94579b6b   1         1         1         56m
rs/kube-dns-794845bc6f                   1         1         0         19s

NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/calico-policy-controller   1         1         1            1           4h
deploy/kube-dns                   1         1         1            0           19s

NAME                                     DESIRED   CURRENT   READY     AGE
rs/calico-policy-controller-5586b678b5   0         0         0         1h
rs/calico-policy-controller-57dd959cc9   0         0         0         4h
rs/calico-policy-controller-6d94579b6b   1         1         1         56m
rs/kube-dns-794845bc6f                   1         1         0         19s

NAME                                           READY     STATUS              RESTARTS   AGE
po/calico-policy-controller-6d94579b6b-vksgv   1/1       Running             0          56m
po/kube-dns-794845bc6f-464d8                   0/3       ContainerCreating   0          19s

NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
svc/kube-dns   ClusterIP   10.254.0.2   <none>        53/UDP,53/TCP   19s

建立 calico-node.service 文件

上一步注释了 calico.yaml 中 Calico Node 相关内容，为了防止自动获取 IP 出现问题，将其移动到 Systemd，Systemd service 配置以下，每一个节点都要安装 calico-node 的 Service，其余节点请自行修改 ip。

cat <<EOF > /lib/systemd/system/calico-node.service
[Unit]
Description=calico node
After=docker.service
Requires=docker.service

[Service]
User=root
PermissionsStartOnly=true
ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \
  -e ETCD_ENDPOINTS=https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379 \
  -e ETCD_CA_CERT_FILE=/etc/kubernetes/ssl/ca.pem \
  -e ETCD_CERT_FILE=/etc/kubernetes/ssl/etcd.pem \
  -e ETCD_KEY_FILE=/etc/kubernetes/ssl/etcd-key.pem \
  -e NODENAME=${HOSTNAME} \
  -e IP= \
  -e NO_DEFAULT_POOLS= \
  -e AS= \
  -e CALICO_LIBNETWORK_ENABLED=true \
  -e IP6= \
  -e CALICO_NETWORKING_BACKEND=bird \
  -e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \
  -e FELIX_HEALTHENABLED=true \
  -e CALICO_IPV4POOL_CIDR=10.233.0.0/16 \
  -e CALICO_IPV4POOL_IPIP=always \
  -e IP_AUTODETECTION_METHOD=interface=eth0 \
  -e IP6_AUTODETECTION_METHOD=interface=eth0 \
-v /etc/kubernetes/ssl:/etc/kubernetes/ssl \
  -v /var/run/calico:/var/run/calico \
  -v /lib/modules:/lib/modules \
  -v /run/docker/plugins:/run/docker/plugins \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/log/calico:/var/log/calico \
  quay.io/calico/node:v2.6.2
ExecStop=/usr/bin/docker rm -f calico-node
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

启动 Calico Node

Calico Node 采用 Systemd 方式启动，在每一个节点配置好 Systemd service后，每一个节点修改对应的 calico-node.service 中的 IP 和节点名称，而后启动便可

systemctl daemon-reload
systemctl restart calico-node
systemctl restart kubelet

在全部节点启动 Calico-node
systemctl enable calico-node.service && systemctl start calico-node.service

在master查看 Calico nodes

cat <<EOF > ~/calico-rc
export ETCD_ENDPOINTS="https://172.16.16.86:2379,https://172.16.16.246:2379,https://172.16.16.250:2379"
export ETCD_CA_CERT_FILE="/etc/kubernetes/ssl/ca.pem"
export ETCD_CERT_FILE="/etc/kubernetes/ssl/etcd.pem"
export ETCD_KEY_FILE="/etc/kubernetes/ssl/etcd-key.pem"
EOF

. ~/calico-rc
calicoctl get node -o wide

查看 pending 的 pod 是否已执行

kubectl -n kube-system get po

部署 Ingress

Kubernetes 暴露服务的方式目前只有三种：LoadBlancer Service、NodePort Service、Ingress；什么是 Ingress ? Ingress 就是利用 Nginx Haproxy 等负载均衡工具来暴露 Kubernetes 服务

配置调度 node

#ingress 有多种方式 1. deployment 自由调度 replicas 2. daemonset 全局调度分配到全部node里

#deployment 自由调度过程当中，因为咱们须要约束 controller 调度到指定的 node 中，因此须要对 node 进行 label 标签

#默认以下:

[root@incubator-dc-016 k8s]# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
172.16.16.246   Ready     <none>    4h        v1.8.3
172.16.16.250   Ready     <none>    4h        v1.8.3
172.16.16.86    Ready     <none>    5h        v1.8.3
[root@incubator-dc-016 k8s]# 

# 对 86 打上 label
kubectl label nodes 172.16.16.86 ingress=proxy

# 打完标签之后
NAME            STATUS    ROLES     AGE       VERSION   LABELS
172.16.16.246   Ready     <none>    4h        v1.8.3    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.16.16.246
172.16.16.250   Ready     <none>    4h        v1.8.3    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.16.16.250
172.16.16.86    Ready     <none>    5h        v1.8.3    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress=proxy,kubernetes.io/hostname=172.16.16.86

下载Ingress镜像

#官方镜像
gcr.io/google_containers/defaultbackend:1.0
gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.17

#国内镜像
jicki/defaultbackend:1.0
jicki/nginx-ingress-controller:0.9.0-beta.17

下载yaml文件

#Ingress yaml 文件模板
#default-backend.yaml
cat <<EOF >default-backend.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  selector:
    k8s-app: default-http-backend
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: registry.cn-qingdao.aliyuncs.com/kube8s/defaultbackend:1.0 
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend
EOF
#rbac.yaml

cat <<EOF >rbac.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        ingress: proxy
      containers:
        - name: nginx-ingress-controller
          image: jicki/nginx-ingress-controller:0.9.0-beta.17
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --apiserver-host=http://172.16.16.86:8080
#- --configmap=$(POD_NAMESPACE)/nginx-configuration
#- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
#- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MASTER
              value: http://172.16.16.86:8080
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
EOF

#with-rbac.yaml
cat <<EOF >with-rbac.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        ingress: proxy
      containers:
        - name: nginx-ingress-controller
          image: jicki/nginx-ingress-controller:0.9.0-beta.17
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --apiserver-host=http://172.16.16.86:8080
#- --configmap=$(POD_NAMESPACE)/nginx-configuration
#- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
# - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MASTER
              value: http://172.16.16.86:8080
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
EOF

导入yaml文件

kubectl apply -f default-backend.yaml 
kubectl apply -f rbac.yaml
kubectl apply -f with-rbac.yaml
[root@incubator-dc-016 Ingress]# curl http://172.16.16.86:8080/healthz

查看ingress服务

kubectl get svc -n kube-system

部署 Dashboard

###下载dashboard镜像

#官方镜像
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3

#国内镜像
jicki/kubernetes-dashboard-amd64:v1.6.3

下载yaml文件

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-controller.yaml

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-service.yaml

#由于开启了 RBAC 因此这里须要建立一个 RBAC 认证

vi dashboard-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: dashboard
subjects:
  - kind: ServiceAccount
    name: dashboard
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

#Dashboard yaml文件模板

#dashboard-controller.yaml
cat <<EOF >dashboard-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: dashboard
      containers:
      - name: kubernetes-dashboard
        image: jicki/kubernetes-dashboard-amd64:v1.6.3
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9090
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
EOF

#dashboard-service.yaml
cat <<EOF >dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    k8s-app: kubernetes-dashboard
  ports:
    - port: 80
      targetPort: 9090
EOF

导入yaml文件

kubectl apply -f .
deployment "kubernetes-dashboard" created
serviceaccount "dashboard" created
clusterrolebinding "dashboard" created
service "kubernetes-dashboard" created

查看Dashboard服务

kubectl get svc -n kube-system

这里部署结束，看的比较复杂，但也是用心一笔笔操做的来的，如今都是kuberadmin部署k8s,但至少也要手动去部署一遍，这样才能了解里面的意思和原理，排查错误也方便，附上报错问题及解决方法

报错汇总

没法重启docker

Registered Authentication Agent for unix-process:26237:1270527351
解决：echo 1 > /proc/sys/vm/drop_caches

etcd启动不了

open /etc/kubernetes/ssl/etcd.pem: permission denied
解决：
chmod +x /etc/kubernetes/ssl/etcd.pem
chmod 755 /etc/kubernetes/ssl/

Etcd 127.0.0.1错误

Apr 07 14:44:59 incubator-dc-016 etcd[490]: The scheme of client url http://127.0.0.1:2379 is HTTP while peer key/cert files are presented. Ignored key/cert files.
Apr 07 14:44:59 incubator-dc-016 etcd[490]: listening for client requests on 127.0.0.1:2379
Apr 07 14:44:59 incubator-dc-016 etcd[490]: listening for client requests on 172.16.16.86:2379
Apr 07 14:44:59 incubator-dc-016 etcd[490]: create snapshot directory error: mkdir /var/lib/etcd/member/snap: permission denied

解决：
解决方法和思路:
删除全部etcd数据,从新初始化.

rm -rf /var/lib/etcd/*
systemctl daemon-reload && systemctl restart etcd
systemctl status etcd.service

Api启动不了

端口被占用错误
failed to listen on 172.16.16.86:6443: listen tcp 172.16.16.86:6443: bind: address already in use

解决：
发现是docker占用，中止docker nginx使用
而后重启api服务

Proxy启动不了

Failed at step CHDIR spawning /usr/local/bin/kube-proxy: No such file or directory

解决：
须要mkdir -p /var/lib/kube-proxy

calico启动不了

calico版本以3.2.6版本为基础，原则上如今的安装不能低于3.1，不然会出现各类问题，已踩过相关的坑。
当出现Kubernetes Calico node ‘XXXXXXXXXXX’ already using IPv4 Address XXXXXXXXX, CrashLoopBackOff错误时，多是calico的版本太低

ERROR: Error accessing the Calico datastore: open /etc/kubernetes/ssl/etcd.pem: no such file or directoryCalico node failed to start
-v /etc/kubernetes/ssl:/etc/kubernetes/ssl \

Apr 21 16:06:34 incubator-dc-002 docker[22733]: ERROR: Couldn't autodetect a management IPv4 address:
Apr 21 16:06:34 incubator-dc-002 docker[22733]: -  provide an IPv4 address by configuring one in the node resource, or
Apr 21 16:06:34 incubator-dc-002 docker[22733]: -  provide an IPv4 address using the IP environment, or
Apr 21 16:06:34 incubator-dc-002 docker[22733]: -  if auto-detecting, use a different autodetection method.
  -e IP_AUTODETECTION_METHOD=interface=eth0 \
  -e IP6_AUTODETECTION_METHOD=interface=eth0 \

但愿你们共同进步，一块儿学习，往更优秀的方向走！

K8s手动方式搭建平台及问题汇总

k8s概要

kubernetes的特色：

kubernetes的体系架构

kubernetes 组件特色

手动部署安装

初始化环境

配置hosts地址

关闭防火墙

关闭selinux

关闭swap

安装go 语言环境（按需）

建立K8s集群验证

安装cfssl

建立CA证书配置

生成CA证书和私钥

分发证书

拷贝全部文件到目录下

这里要将文件拷贝到全部的k8s 机器上

安装docker

导入

更新 repo

删除已安装的docker

安装docker

更改docker配置

修改其余配置

从新读取配置，启动 docker

安装etcd集群

生成 etcd 密钥

查看生成

拷贝到etcd服务器

修改etcd配置

启动etcd

验证etcd集群状态

安装kubectl 工具

验证安装

建立 admin 证书

生成 admin 证书和私钥

查看生成

配置 kubectl kubeconfig 文件

配置 客户端认证

kubectl config文件

部署 Kubernetes Master 节点

安装Master节点组件

建立kubernetes 证书

生成 kubernetes 证书和私钥

拷贝到目录

配置 kube-apiserver

生成 token

建立 token.csv 文件

建立 kube-apiserver.service 文件

启动 kube-apiserver

配置 kube-controller-manager

启动 kube-controller-manager

配置 kube-scheduler

启动 kube-scheduler

验证 Master 节点

配置 kubelet

先建立认证请求

建立 kubelet kubeconfig 文件

建立 kubelet.service 文件

启动 kubelet

配置 TLS 认证

增长认证

验证 nodes

配置文件

秘钥文件

配置 kube-proxy

建立 kube-proxy 证书

生成 kube-proxy 证书和私钥

查看生成

拷贝到目录

建立 kube-proxy kubeconfig 文件

配置客户端认证

配置关联

配置默认关联

拷贝到目录

建立 kube-proxy.service 文件

启动 kube-proxy

部署Kubernetes Node节点

配置客户端认证

配置调度 node