k8s部署高可用Ingress

时间 2019-11-18

标签 k8s 部署可用 ingress 繁體版

原文原文链接

部署高可用Ingress

官网地址https://kubernetes.github.io/ingress-nginx/deploy/html

获取ingress的编排文件node

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yamlnginx

增长节点标签git

kubectl label node k8snode1 ingresscontroller=true
kubectl get nodes --show-labelsgithub

附：apache

删除标签
kubectl label node k8snode1 ingresscontroller-

更新标签
kubectl label node k8snode1 ingresscontroller=false --overwrite

一、修改Deployment为DaemonSet，并注释掉副本数vim

kind: DaemonSet
  #replicas: 1

二、启用hostNetwork网络，并指定运行节点后端

hostNetwork暴露ingress-nginx controller的相关业务端口到主机，这样node节点主机所在网络的其余主机，均可以经过该端口访问到此应用程序。api

nodeSelector指定以前添加ingresscontroller=true标签的node网络

hostNetwork: true
      nodeSelector:
        ingresscontroller: 'true'

三、修改镜像地址
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.24.1

四、修改容器端口

args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=88 #默认80
            - --https-port=4433 #默认443
            
            
          ports:
            - name: http
              containerPort: 88 #更改成88
            - name: https
              containerPort: 4433 #更改成4433

五、增长master节点容忍

tolerations: #增长容忍，可分配到master节点
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"

六、mandatory.yaml以下
vim mandatory.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      nodeSelector:
        ingresscontroller: 'true'
      tolerations: #增长容忍，可分配到master节点
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=88 #默认80
            - --https-port=4433 #默认443
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 88 #更改成88
            - name: https
              containerPort: 4433 #更改成4433
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

测试

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-static
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-static
  labels:
    name: nginx-static
spec:
  replicas: 1
  template:
    metadata:
      labels:
       name: nginx-static
    spec:
      containers:
       - name: nginx-static
         image: nginx:latest
         volumeMounts:
          - mountPath: /etc/localtime
            name: vol-localtime
            readOnly: true

         ports:
          - containerPort: 80
      volumes:
         - name: vol-localtime
           hostPath:
            path: /etc/localtime
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-static
  labels:
   name: nginx-static
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    name: http
  selector:
    name: nginx-static

--- 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: submodule-checker-ingress
spec:
  rules:
  - host: nginx.weave.pub
    http:
      paths:
      - backend:
          serviceName: nginx-static
          servicePort: 80

kubectl create -f nginx-static.yaml

访问

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.x.x.x master1 nginx.test.pub
10.x.x.x master2

[root@master1 ingress]# curl nginx.test.pub
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>

1.1 Ingress简介

1.1.1需求

coredns是实现pods之间经过域名访问，若是外部须要访问service服务，需访问对应的NodeIP:Port。可是因为NodePort须要指定宿主机端口，一旦服务多起来，多个端口就难以管理。那么，这种状况下，使用Ingress暴露服务更加合适。

1.1.2 Ingress组件说明

使用Ingress时通常会有三个组件：反向代理负载均衡器、Ingress Controller、Ingress

一、反向代理负载均衡器

反向代理负载均衡器很简单，说白了就是 nginx、apache 等中间件，新版k8s已经将Nginx与Ingress Controller合并为一个组件，因此Nginx无需单独部署，只须要部署Ingress Controller便可。在集群中反向代理负载均衡器能够自由部署，可使用 Replication Controller、Deployment、DaemonSet 等等方式

二、Ingress Controller

Ingress Controller 实质上能够理解为是个监视器，Ingress Controller 经过不断地跟 kubernetes API 打交道，实时的感知后端 service、pod 等变化，好比新增和减小 pod，service 增长与减小等；当获得这些变化信息后，Ingress Controller 再结合下文的 Ingress 生成配置，而后更新反向代理负载均衡器，并刷新其配置，达到服务发现的做用

三、Ingress

Ingress 简单理解就是个规则定义；好比说某个域名对应某个 service，即当某个域名的请求进来时转发给某个 service;这个规则将与 Ingress Controller 结合，而后 Ingress Controller 将其动态写入到负载均衡器配置中，从而实现总体的服务发现和负载均衡

总体关系以下图所示：

从上图中能够很清晰的看到，实际上请求进来仍是被负载均衡器拦截，好比 nginx，而后 Ingress Controller 经过跟 Ingress 交互得知某个域名对应哪一个 service，再经过跟 kubernetes API 交互得知 service 地址等信息；综合之后生成配置文件实时写入负载均衡器，而后负载均衡器 reload 该规则即可实现服务发现，即动态映射。

3.1.3 Nginx-Ingress工做原理

ingress controller经过和kubernetes api交互，动态的去感知集群中ingress规则变化；而后读取它，按照自定义的规则，规则就是写明了哪一个域名对应哪一个service，生成一段nginx配置；再写到nginx-ingress-control的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中；而后reload一下使配置生效。以此达到域名分配置和动态更新的问题。

说明：基于nginx服务的ingress controller根据不一样的开发公司，又分为：

k8s社区的ingres-nginx（https://github.com/kubernetes/ingress-nginx）
nginx公司的nginx-ingress（https://github.com/nginxinc/kubernetes-ingress）

3.2.4 Ingress Controller高可用架构

做为集群流量接入层，Ingress Controller的高可用性显得尤其重要，高可用性首先要解决的就是单点故障问题，通常经常使用的是采用多副本部署的方式，咱们在Kubernetes集群中部署高可用Ingress Controller接入层一样采用多节点部署架构，同时因为Ingress做为集群流量接入口，建议采用独占Ingress节点的方式，以免业务应用与Ingress服务发生资源争抢。

如上述部署架构图，由多个独占Ingress实例组成统一接入层承载集群入口流量，同时可依据后端业务流量水平扩缩容Ingress节点。固然若是您前期的集群规模并不大，也能够采用将Ingress服务与业务应用混部的方式，但建议进行资源限制和隔离。