Kubernetes 部署 Nginx Ingress Controller 之 nginxinc/kubernetes-ingress

更新：这里用的是 nginxinc/kubernetes-ingress ，还有个 kubernetes/ingress-nginx ，它们的区别见 Differences Between nginxinc/kubernetes-ingress and kubernetes/ingress-nginx Ingress Controllers ，后来咱们选用了 kubernetes/ingress-nginx ，详见博文。

开始天真地觉得只要写一个 ingress 配置文件并部署好就好了。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cnblogs-ingress
spec:
  rules:
  - host: q.cnblogs.com
    http:
      paths:
        - backend:
            serviceName: q-web
            servicePort: 80

# kubectl apply -f cnblogs-ingress.yaml
# kubectl get ingress
NAME              HOSTS           ADDRESS   PORTS   AGE
cnblogs-ingress   q.cnblogs.com             80      6h18

但部署后发现全部 node 服务器上没有任何进程监听 80 端口，显然不对。

从 k8s 帮助文档中知道了答案：

You must have an ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.
In order for the Ingress resource to work, the cluster must have an ingress controller running.
Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster. Use this page to choose the ingress controller implementation that best fits your cluster.

原来 k8s 没有内置 ingress controller ，须要安装第三方的 ingress controller ，好比 nginx ingress controller ，上面经过 cnblogs-ingress.yaml 只是建立了 ingress 资源。那为何经过 deployment.yaml 建立了 deployment 资源就能正常部署 pod ？那是由于 kube-controller-manager 中内置了 deployment controller 。

咱们选用 nginx ingress controller ，部署操做步骤以下（参考文档）：
1）从 github 上签出 kubernetes-ingress 仓库

$ git clone https://github.com/nginxinc/kubernetes-ingress/
$ cd kubernetes-ingress
$ git checkout v1.6.1 -f
$ cd deployments

2）建立 namespace 与 ServiceAccount ，都叫 nginx-ingress

kubectl apply -f common/ns-and-sa.yaml

3）建立 cluster role 与 cluster role binding

kubectl apply -f rbac/rbac.yaml

4）建立 secret
使用本身的证书文件建立 secret

kubectl create secret tls default-server-secret --cert=path/to/cert.pem --key=path/to/key.pem

或者使用 nginx-ingress 自带的证书建立 sescret

kubectl apply -f common/default-server-secret.yaml

5）建立 ConfigMap

kubectl apply -f common/nginx-config.yaml

6）建立 custom resource definitions

kubectl apply -f common/custom-resource-definitions.yaml

7）建立 DaemonSet

kubectl apply -f daemon-set/nginx-ingress.yaml

8）查看 pod 是否部署成功

$ kubectl get pods --namespace=nginx-ingress                                                                                                1 ↵
NAME                  READY   STATUS    RESTARTS   AGE
nginx-ingress-7xdzp   1/1     Running   5          12m
nginx-ingress-rs4th   1/1     Running   0          114s
nginx-ingress-w2fnh   1/1     Running   0          12m
nginx-ingress-z54r6   1/1     Running   5          12m

9）建立监听 31080 端口的 NodePort 类型的 service

配置文件 nodeport.yaml （去掉了443端口）
注0：nodePort 只能使用 30000-32767 范围的端口。
注1：去掉了443端口，咱们在最前端使用了阿里云负载均衡，请求都经过 http 转发。

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  type: NodePort 
  ports:
  - nodePort: 31080
     port: 80
     targetPort: 80
     protocol: TCP
     name: http
  selector:
    app: nginx-ingress

部署命令

kubectl apply -f service/nodeport.yaml

10）检查 nginx-ingress 部署成功

进入 nginx-ingress 容器

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress /bin/bash

查看 nginx 配置

cat /etc/nginx/conf.d/production-cnblogs-ingress.conf

确认 ingress 中添加的转发规则已被导入

upstream production-cnblogs-ingress-q.cnblogs.com-q-web-80 {
    zone production-cnblogs-ingress-q.cnblogs.com-q-web-80 256k;
    random two least_conn;
    
    server 192.168.107.211:80 max_fails=1 fail_timeout=10s max_conns=0;
    server 192.168.186.72:80 max_fails=1 fail_timeout=10s max_conns=0;  
}

server {
    listen 80;
    server_tokens on;
    server_name q.cnblogs.com;
    
    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        
        proxy_pass http://production-cnblogs-ingress-q.cnblogs.com-q-web-80;
    }
}

至此 nginx-ingress 部署成功。

解决转发 X-Forwarded-Proto 请求头问题

解决方法：在 ingress 配置文件中添加 nginx.org/redirect-to-https: "true"，详见博问 K8s Nginx Ingress Controller 转发 X-Forwarded-Proto 请求头的问题

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/redirect-to-https: "true"

添加 proxy_set_header 配置

经过 ingress 的 nginx.org/location-snippets 注解添加以下的配置：

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    nginx.org/location-snippets: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

七层负载均衡对 nginx-ingress 进行健康检查问题

详见博问 阿里云负载均衡对 K8s Nginx Ingress 的健康检查问题

基于二级域名自动转发到 service 的实现方法

详见博问 K8s Ingress 如何自动根据主机名中的二级域名匹配 service

应用获取不到客户端真实 IP 地址的问题

详见博问 K8s 中 ASP.NET Core 应用获取不到客户端真实 IP 地址