参考:
https://kubernetes.github.io/ingress-nginx/examples/
https://github.com/kubernetes/ingress-nginx/tree/main/docs/examples
https://cloud.tencent.com/developer/article/1761376html
除非另有说明,在示例中使用的TLS秘密是2048位RSA密钥/证书对与任意选择的主机名,建立以下:node
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc" Generating a 2048 bit RSA private key ................+++ ................+++ writing new private key to 'tls.key' ----- $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret "tls-secret" created
注意:若是使用CA认证,以下所述,您将须要与CA签署服务器证书nginx
CA认证也被称为相互身份验证容许服务器和客户端经过一个共同的CA.验证对方的身份git
咱们有一个一般从证书颁发机构得到的 CA 证书,并使用它来签署咱们的服务器证书和客户端证书。 那么每次咱们要访问咱们的后端时,都必须经过客户端证书。github
这些说明基于如下博客(https://medium.com/@awkwardferny/configuring-certificate-based-mutual-authentication-with-kubernetes-ingress-nginx-20e7e38fdfca)json
生成 CA 密钥和证书:后端
openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority'
生成服务器密钥和证书并使用 CA 证书签名:api
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt
生成客户端密钥和证书并使用 CA 证书签名:安全
openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt
完成后,您能够继续按照此处(https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/#creating-certificate-secrets)的说明进行操做。服务器
全部须要测试 HTTP 服务的示例都使用标准的 http-svc pod,您能够按以下方式部署:
$ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml service "http-svc" created replicationcontroller "http-svc" created $ kubectl get po NAME READY STATUS RESTARTS AGE http-svc-p1t3t 1/1 Running 0 1d $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d
您能够经过暂时公开它来测试 HTTP 服务是否有效:
$ kubectl patch svc http-svc -p '{"spec":{"type": "LoadBalancer"}}' "http-svc" patched $ kubectl get svc http-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d $ kubectl describe svc http-svc Name: http-svc Namespace: default Labels: app=http-svc Selector: app=http-svc Type: LoadBalancer IP: 10.0.122.116 LoadBalancer Ingress: 108.59.87.136 Port: http 80/TCP NodePort: http 30301/TCP Endpoints: 10.180.1.6:8080 Session Affinity: None Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {service-controller } Normal Type ClusterIP -> LoadBalancer 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer $ curl 108.59.87.136 CLIENT VALUES: client_address=10.240.0.3 command=GET real path=/ query=nil request_version=1.1 request_uri=http://108.59.87.136:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* host=108.59.87.136 user-agent=curl/7.46.0 BODY: -no body in request- $ kubectl patch svc http-svc -p '{"spec":{"type": "NodePort"}}' "http-svc" patched
示例:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-test annotations: nginx.ingress.kubernetes.io/affinity: "cookie" nginx.ingress.kubernetes.io/session-cookie-name: "route" nginx.ingress.kubernetes.io/session-cookie-expires: "172800" nginx.ingress.kubernetes.io/session-cookie-max-age: "172800" spec: rules: - host: stickyingress.example.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: /
验证:
$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) Rules: Host Path Backends ---- ---- -------- stickyingress.example.com / nginx-service:80 (<none>) Annotations: affinity: cookie session-cookie-name: INGRESSCOOKIE session-cookie-expires: 172800 session-cookie-max-age: 172800 Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {nginx-ingress-controller } Normal CREATE default/nginx-test $ curl -I http://stickyingress.example.com HTTP/1.1 200 OK Server: nginx/1.11.9 Date: Fri, 10 Feb 2017 14:11:12 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Set-Cookie: INGRESSCOOKIE=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: "58875e6b-264" Accept-Ranges: bytes
$ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo
$ kubectl create secret generic basic-auth --from-file=auth secret "basic-auth" created
$ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK kind: Secret metadata: name: basic-auth namespace: default type: Opaque
echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-with-auth annotations: # type of authentication nginx.ingress.kubernetes.io/auth-type: basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo' spec: rules: - host: foo.bar.com http: paths: - path: / backend: serviceName: http-svc servicePort: 80 " | kubectl create -f -
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 > Host: foo.bar.com > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 05:27:23 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm="Authentication Required - foo" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor="white"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.10.0</center> </body> </html> * Connection #0 to host 10.2.29.4 left intact
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' > GET / HTTP/1.1 > Host: foo.bar.com > Authorization: Basic Zm9vOmJhcg== > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 06:05:26 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < CLIENT VALUES: client_address=10.2.29.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.43.0 x-request-id=e426c7829ef9f3b18d40730857c3eddb x-forwarded-for=10.2.29.1 x-forwarded-host=foo.bar.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.29.1 x-scheme=http BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request-
kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: # Enable client certificate authentication nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" # Create the secret containing the trusted ca certificates nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret" # Specify the verification depth in the client certificates chain nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" # Specify an error page to be redirected to verification errors nginx.ingress.kubernetes.io/auth-tls-error-page: "http://www.mysite.com/error-cert.html" # Specify if certificates are passed to upstream server nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true" name: nginx-test namespace: default spec: rules: - host: mydomain.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: / tls: - hosts: - mydomain.com secretName: tls-secret
$ kubectl create -f ingress.yaml ingress "external-auth" created $ kubectl get ing external-auth NAME HOSTS ADDRESS PORTS AGE external-auth external-auth-01.sample.com 172.17.4.99 80 13s $ kubectl get ing external-auth -o yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd creationTimestamp: 2016-10-03T13:50:35Z generation: 1 name: external-auth namespace: default resourceVersion: "2068378" selfLink: /apis/networking/v1beta1/namespaces/default/ingresses/external-auth uid: 5c388f1d-8970-11e6-9004-080027d2dc94 spec: rules: - host: external-auth-01.sample.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: / status: loadBalancer: ingress: - ip: 172.17.4.99 $
... metadata: name: application annotations: nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth" nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri" ...
适用于特定Ingress:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-configuration-snippet annotations: nginx.ingress.kubernetes.io/configuration-snippet: | more_set_headers "Request-Id: $req_id"; spec: rules: - host: custom.configuration.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: /
使用于全部Ingress:
apiVersion: v1 data: X-Different-Name: "true" X-Request-Start: t=${msec} X-Using-Nginx-Controller: "true" kind: ConfigMap metadata: name: custom-headers namespace: ingress-nginx
apiVersion: v1 data: proxy-set-headers: "ingress-nginx/custom-headers" kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx
使用 ConfigMap 能够自定义 NGINX 配置。
例如,若是咱们想更改超时,咱们须要建立一个 ConfigMap:
$ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: "10" proxy-read-timeout: "120" proxy-send-timeout: "120" kind: ConfigMap metadata: name: ingress-nginx-controller
若是 Configmap 更新,NGINX 将使用新配置从新加载。
调优参数:
net.core.somaxconn = 32768 net.ipv4.ip_local_port_range 1024 65000
此示例旨在演示使用 Init Container 来调整 sysctl 默认值 kubectl patch
kubectl patch deployment -n ingress-nginx nginx-ingress-controller \ --patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/sysctl/patch.json)"
变化:
在 NGINX 博客的一篇文章中(https://www.nginx.com/blog/tuning-nginx/),能够看到对这些变化的解释。
可使用如下注释控制重写:
从版本 0.22.0 开始,使用注解的入口定义 nginx.ingress.kubernetes.io/rewrite-target不向后兼容之前的版本。 在版本 0.22.0 及更高版本中,请求 URI 中须要传递到重写路径的任何子字符串都必须在 明肯定义 捕获组中 。
捕获的组 按时间顺序保存在编号的占位符中,格式为 $1, $2 ... $n. 这些占位符能够用做参数 rewrite-target注解。
建立带有重写注释的 Ingress 规则:
$ echo ' apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 name: rewrite namespace: default spec: rules: - host: rewrite.bar.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: /something(/|$)(.*) ' | kubectl create -f -
在这个入口定义中,任何被捕获的字符 (.*)将分配给占位符 $2,而后将其用做参数 rewrite-target注解。
例如,上面的入口定义将致使如下重写:
建立带有 app-root 注释的 Ingress 规则:
$ echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/app-root: /app1 name: approot namespace: default spec: rules: - host: approot.bar.com http: paths: - backend: serviceName: http-svc servicePort: 80 path: / " | kubectl create -f -
检查重写是否有效
$ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT Content-Type: text/html Content-Length: 162 Location: http://stickyingress.example.com/app1 Connection: keep-alive
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: todo namespace: default annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/app-root: /app/ nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/configuration-snippet: | ## 借助 ingress-nginx 中的 configuration-snippet 来对静态资源作一次跳转 rewrite ^(/app)$ $1/ redirect; ## 应用在最后添加一个 / 这样的 slash rewrite ^/stylesheets/(.*)$ /app/stylesheets/$1 redirect; rewrite ^/images/(.*)$ /app/images/$1 redirect; spec: rules: - host: todo.example.com http: paths: - backend: serviceName: todo servicePort: 3000 path: /app(/|$)(.*)
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-test spec: tls: - hosts: - foo.bar.com # This assumes tls-secret exists and the SSL # certificate contains a CN for foo.bar.com secretName: tls-secret rules: - host: foo.bar.com http: paths: - path: / backend: # This assumes http-svc exists and routes to healthy endpoints serviceName: http-svc servicePort: 80
$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) TLS: tls-secret terminates Rules: Host Path Backends ---- ---- -------- * http-svc:80 (<none>) Annotations: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {nginx-ingress-controller } Normal CREATE default/nginx-test 7s 7s 1 {nginx-ingress-controller } Normal UPDATE default/nginx-test 7s 7s 1 {nginx-ingress-controller } Normal CREATE ip: 104.198.183.6 7s 7s 1 {nginx-ingress-controller } Warning MAPPING Ingress rule 'default/nginx-test' contains no path definition. Assuming / $ curl 104.198.183.6 -L curl: (60) SSL certificate problem: self signed certificate More details here: http://curl.haxx.se/docs/sslcerts.html $ curl 104.198.183.6 -Lk CLIENT VALUES: client_address=10.240.0.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://35.186.221.137:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=Keep-Alive host=35.186.221.137 user-agent=curl/7.46.0 via=1.1 google x-cloud-trace-context=f708ea7e369d4514fc90d51d7e27e91d/13322322294276298106 x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY:
在今天的大多数集群中,默认状况下,全部资源(例如 Deployments 和 ReplicatSets)都具备建立 pod 的权限。 然而,Kubernetes 提供了一种更细粒度的受权策略,称为 Pod 安全策略 (PSP) 。
PSP 容许集群全部者定义每一个对象的权限,例如建立 pod。 若是您在集群上启用了 PSP,而且您部署了 ingress-nginx,您将须要为 Deployment 提供建立 pod 的权限。
在应用任何对象以前,首先经过运行来应用 PSP 权限:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/psp/psp.yaml
注意:必须在建立 Deployment 和 ReplicaSet 以前授予 PSP 权限。
在平常工做中咱们常常须要对服务进行版本更新升级,因此咱们常常会使用到滚动升级、蓝绿发布、灰度发布等不一样的发布操做。而 ingress-nginx 支持经过 Annotations 配置来实现不一样场景下的灰度发布和测试,能够知足金丝雀发布、蓝绿部署与 A/B 测试等业务场景。ingress-nginx 的 Annotations 支持如下 4 种 Canary 规则:
nginx.ingress.kubernetes.io/canary-by-header:基于 Request Header 的流量切分,适用于灰度发布以及 A/B 测试。当 Request Header 设置为 always 时,请求将会被一直发送到 Canary 版本;当 Request Header 设置为 never时,请求不会被发送到 Canary 入口;对于任何其余 Header 值,将忽略 Header,并经过优先级将请求与其余金丝雀规则进行优先级的比较。
nginx.ingress.kubernetes.io/canary-by-header-value:要匹配的 Request Header 的值,用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务。当 Request Header 设置为此值时,它将被路由到 Canary 入口。该规则容许用户自定义 Request Header 的值,必须与上一个 annotation (即:canary-by-header) 一块儿使用。
nginx.ingress.kubernetes.io/canary-weight:基于服务权重的流量切分,适用于蓝绿部署,权重范围 0 - 100 按百分比将请求路由到 Canary Ingress 中指定的服务。权重为 0 意味着该金丝雀规则不会向 Canary 入口的服务发送任何请求,权重为 100 意味着全部请求都将被发送到 Canary 入口。
nginx.ingress.kubernetes.io/canary-by-cookie:基于 cookie 的流量切分,适用于灰度发布与 A/B 测试。用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务的cookie。当 cookie 值设置为 always 时,它将被路由到 Canary 入口;当 cookie 值设置为 never 时,请求不会被发送到 Canary 入口;对于任何其余值,将忽略 cookie 并将请求与其余金丝雀规则进行优先级的比较。
须要注意的是金丝雀规则按优先顺序进行排序:canary-by-header - > canary-by-cookie - > canary-weight
总的来讲能够把以上的四个 annotation 规则划分为如下两类:
建立一个用于 production 环境访问的 Ingress 资源对象:
# production-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: production annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: echo.example.com http: paths: - backend: serviceName: production servicePort: 80
命令行访问应用:
curl echo.example.com
建立一个基于权重的 Canary 版本的应用路由 Ingress 对象。
# canary-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: canary annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制,首先须要启用 Canary nginx.ingress.kubernetes.io/canary-weight: "30" # 分配30%流量到当前Canary版本 spec: rules: - host: echo.example.com http: paths: - backend: serviceName: canary servicePort: 80
命令行访问应用:
curl -s echo.example.com
在上面的 Canary 版本的 Ingress 对象中新增一条 annotation 配置 nginx.ingress.kubernetes.io/canary-by-header: canary(这里的 value 能够是任意值),使当前的 Ingress 实现基于 Request Header 进行流量切分,因为 canary-by-header 的优先级大于 canary-weight,因此会忽略原有的 canary-weight 的规则。
annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制,首先须要启用 Canary nginx.ingress.kubernetes.io/canary-by-header: canary # 基于header的流量切分 nginx.ingress.kubernetes.io/canary-weight: "30" # 会被忽略,由于配置了 canary-by-header Canary版本
注意:当 Request Header 设置为 never 或 always 时,请求将不会或一直被发送到 Canary 版本,对于任何其余 Header 值,将忽略 Header,并经过优先级将请求与其余 Canary 规则进行优先级的比较。
命令行访问应用:
curl -s -H "canary: never" echo.example.com curl -s -H "canary: always" echo.example.com curl -s -H "canary: other-value" echo.example.com
当咱们请求设置的 Header 值为 canary: other-value 时,ingress-nginx 会经过优先级将请求与其余 Canary 规则进行优先级的比较,咱们这里也就会进入 canary-weight: "30" 这个规则去。
这个时候咱们能够在上一个 annotation (即 canary-by-header)的基础上添加一条 nginx.ingress.kubernetes.io/canary-by-header-value: user-value 这样的规则,就能够将请求路由到 Canary Ingress 中指定的服务了。
annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制,首先须要启用 Canary nginx.ingress.kubernetes.io/canary-by-header-value: user-value nginx.ingress.kubernetes.io/canary-by-header: canary # 基于header的流量切分 nginx.ingress.kubernetes.io/canary-weight: "30" # 分配30%流量到当前Canary版本
一样更新 Ingress 对象后,从新访问应用,当 Request Header 知足 canary: user-value 时,全部请求就会被路由到 Canary 版本。
curl -s -H "canary: user-value" echo.example.com
一样咱们更新 Canary 版本的 Ingress 资源对象,采用基于 Cookie 来进行流量切分:
annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" # 要开启灰度发布机制,首先须要启用 Canary nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing" # 基于 cookie nginx.ingress.kubernetes.io/canary-weight: "30" # 会被忽略,由于配置了 canary-by-cookie
更新上面的 Ingress 资源对象后,咱们在请求中设置一个 users_from_Beijing=always 的 Cookie 值,再次访问应用的域名:
curl -s -b "users_from_Beijing=always" echo.example.com
能够看到应用都被路由到了 Canary 版本的应用中去了,若是咱们将这个 Cookie 值设置为 never,则不会路由到 Canary 应用中。