Kebernetes 学习总结(14) K8s Horizontal Pod Autoscaling

时间 2019-11-06

标签 kebernetes 学习总结 k8s horizontal pod autoscaling 繁體版

原文原文链接

1、概述
HPA全称Horizontal Pod Autoscaling，即pod的水平自动扩展。HPA的操做对象是RC、RS或Deployment对应的Pod，根据观察到的CPU实际使用量与用户的指望值进行比对，作出是否须要增减实例数量的决策。
它根据Pod当前系统的负载来自动水平扩容，若是系统负载超过预约值，就开始增长Pod的个数，若是低于某个值，就自动减小Pod的个数。目前K8S的HPA只能根据CPU和内存去度量系统的负载，并且目前还依赖heapster去收集CPU的使用状况。
HPA经过按期（按期轮询的时间经过–horizontal-pod-autoscaler-sync-period选项来设置，默认的时间为30秒）经过Status.PodSelector来查询pods的状态，得到pod的CPU使用率。而后，经过现有pods的CPU使用率的平均值（计算方式是最近的pod使用量除以设定的每一个Pod的CPU使用率限额，例最近一分钟从heapster中得到的平均值除以Pod的CPU limits值）跟目标使用率进行比较，而且在扩容时，还要遵循预先设定的副本数限制：MinReplicas <= Replicas <= MaxReplicas。
计算扩容后Pod的个数：sum(最近一分钟内某个Pod的CPU使用率/量的平均值)/CPU使用上限的整数+1
考虑到自动扩展的决策可能须要一段时间才会生效，甚至在短期内会引入一些噪声。例如当pod所须要的CPU负荷过大，从而运行一个新的pod进行分流，在建立过程当中，系统的CPU使用量可能会有一个攀升的过程。因此，在每一次做出决策后的一段时间内，将再也不进行扩展决策。对于ScaleUp而言，这个时间段为3分钟，Scaledown为5分钟。
HPA容许必定范围内的CPU使用量的不稳定，只有avg(CurrentPodsConsumption) / Target小于90%或者大于110%时才会触发扩容或缩容，避免频繁扩容、缩容形成颠簸。
为了简便，选用了相对比率（90%的CPU资源）而不是0.6个CPU core来描述扩容、缩容条件。若是选择使用绝对度量，用户须要保证目标（限额）要比请求使用的低，不然，过载的Pod未必可以消耗那么多，从而自动扩容永远不会被触发：假设设置CPU为1个核，那么这个pod只能使用1个核，可能Pod在过载的状况下也不能彻底利用这个核，因此扩容不会发生。在修改申请资源时，还有同时调整扩容的条件，好比将1个core变为1.2core，那么扩容条件应该同步改成1.2core，真是太麻烦了，与自动扩容的目标相悖。
例1：html

[root@docker79 ~]# kubectl get pods
No resources found.
[root@docker79 ~]# kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 --requests='cpu=50m,memory=256Mi' --limits='cpu=50m,memory=256Mi' --labels='app=myapp' --expose --port=80
service/myapp created
deployment.apps/myapp created
[root@docker79 ~]# kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
myapp-6985749785-l4c87   1/1       Running   0          15s
[root@docker79 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   20d
myapp        ClusterIP   10.100.44.181   <none>        80/TCP    20s
[root@docker79 ~]# kubectl autoscale deployment myapp --min=1 --max=4 --cpu-percent=40
horizontalpodautoscaler.autoscaling/myapp autoscaled
[root@docker79 ~]# kubectl get hpa
NAME      REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
myapp     Deployment/myapp   <unknown>/40%   1         4         0          6s
[root@docker79 ~]#
[root@docker79 ~]# kubectl patch svc myapp -p '{"spec":{"type":"NodePort"}}'
service/myapp patched
[root@docker79 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        20d
myapp        NodePort    10.100.44.181   <none>        80:30193/TCP   4m
[root@docker79 ~]#
[root@docker79 ~]# kubectl get hpa
NAME      REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
myapp     Deployment/myapp   0%/40%    1         4         1          6m
[root@docker79 ~]#

以上建立hap，并定义了CPU使用率达到40%时进行扩展，下面开启 ab压测，而后使其自动扩展docker

[root@docker79 ~]# ab -c 100 -n 50000 http://docker78:30193/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking docker78 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
......

[root@docker79 ~]# kubectl describe hpa
Name:                                                  myapp
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 17 Sep 2018 16:40:26 +0800
Reference:                                             Deployment/myapp
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  102% (51m) / 40%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 3 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  14s   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
[root@docker79 ~]#
[root@docker79 ~]# kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
myapp-6985749785-fvxbz   1/1       Running   0          1m
myapp-6985749785-l4c87   1/1       Running   0          14m
myapp-6985749785-xdmnw   1/1       Running   0          1m
[root@docker79 ~]#

例2，接上以manifests yaml的格式定义HorizontalPodAutoscaler ，并同时定义CPU、Memory资源的阀值，以便pod扩展：shell

[root@docker79 ~]# cd manifests/
[root@docker79 manifests]# mkdir hpa
[root@docker79 manifests]# cd hpa
[root@docker79 hpa]# vim hpa-demo-v2.yaml
[root@docker79 hpa]# cat hpa-demo-v2.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa-v2
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 55
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 50Mi
[root@docker79 hpa]#
[root@docker79 hpa]# kubectl delete hpa myapp
horizontalpodautoscaler.autoscaling "myapp" deleted
[root@docker79 hpa]#
[root@docker79 hpa]# kubectl apply -f hpa-demo-v2.yaml
horizontalpodautoscaler.autoscaling/myapp-hpa-v2 created
[root@docker79 hpa]# kubectl get hpa
NAME           REFERENCE          TARGETS                         MINPODS   MAXPODS   REPLICAS   AGE
myapp-hpa-v2   Deployment/myapp   <unknown>/50Mi, <unknown>/55%   1         5         0          6s
[root@docker79 hpa]#

再次开启压测apache

[root@docker79 ~]# ab -c 100 -n 30000 http://docker77:30193/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking docker77 (be patient)
Completed 3000 requests
Completed 6000 requests
......
[root@docker79 hpa]# kubectl describe hpa
Name:                                                  myapp-hpa-v2
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"myapp-hpa-v2","namespace":"default"},"spec":{...
CreationTimestamp:                                     Mon, 17 Sep 2018 17:30:53 +0800
Reference:                                             Deployment/myapp
Metrics:                                               ( current / target )
  resource memory on pods:                             3442688 / 50Mi
  resource cpu on pods  (as a percentage of request):  68% (34m) / 55%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       2 current / 3 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  9s    horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
[root@docker79 hpa]#

仔细观察hpa 的信息，发现已经扩展了两个new pods。
例3：
pods 能够自定义一些资源指标，并将其输出成Restful风格的API，并支持让prometheus 读取、认识这些自定义的指标值，而后hpa 就能够根据这些指标值进行扩展或收缩。以下所示：vim

[root@docker79 hpa]# cat hpa-demo-custom.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa-v2
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: http_requests
      targetAverageValue: 800m
[root@docker79 hpa]#

因为hpa 我使用还不是特别熟练，因此简单总结这么多，后续慢慢补充。你们勿喷。 api