k8s使用kube-router网络插件并监控流量状态

时间  2019-11-24
标签 k8s 使用 kube router 网络 插件 监控 流量 状态 栏目 系统网络 繁體版
原文   原文链接

简介

kube-router是一个新的k8s的网络插件，使用lvs作服务的代理及负载均衡，使用iptables来作网络的隔离策略。部署简单，只须要在每一个节点部署一个daemonset便可，高性能，易维护。支持pod间通讯，以及服务的代理。node

安装

# 本次实验从新建立了集群，使用以前测试其余网络插件的集群环境没有成功
# 多是因为环境干扰，实验时须要注意

# 建立kube-router目录下载相关文件
mkdir kube-router && cd kube-router
wget https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml
wget https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter-all-features.yaml

# 如下两种部署方式任选其一

# 1. 只启用 pod网络通讯，网络隔离策略 功能
kubectl apply -f kubeadm-kuberouter.yaml

# 2. 启用 pod网络通讯，网络隔离策略，服务代理 全部功能
# 删除kube-proxy和其以前配置的服务代理
kubectl apply -f kubeadm-kuberouter-all-features.yaml
kubectl -n kube-system delete ds kube-proxy

# 在每一个节点上执行
docker run --privileged --net=host registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.10.2 kube-proxy --cleanup

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system
复制代码

测试

# 启动用于测试的deployment
kubectl run nginx --replicas=2 --image=nginx:alpine --port=80
kubectl expose deployment nginx --type=NodePort --name=example-service-nodeport
kubectl expose deployment nginx --name=example-service

# dns及访问测试
kubectl run curl --image=radial/busyboxplus:curl -i --tty
nslookup kubernetes
nslookup example-service
curl example-service

# 清理
kubectl delete svc example-service example-service-nodeport
kubectl delete deploy nginx curl
复制代码

监控相关数据并可视化

从新部署kube-router

# 修改yml文件
cp kubeadm-kuberouter-all-features.yaml kubeadm-kuberouter-all-features.yaml.ori
vim kubeadm-kuberouter-all-features.yaml
...
spec:
  template:
    metadata:
      labels:
        k8s-app: kube-router
        tier: node
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
		# 添加以下参数，让prometheus收集数据
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: kube-router
      serviceAccount: kube-router
      containers:
      - name: kube-router
        image: cloudnativelabs/kube-router
        imagePullPolicy: Always
        args:
		# 添加以下参数开启metrics
        - --metrics-path=/metrics
        - --metrics-port=8080
        - --run-router=true
        - --run-firewall=true
        - --run-service-proxy=true
        - --kubeconfig=/var/lib/kube-router/kubeconfig
...

# 从新部署
kubectl delete ds kube-router -n kube-system
kubectl apply -f kubeadm-kuberouter-all-features.yaml

# 测试获取metrics
curl http://127.0.0.1:8080/metrics
复制代码

部署prometheus

复制以下内容到prometheus.yml文件nginx

---
apiVersion: v1
kind: ConfigMap
metadata:
 name: prometheus
 namespace: kube-system
data:
  prometheus.yml: |-
 global:
 scrape_interval: 15s
 scrape_configs:

    # scrape config for API servers
 - job_name: 'kubernetes-apiservers'
 kubernetes_sd_configs:
 - role: endpoints
 scheme: https
 tls_config:
 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 relabel_configs:
 - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
 action: keep
 regex: default;kubernetes;https

    # scrape config for nodes (kubelet)
 - job_name: 'kubernetes-nodes'
 scheme: https
 tls_config:
 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 kubernetes_sd_configs:
 - role: node
 relabel_configs:
 - action: labelmap
 regex: __meta_kubernetes_node_label_(.+)
 - target_label: __address__
 replacement: kubernetes.default.svc:443
 - source_labels: [__meta_kubernetes_node_name]
 regex: (.+)
 target_label: __metrics_path__
 replacement: /api/v1/nodes/${1}/proxy/metrics

    # Scrape config for Kubelet cAdvisor.
    #
    # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
    # (those whose names begin with 'container_') have been removed from the
    # Kubelet metrics endpoint. This job scrapes the cAdvisor endpoint to
    # retrieve those metrics.
    #
    # In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
    # HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
    # in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
    # the --cadvisor-port=0 Kubelet flag).
    #
    # This job is not necessary and should be removed in Kubernetes 1.6 and
    # earlier versions, or it will cause the metrics to be scraped twice.
 - job_name: 'kubernetes-cadvisor'
 scheme: https
 tls_config:
 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 kubernetes_sd_configs:
 - role: node
 relabel_configs:
 - action: labelmap
 regex: __meta_kubernetes_node_label_(.+)
 - target_label: __address__
 replacement: kubernetes.default.svc:443
 - source_labels: [__meta_kubernetes_node_name]
 regex: (.+)
 target_label: __metrics_path__
 replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

    # scrape config for service endpoints.
 - job_name: 'kubernetes-service-endpoints'
 kubernetes_sd_configs:
 - role: endpoints
 relabel_configs:
 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
 action: keep
 regex: true
 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
 action: replace
 target_label: __scheme__
 regex: (https?)
 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
 action: replace
 target_label: __metrics_path__
 regex: (.+)
 - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
 action: replace
 target_label: __address__
 regex: ([^:]+)(?::\d+)?;(\d+)
 replacement: $1:$2
 - action: labelmap
 regex: __meta_kubernetes_service_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: kubernetes_namespace
 - source_labels: [__meta_kubernetes_service_name]
 action: replace
 target_label: kubernetes_name

    # Example scrape config for pods
 - job_name: 'kubernetes-pods'
 kubernetes_sd_configs:
 - role: pod

 relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
 action: keep
 regex: true
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
 action: replace
 target_label: __metrics_path__
 regex: (.+)
 - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
 action: replace
 regex: ([^:]+)(?::\d+)?;(\d+)
 replacement: $1:$2
 target_label: __address__
 - action: labelmap
 regex: __meta_kubernetes_pod_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: namespace
 - source_labels: [__meta_kubernetes_pod_name]
 action: replace
 target_label: pod_name

---
apiVersion: v1
kind: Service
metadata:
 annotations:
    prometheus.io/scrape: 'true'
 labels:
 name: prometheus
 name: prometheus
 namespace: kube-system
spec:
 selector:
 app: prometheus
 type: NodePort
 ports:
 - name: prometheus
 protocol: TCP
 port: 9090
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: prometheus
 namespace: kube-system
spec:
 replicas: 1
 selector:
 matchLabels:
 app: prometheus
 template:
 metadata:
 name: prometheus
 labels:
 app: prometheus
 annotations:
        sidecar.istio.io/inject: "false"
 spec:
 serviceAccountName: prometheus
 containers:
 - name: prometheus
 image: docker.io/prom/prometheus:v2.2.1
 imagePullPolicy: IfNotPresent
 args:
 - '--storage.tsdb.retention=6h'
 - '--config.file=/etc/prometheus/prometheus.yml'
 ports:
 - name: web
 containerPort: 9090
 volumeMounts:
 - name: config-volume
 mountPath: /etc/prometheus
 volumes:
 - name: config-volume
 configMap:
 name: prometheus
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: prometheus
 namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: prometheus
rules:
- apiGroups: [""]
 resources:
 - nodes
 - services
 - endpoints
 - pods
 - nodes/proxy
 verbs: ["get", "list", "watch"]
- apiGroups: [""]
 resources:
 - configmaps
 verbs: ["get"]
- nonResourceURLs: ["/metrics"]
 verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: prometheus
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: prometheus
subjects:
- kind: ServiceAccount
 name: prometheus
 namespace: kube-system
---
复制代码

部署测试git

# 部署
kubectl apply -f prometheus.yml

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system

# 访问prometheus
# 输入 kube_router 关键字查找 看有无提示出现
prometheusNodePort=$(kubectl get svc -n kube-system | grep prometheus | awk '{print $5}' | cut -d '/' -f 1 | cut -d ':' -f 2)
nodeName=$(kubectl get no | grep '<none>' | head -1 | awk '{print $1}')
nodeIP=$(ping -c 1 $nodeName | grep PING | awk '{print $3}' | tr -d '()')
echo "http://$nodeIP:"$prometheusNodePort
复制代码

部署grafana

复制以下内容到grafana.yml文件github

---
apiVersion: v1
kind: Service
metadata:
 name: grafana
 namespace: kube-system
spec:
 type: NodePort
 ports:
 - port: 3000
 protocol: TCP
 name: http
 selector:
 app: grafana
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: grafana
 namespace: kube-system
spec:
 replicas: 1
 template:
 metadata:
 labels:
 app: grafana
 spec:
 serviceAccountName: grafana
 containers:
 - name: grafana
 image: grafana/grafana
 imagePullPolicy: IfNotPresent
 ports:
 - containerPort: 3000
 volumeMounts:
 - mountPath: /var/lib/grafana
 name: grafana-data
 volumes:
 - name: grafana-data
 emptyDir: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: grafana
 namespace: kube-system
---
复制代码

部署测试web

# 部署
kubectl apply -f grafana.yml

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system


# 访问grafana
grafanaNodePort=$(kubectl get svc -n kube-system | grep grafana | awk '{print $5}' | cut -d '/' -f 1 | cut -d ':' -f 2)
nodeName=$(kubectl get no | grep '<none>' | head -1 | awk '{print $1}')
nodeIP=$(ping -c 1 $nodeName | grep PING | awk '{print $3}' | tr -d '()')
echo "http://$nodeIP:"$grafanaNodePort

# 默认用户密码
admin/admin
复制代码

导入并查看dashboard

# 下载官方dashboard的json文件
wget https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/dashboard/kube-router.json
复制代码

建立名为Prometheus类型也为Prometheus的数据源，链接地址为http://prometheus:9090/docker

选择刚刚下载的json文件导入dashboardjson

查看dashboardvim

相关文章
相关标签/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公众号
   欢迎关注本站公众号,获取更多信息
联系我们 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程