容器编排系统K8s之crd资源

时间 2021-01-17

标签 html node mysql linux nginx git 程序员 github sql mongodb 栏目 HTML 繁體版

原文原文链接

　　前文咱们了解了k8s节点污点和pod的对节点污点容忍度相关话题，回顾请参考：http://www.javashuo.com/article/p-cymldyou-nz.html；今天咱们来聊一下扩展k8s相关话题；html

　　k8s上建立资源对象的过程node

　　咱们知道在k8s上，资源是有类型的，不一样类型的资源，其定义方式和使用的字段各不相同；用户建立一个资源，实际上就是把k8s抽象的资源作实例化，即把k8s抽象出来的资源，经过资源清单赋值，建立出来的对象就是咱们实例化对应类型资源的结果；用户建立一个资源，首先会把请求发送给apiserver，经过apiserver的认证、受权、准入控制之后，对应建立资源的定义就存放在etcd中，控制器经过watch机制监视apiserver上的资源变更，经过对应资源变更事件触发对应类型资源的控制器将对应资源建立出来，并经过控制器内部的和解循环监控着对应资源状态是否和用户定义的指望状态同样，若是发现不同，内部和解循环就会被触发，对应控制器会向apiserver发起建立资源的请求，将对应资源重建，让对应资源的状态始终知足用户指望的状态；从上述的过程来看，用户建立一个资源分两个步骤，第一步是将对应请求发送给apiserver，经过apiserver把对应资源定义的信息存放在etcd中；第二个步骤是对应资源类型的控制器经过apiserver从etcd中读取对应资源的定义，将其建立出来；对于etcd来讲，它本是就一个kv数据库，能够存储任意类型的kv数据，但在在k8s上，apiserver将不一样类型的资源定义抽象成不一样的资源，使得用户建立对应资源，必须是知足对应类型资源定义的规范，而后将规范的定义存放在etcd中；简单讲apiserver就是把用户存入etcd中的数据作了一层抽象，使得用户不可以随意将任意数据存储到etcd中，存入etcd中的数据必须是知足对应apiserver接口定义的规范；这就比如咱们在使用mysql数据库时，必须遵照对应库中的对应表的定义；mysql

　　在k8s上建立自定义资源类型linux

　　在k8s上，资源的类型有不少，好比pod，service，PersistentVolume，PersistentVolumeClaim等等，这些都是一些基础的资源类型；咱们要建立某种资源，直接使用对应的资源类型，实例化一个对象便可；假如咱们要在k8s上建立一个集群，咱们是否是能够直接使用某种类型的资源，实例化一个集群对象呢？理论上是能够的，可是前提是对应k8s上有对应类型的资源；有对应类型的资源，用户就能够把对应建立资源的定义存放在etcd中；除了有对应类型的资源，咱们还须要有对应的控制器将对应资源建立出来；这样一来对于不一样集群或应用来讲，其组织方式和逻辑都不一样，使用的资源类型和控制器也有所不一样；用户要想实例化更高级的资源来，就必须本身手动实现定义其资源类型，将对应资源类型实例化为对象；除此以外必要时还须要本身实现对应资源的控制器；简单讲用户想要实现更高级的资源类型，就必须扩展示有k8s的资源类型和控制器；nginx

　　在k8s上扩展资源类型的方式有三种，第一种是crd，crd是k8s内建的资源类型，该类型资源主要用来建立用户自定义资源类型的资源；即经过crd资源，能够将用户自定义资源类型转换为k8s上资源类型；第二种是自定义apiserver；这种方式要比第一种方式要复杂一点，须要用户手动开发程序实现对应功能的apiserver，让其用户建立自定义类型资源可以经过自定义apiserver实现；第三种方式就是修改现有k8sapiserver，让其支持对应用户自定义资源类型；git

　　自定义控制器程序员

　　自定义资源类型咱们可使用crd资源实现，也可使用自定义apiserver或修改原有apiserver代码实现，可是只有资源类型是不可以让对应自定义类型资源实例化为一个自定义资源对象，只有自定义资源类型，用户建立对应资源类型的资源对象时，只能把对应资源类型的定义信息写入到etcd中，它不能真正的跑起来，要想真正的跑起来，咱们还须要一个自定义控制器，专门负责监听对应的资源类型的资源变化，将对应资源实例化为对应k8s上的资源对象；固然不是全部的自定义类型的资源都须要自定义控制器，若是对应自定义类型资源调用了底层基础控制器来管控对应自定义资源，那么对应自定义类型资源就不须要使用自定义控制器；咱们知道控制器是k8s上的一个重要组件，它的工做逻辑是注册监听在apiserver上对应类型的资源变更，若是对应资源状态不知足用户指望状态，它就会根据内部的和解循环来请求apiserver将对应类型资源的定义发送给它，而后根据资源定义来重建对应的资源，让其状态始终和用户指望的状态保持一致；自定义控制器也是一样的逻辑，使用自定义控制器的目的也是让对应自定义类型资源可以被自定义控制器监听，一旦对应资源发生变更，它可以将其在k8s上建立出来，并一直保持和用户指望的状态吻合；自定义控制器和自定义资源类型能够分开实现，也能够合并在一块儿实现，即自定义控制器程序可以自动建立crd资源，让其对应自定义类型资源可以被k8s识别并将其建立出来；具体是分开实现仍是合并在一块儿实现，取决开发自定义控制器程序员；github

　　crd资源定义帮助sql

[root@master01 ~]# kubectl explain crd
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

DESCRIPTION:
     CustomResourceDefinition represents a resource that should be exposed on
     the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>

   spec <Object> -required-
     spec describes how the user wants the resources to appear

   status       <Object>
     status indicates the actual state of the CustomResourceDefinition

[root@master01 ~]#

　　提示：crd资源是k8s上的标准资源之一，它的定义主要有apiVersion，kind，metadata，spec和status；其中kind类型为CustomResourceDefinition，apiVersion是apiextensions.k8s.io/v1；这两个是固定格式；spec字段是用来定义对应指定资源类型资源的相关属性；mongodb

　　crd.spec字段说明

[root@master01 ~]# kubectl explain crd.spec
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

RESOURCE: spec <Object>

DESCRIPTION:
     spec describes how the user wants the resources to appear

     CustomResourceDefinitionSpec describes how a user wants their resource to
     appear

FIELDS:
   conversion   <Object>
     conversion defines conversion settings for the CRD.

   group        <string> -required-
     group is the API group of the defined custom resource. The custom resources
     are served under `/apis/<group>/...`. Must match the name of the
     CustomResourceDefinition (in the form `<names.plural>.<group>`).

   names        <Object> -required-
     names specify the resource and kind names for the custom resource.

   preserveUnknownFields        <boolean>
     preserveUnknownFields indicates that object fields which are not specified
     in the OpenAPI schema should be preserved when persisting to storage.
     apiVersion, kind, metadata and known fields inside metadata are always
     preserved. This field is deprecated in favor of setting
     `x-preserve-unknown-fields` to true in
     `spec.versions[*].schema.openAPIV3Schema`. See
     https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
     for details.

   scope        <string> -required-
     scope indicates whether the defined custom resource is cluster- or
     namespace-scoped. Allowed values are `Cluster` and `Namespaced`.

   versions     <[]Object> -required-
     versions is the list of all API versions of the defined custom resource.
     Version names are used to compute the order in which served versions are
     listed in API discovery. If the version string is "kube-like", it will sort
     above non "kube-like" version strings, which are ordered lexicographically.
     "Kube-like" versions start with a "v", then are followed by a number (the
     major version), then optionally the string "alpha" or "beta" and another
     number (the minor version). These are sorted first by GA > beta > alpha
     (where GA is a version with no suffix such as beta or alpha), and then by
     comparing major version, then minor version. An example sorted list of
     versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
     foo1, foo10.

[root@master01 ~]#

　　提示：crd.spec中group字段使用来描述对应自定义类型资源的群组名称，其值为字符串；names字段是用来描述自定义类型资源对应类型，名称等等，其值为一个对象；scope字段用来定义对应自定义资源是那个级别的资源；该字段的值只能为Cluster或Namespaced；versions字段是用来指定对应自定义资源的版本，以及对应类型资源的属性字段等信息，该字段为一个列表对象；

　　示例：定义一个自定义类型资源

[root@master01 ~]# cat crontab-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # 名字必需与下面的 spec 字段匹配，而且格式为 '<名称的复数形式>.<组名>'
  name: crontabs.stable.example.com
spec:
  # 组名称，用于 REST API: /apis/<组>/<版本>
  group: stable.example.com
  # 列举此 CustomResourceDefinition 所支持的版本
  versions:
    - name: v1
      # 每一个版本均可以经过 served 标志来独立启用或禁止
      served: true
      # 其中一个且只有一个版本必需被标记为存储版本
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # 能够是 Namespaced 或 Cluster
  scope: Namespaced
  names:
    # 名称的复数形式，用于 URL：/apis/<组>/<版本>/<名称的复数形式>
    plural: crontabs
    # 名称的单数形式，做为命令行使用时和显示时的别名
    singular: crontab
    # kind 一般是单数形式的驼峰编码（CamelCased）形式。你的资源清单会使用这一形式。
    kind: CronTab
    # shortNames 容许你在命令行使用较短的字符串来匹配资源
    shortNames:
    - ct
[root@master01 ~]#

　　应用资源清单前使用kubectl get crontab

[root@master01 ~]# kubectl get crontab
error: the server doesn't have a resource type "crontab"
[root@master01 ~]#

　　提示：在没有应用资源清单前使用kubectl get crontab，它会提示咱们所没有类型为crontab的资源

　　应用资源清单

[root@master01 ~]# kubectl apply -f crontab-crd.yaml
customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created
[root@master01 ~]# kubectl get crontab
No resources found in default namespace.
[root@master01 ~]#

　　提示：应用资源清单后，再次使用kubectl get crontab就没有报错了，只是提示在default名称空间没有对应类型资源；

　　查看crd

[root@master01 ~]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]#

　　查看详情

[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]# kubectl describe crd/crontabs.stable.example.com
Name:         crontabs.stable.example.com
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2021-01-12T12:39:00Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:conditions:
    Manager:      kube-apiserver
    Operation:    Update
    Time:         2021-01-12T12:39:00Z
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:scope:
        f:versions:
      f:status:
        f:storedVersions:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:39:00Z
  Resource Version:  805506
  UID:               b92a90f4-c953-4876-a496-030c9ba023fd
Spec:
  Conversion:
    Strategy:  None
  Group:       stable.example.com
  Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Scope:       Namespaced
  Versions:
    Name:  v1
    Schema:
      openAPIV3Schema:
        Properties:
          Spec:
            Properties:
              Cron Spec:
                Type:  string
              Image:
                Type:  string
              Replicas:
                Type:  integer
            Type:      object
        Type:          object
    Served:            true
    Storage:           true
Status:
  Accepted Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Conditions:
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>
[root@master01 ~]#

　　使用自定义资源类型crontab建立资源

[root@master01 ~]# cat my-crontab.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
[root@master01 ~]#

　　提示：上述资源清单表示建立一个类型为Crontab的资源，该资源的群组版本为stable.example.com/v1;

　　应用资源清单

[root@master01 ~]# kubectl apply -f my-crontab.yaml
crontab.stable.example.com/my-new-cron-object created
[root@master01 ~]# kubectl get ct
NAME                 AGE
my-new-cron-object   5s
[root@master01 ~]# kubectl describe ct/my-new-cron-object
Name:         my-new-cron-object
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  stable.example.com/v1
Kind:         CronTab
Metadata:
  Creation Timestamp:  2021-01-12T12:45:29Z
  Generation:          1
  Managed Fields:
    API Version:  stable.example.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:cronSpec:
        f:image:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:45:29Z
  Resource Version:  806182
  UID:               31a88a3d-fa99-42b8-80f6-3e4559efdc40
Spec:
  Cron Spec:  * * * * */5
  Image:      my-awesome-cron-image
Events:       <none>
[root@master01 ~]#

　　提示：能够看到对应类型资源已经建立成功；以上示例只是单纯的crd的使用示例，没有任何实质的做用；

　　部署自定义控制器

　　示例：部署mongodb-aperator

　　一、克隆项目

[root@master01 ~]# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git
Cloning into 'mongodb-kubernetes-operator'...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411
Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done.
Resolving deltas: 100% (2621/2621), done.
[root@master01 ~]#

　　二、建立名称空间mongodb，并进入到mongodb-kubernetes-operator目录应用crd资源，建立自定义资源类型

[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb
namespace/mongodb created
[root@master01 mongodb-kubernetes-operator]# kubectl get ns
NAME                   STATUS   AGE
default                Active   35d
ingress-nginx          Active   22d
kube-node-lease        Active   35d
kube-public            Active   35d
kube-system            Active   35d
kubernetes-dashboard   Active   11d
mongodb                Active   4s
[root@master01 mongodb-kubernetes-operator]# ls
agent    build  deploy     docs    go.sum      pkg        release.json      scripts  testdata  version
APACHE2  cmd    dev_notes  go.mod  LICENSE.md  README.md  requirements.txt  test     tools.go
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created
[root@master01 mongodb-kubernetes-operator]#

　　验证：查看mongodb类型资源是否已经建立成功？

[root@master01 mongodb-kubernetes-operator]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
mongodb.mongodb.com                                   2021-01-13T06:38:22Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com
NAME                  CREATED AT
mongodb.mongodb.com   2021-01-13T06:38:22Z
[root@master01 mongodb-kubernetes-operator]#

　　三、安装operator

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb
deployment.apps/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
serviceaccount/mongodb-kubernetes-operator created
[root@master01 mongodb-kubernetes-operator]#

　　提示：mongodb-kubernetes-operator这个项目是将自定义控制器和自定义资源类型分开实现的；其operator只负责建立和监听对应资源类型的变化，在资源有变化时，实例化为对应资源对象，并保持对应资源对象状态吻合用户指望状态；上述四个清单中主要是建立了一个sa帐户，并对对应的sa用户受权；

　　operator.yaml内容

[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-kubernetes-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb-kubernetes-operator
  template:
    metadata:
      labels:
        name: mongodb-kubernetes-operator
    spec:
      serviceAccountName: mongodb-kubernetes-operator
      containers:
        - name: mongodb-kubernetes-operator
          image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0
          command:
            - mongodb-kubernetes-operator
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "mongodb-kubernetes-operator"
            - name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments
              value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1
            - name: VERSION_UPGRADE_HOOK_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
            - name: MONGODB_IMAGE
              value: "library/mongo"
            - name: MONGODB_REPO_URL
              value: "registry.hub.docker.com"
[root@master01 mongodb-kubernetes-operator]#

　　提示：上述资源主要用deploy控制器运行对应自定义控制器为一个pod；

　　验证：查看operator是否正常运行

[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          26s
[root@master01 mongodb-kubernetes-operator]#

　　提示：可以看到operator正常运行，就表示operator已经安装成功；

　　验证：使用自定义资源类型建立一个mongodb 副本集集群

[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml 
---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: example-mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram

# the user credentials will be generated from this secret
# once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
  name: my-user-password
type: Opaque
stringData:
  password: 58LObjiMpxcjP1sMDW
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]#

　　应用清单

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              0/2     Pending   0          9s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          88s
[root@master01 mongodb-kubernetes-operator]#

　　提示：这里能够看到对应pod处于pending状态；

　　查看pod详细信息

[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events"
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  66s (x2 over 66s)  default-scheduler  0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
[root@master01 mongodb-kubernetes-operator]#

　　提示：这里提示没有能够用的pvc；

　　删除mongodb名称空间下pvc

[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
NAME                            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Pending                                                     92s
[root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb
persistentvolumeclaim "data-volume-example-mongodb-0" deleted
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 mongodb-kubernetes-operator]#

　　建立pv和pvc

[root@master01 mongodb-kubernetes-operator]# cd
[root@master01 ~]# cat pv-demo.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v1
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v1
    server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v2
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v2
    server: 192.168.0.99
---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v3
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v3
    server: 192.168.0.99
[root@master01 ~]#

　　应用清单建立pv

[root@master01 ~]# kubectl apply -f pv-demo.yaml
persistentvolume/nfs-pv-v1 created
persistentvolume/nfs-pv-v2 created
persistentvolume/nfs-pv-v3 created
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
[root@master01 ~]#

　　建立pvc清单

[root@master01 ~]# cat pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-0 
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-1
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-2
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
[root@master01 ~]#

　　应用清单建立pvc

[root@master01 ~]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 ~]# kubectl apply -f pvc-demo.yaml
persistentvolumeclaim/data-volume-example-mongodb-0 created
persistentvolumeclaim/data-volume-example-mongodb-1 created
persistentvolumeclaim/data-volume-example-mongodb-2 created
[root@master01 ~]# kubectl get pvc -n mongodb    
NAME                            STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Bound    nfs-pv-v1   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-1   Bound    nfs-pv-v2   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-2   Bound    nfs-pv-v3   1Gi        RWO,ROX,RWX                   6s
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-0                           102s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-1                           102s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-2                           102s
[root@master01 ~]#

　　提示：能够看到对应pvc和pv已经绑定好了；

　　验证：查看mongodb副本集集群是否跑起来了？

[root@master01 ~]# kubectl get pods -n mongodb
NAME                                           READY   STATUS            RESTARTS   AGE
example-mongodb-0                              2/2     Running           0          6m19s
example-mongodb-1                              0/2     PodInitializing   0          10s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running           0          7m38s
[root@master01 ~]# kubectl get pods -n mongodb -w
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          6m35s
example-mongodb-1                              1/2     Running   0          26s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          7m54s
example-mongodb-1                              2/2     Running   0          43s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Init:0/1   0          0s
example-mongodb-2                              0/2     Init:0/1   0          1s
example-mongodb-2                              0/2     Terminating   0          4s
example-mongodb-2                              0/2     Terminating   0          6s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Init:0/1      0          0s
example-mongodb-2                              0/2     Init:0/1      0          1s
example-mongodb-2                              0/2     PodInitializing   0          7s
example-mongodb-2                              1/2     Running           0          14s
example-mongodb-2                              2/2     Running           0          36s
^C[root@master01 ~]# kubectl get pods -n mongodb 
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          8m
example-mongodb-1                              2/2     Running   0          111s
example-mongodb-2                              2/2     Running   0          48s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          9m19s
[root@master01 ~]#

　　提示：能够看到对应的pod已经正常跑起来了；

　　验证：使用mongo 链接mongodbpod，看看对应副本集集群是否工做正常？

[root@master01 ~]# kubectl get pods -n mongodb -o wide
NAME                                           READY   STATUS    RESTARTS   AGE     IP             NODE             NOMINATED NODE   READINESS GATES
example-mongodb-0                              2/2     Running   0          9m12s   10.244.4.101   node04.k8s.org   <none>           <none>
example-mongodb-1                              2/2     Running   0          3m3s    10.244.2.130   node02.k8s.org   <none>           <none>
example-mongodb-2                              2/2     Running   0          2m      10.244.1.130   node01.k8s.org   <none>           <none>
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          10m     10.244.3.116   node03.k8s.org   <none>           <none>
[root@master01 ~]# mongo 10.244.4.101
MongoDB shell version v4.4.3
connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") }
MongoDB server version: 4.2.6
WARNING: shell and server versions do not match
example-mongodb:PRIMARY> show dbs
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
Error: Authentication failed.
0
example-mongodb:PRIMARY> use admin
switched to db admin
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
1
example-mongodb:PRIMARY> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
example-mongodb:PRIMARY> db.isMaster()
{
        "hosts" : [
                "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017"
        ],
        "setName" : "example-mongodb",
        "setVersion" : 1,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "electionId" : ObjectId("7fffffff0000000000000003"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "lastWriteDate" : ISODate("2021-01-13T06:52:21Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "majorityWriteDate" : ISODate("2021-01-13T06:52:21Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2021-01-13T06:52:27.873Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 153,
        "minWireVersion" : 0,
        "maxWireVersion" : 8,
        "readOnly" : false,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610520741, 1),
                "signature" : {
                        "hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="),
                        "keyId" : NumberLong("6917119940596072451")
                }
        },
        "operationTime" : Timestamp(1610520741, 1)
}
example-mongodb:PRIMARY>

　　提示：能够看到三个mongodb pod是副本集关系；其中example-mongodb-0是主节点，其余两个是从节点；

　　最后仍是说一下，我在作上面的实验中，虽然mongodb operator已经正常工做，可是用mongo这个客户端工具链接到主节点无法写数据，提示没有权限；但对应的用户在对应库是有读写权限的；在admin库下建立用户能提示用户添加成功，可是过几秒查询用户信息，发现用户不存在，不知道什么缘由，有知道的朋友麻烦告知下（博主邮箱：linux-1874@qq.com），博主将感激涕零。。