本文介绍在k8s环境中进行jenkins server的部署和配置。Jenkins是一个开源的、功能强大的持续集成和持续构建工具,采用master和salve架构,咱们经过将jenkins集成环境部署在k8s集群中,能够实现jenkins slave按需建立、动态的伸缩。同时也提供了在k8s环境中应用的持续部署解决方案。java
一、编译jenkins server docker镜像,默认的jenkis镜像已包含jdk,版本为1.8.0_171node
# cat dockerfile FROM jenkins MAINTAINER ylw@fjhb.cn ENV MAVEN_HOME /usr/local/maven ENV JAVA_HOME /usr/local/java ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar ENV PATH ${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${PATH} COPY apache-maven-3.5.4 /usr/local/maven USER root RUN mkdir -p /usr/local/maven/repository && ln -s /usr/java/jdk1.8.0_171 /usr/local/java
# docker build -t harbor.59iedu.com/fjhb/jenkins:2018-08-12-v1 . # docker push harbor.59iedu.com/fjhb/jenkins:2018-08-12-v1
二、编译jenkins slave镜像
能够根据实际状况配置maven内网私服nexus,私服能够避免编译过程当中经过公网下载依赖的jar包,配置私服须要把对应的setting.xml文件打包到apache-maven-3.5.4/conf目录下;
libltdl.so.7文件的获取路径为操做系统路径/usr/lib64/libltdl.so.7(其实是个软连接,须要copy出来重命名)
slave.jar文件的获取路径为http://jenkins-server/jnlpJars/slave.jargit
# cat Dockerfile FROM openshift/base-centos7 MAINTAINER ylw@fjhb.cn COPY apache-maven-3.5.4 /usr/local/maven COPY jdk1.8.0_171 /usr/local/java COPY kubectl /usr/local/bin/kubectl COPY libltdl.so.7 /usr/lib64/libltdl.so.7 COPY slave.jar /usr/share/jenkins/slave.jar COPY jenkins-slave /usr/local/bin/jenkins-slave ENV HOME /home/jenkins ENV AGENT_WORKDIR=/home/jenkins/agent ENV JAVA_HOME /usr/local/java ENV MAVEN_HOME /usr/local/maven/ ENV CLASSPATH .:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar ENV PATH ${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${PATH} ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2" RUN chmod 755 /usr/share/jenkins \ && chmod 644 /usr/share/jenkins/slave.jar RUN mkdir -p /home/jenkins/.jenkins \ && mkdir -p ${AGENT_WORKDIR} \ && yum -y install git subversion sshpass VOLUME /home/jenkins/.jenkins VOLUME ${AGENT_WORKDIR} WORKDIR /home/jenkins USER root ENTRYPOINT ["jenkins-slave"]
# docker build -t harbor.59iedu.com/fjhb/jenkins-slave-toolkit:2018-08-10-v1 . # docker push harbor.59iedu.com/fjhb/jenkins-slave-toolkit:2018-08-10-v1
一、建立pv和pvcdocker
# cat pv.yaml --- apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-master-vol spec: capacity: storage: 5Gi accessModes: - ReadWriteMany nfs: path: /home/jenkins server: 192.168.115.6 persistentVolumeReclaimPolicy: Recycle --- apiVersion: v1 kind: PersistentVolume metadata: name: maven-repository spec: capacity: storage: 5Gi accessModes: - ReadWriteMany nfs: path: /home/maven server: 192.168.115.6 persistentVolumeReclaimPolicy: Recycle --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: jenkins-master-claim spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: maven-repository-claim spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi
二、建立deployment和serviceapache
# cat deploy.yaml --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: jenkins-master spec: template: metadata: labels: name: jenkins-master spec: securityContext: fsGroup: 1000 containers: - name: jenkins-master image: harbor.59iedu.com/fjhb/jenkins:2018-08-12-v1 imagePullPolicy: Always ports: - containerPort: 8080 name: http - containerPort: 50000 name: agent volumeMounts: - name: jenkins-master-vol mountPath: /var/jenkins_home - name: maven-repository mountPath: /opt/maven/repository - name: docker mountPath: /usr/bin/docker - name: docker-sock mountPath: /var/run/docker.sock volumes: - name: jenkins-master-vol persistentVolumeClaim: claimName: jenkins-master-claim - name: maven-repository persistentVolumeClaim: claimName: maven-repository-claim - name: docker hostPath: path: /usr/bin/docker - name: docker-sock hostPath: path: /var/run/docker.sock serviceAccount: "jenkins-master" imagePullSecrets: - name: harborsecret --- apiVersion: v1 kind: Service metadata: name: jenkins-master spec: type: NodePort ports: - port: 8080 name: http targetPort: 8080 nodePort: 8452 - port: 50000 name: agent nodePort: 50000 targetPort: 50000 selector: name: jenkins-master
三、rbac受权centos
# cat sa.yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-master namespace: default --- kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins-master rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: jenkins-master roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins-master subjects: - kind: ServiceAccount name: jenkins-master namespace: default
四、default sa的rbac受权api
# cat default-sa.yaml --- kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: default-role rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: default-rolebinding roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: default-role subjects: - kind: ServiceAccount name: default namespace: default
一、经过秘钥解锁jenkins(本文为了方便局域网其余主机访问,在vmware上配置了nat规则)
二、配置代理
三、安装插件
四、建立管理员帐号
架构
一、系统管理 —— 系统设置 —— 新增一个云”kubernetes”
Kubernetes URL: 输入api-server的地址
Jenkins URL: 输入jenkins server的服务名,端口8080
Jenkins Tunnel: 指的是slave链接master的端口,默认是50000
上图pod的模板名称为jenkins-slave,Container的模板名称为jnlp。这里有很是重要的两点要注意:
当Container的模板名称为jnlp的时候,jenkins-slave才会使用下面配置的docker镜像来启动pod,若是不为jnlp,则会使用默认的镜像jenkins/jnlp-slave:alpinessh
当使用自定义的docker镜像来启动jenkins slave pod的时候,下面的command to run(默认值是 sh -c)和arguments to pass to the command(默认值是cat)两个值须要清空。不然会出现jenkins slave jnlp链接不上master的状况,尝试100次链接以后销毁pod,而后再建立一个pod继续尝试链接,无限循环。maven
二、系统管理 —— Configure Global Security
确认jnlp agent的端口默认为50000,若是有修改,要保障这里的配置及前面部署deployment、service的端口配置、前文的云环境Jenkins Tunnel设置保持一致
三、系统管理 —— Global Tool Configuration
在这里设置对应的工具及环境变量,为了不没必要要的问题,前面经过dockerfile把jenkins server 和jenkins slave的环境变量调整成一致, java目录经过软链接的方式实现。
四、系统管理 —— 管理插件
推荐安装的几个插件:maven、 gitlab 、subversion、pipeline、Kubernetes Continuous Deploy、Publish Over SSH
完成插件安装后须要对jenkins server进行重启操做,能够点击“系统管理 ”——“准备关机”来完成重启操做,至此咱们就完成了jenkins server在k8s环境中的部署和配置工做,下文开始介绍使用jenkins完成项目构建和发布。