Kubernetes 原生开发运维实例

1 .Kubernetes Native DevOps Practice — 王磊磊 @TenxCloud 

2 .Agenda • Our DevOps Expectations • Kubernetes Capabilities/Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline / Stage/ Task / Task Template / Version Control • Logging, monitoring, autoscaling, high availability • Extensibility / Integration • CI/CD examples • Future plan 

3 .Our DevOps Expectations • Build a platform and easy to integrate with other DevOps/third-party tools • Easy to be customized as user requirements are diverse • Easy to setup, maintain, extend and scale • Reduce the learning curve for customer and ourselves • Get consistent user experience and data, leverage with PaaS capability • Facilitate our PaaS and micro-service product 

4 .Kubernetes Capabilities/Advantages to Build DevOps Solution • k8s itself is NOT a PaaS or DevOps platform，but … • k8s resources that can be used to build DevOps solution Pod Job CronJob Volumes ConfigMap Secret ResourceQuota / LimitRanges • Scheduler / Affinity • And more … 

5 .Kubernetes Capabilities/Advantages to Build DevOps Solution Pod Spec [] InitContainers Initialize the build environment Affinity / SchedulerName Configure scheduler policy [] Volumes Share files between containers, or cache build files ActiveDeadlineSeconds Timeout of build task [] Containers Build tasks and the dependent environments（sidecar） Container Image - Image of build / dependent environment [] Command - Command to execute [] VolumeMounts - Files to be shared or persisted [] Args - Argument [] Resources - Resource requirement [] Env - Environment variable Lifecycle - Actions defined for postStart/preStop 

6 . Kubernetes Capabilities/Advantages to Build DevOps Solution parallelism Expected maximum number of parallel build tasks completions Expected number of completed build tasks Job Spec activeDeadlineSeconds The timeout of the running build tasks, default: 12 hours Maximum retry count before mark the build task as failed, backoffLimit default: 6 v1.12 alpha ttlSecondsAfterFinished Time to clean up finished build tasks after if finishes Pod Template schedule Cron style scheduler configuration concurrencyPolicy Concurrency policy of CronJob CronJob Spec suspend Whether suspend latter jobs if the previous job is still running successfulJobsHistoryLimit Number of successful/failed history jobs to keep failedJobsHistoryLimit Job Template 

7 .Agenda • Our DevOps Expectations • Kubernetes Capabilities and Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline/Stage/Task/Task Template/Version Control/UI generation/Volume... • Logging, monitoring, autoscaling, high availability • Extensibility/Integration • CI/CD examples • Future plan 

8 . Overall Architecture kubelet can do image GC Scheduling customization Kubernetes Cluster Kubernetes Cluster Job Job Job Job Logging ElasticSearch ElasticSearch Node Node Node Node Node Node Service Pod Pod Pod Pod Pod Pod docker registry k8s API agent to collecting log data CronJob CronJob Monitor/Alert ElasticSearch ElasticSearch Service DevOps DevOps k8s API Unified logging、monitoring、alert with PaaS Operator Operator Consistent data DevOps Node group of build nodes Node group of user applications DevOps DevOps Service Service Cluster Resource Auto Scaling Manager • Pipeline configuration and history in MySQL • Logging in central logging service - ElasticSearch • Metric data in monitoring system - prometheus MySQL MySQL MySQL • Alertmanager to invoke various alert and related actions 

9 .CRD and Operator Design Kubernetes Cluster Kubernetes Cluster Create job BuildJob DevOps Operator Submit buildjob BuildJob Update jobs status to buildjob DevOps DevOpsService DevOps Service sync / watch Manager BuildJob clean history jobs List/Watch buildjob BuildJob Job Job Job Job Pod Pod Pod Pod MySQL MySQL MySQL Build task configuration - map to k8s Job, can also be a raw k8s job Pipeline / Stage / Task Job / Pod / Node info Task Template BuildJob / Job status Pipeline / Stage / Task build logs Version Control 

10 . Basic Concepts（partial） Repository Pipeline / Stage / Task Tasks in same stage can run sequentially or in parallel Task created from template Task Template Managed Project Dockerfile / Scripts Data Volume Common Configuration PVC ConfigMap/Secret 

11 . Logging, Monitoring, Autoscaling, High Availability Kubernetes Cluster vCenter openstack Job Job Job Job push metric data ElasticSearch ElasticSearch ElasticSearch Prometheus ElasticSearch remove / add Node Node Node Node ElasticSearch Push Gateway nodes Pod Infrastructure Layer Pod Pod Cloud provider agent to collecting log data pull metric data Cluster DevOps insufficient resource AutoScaler Operator Monitor/Alert ElasticSearch ElasticSearch Service k8s API realtime log DevOps Service DevOps DevOps Service Manager history log • Build task can also expose custom metric data • Ephemeral build task can push metric to gateway if needed • Cluster autoscaler will add/remove node from build group for scaling • HA is guaranteed by cluster HA, k8s Job controller and cluster autoscaler, can Restful API also use container probe if needed 

12 .Extensibility / Integration • Easy to extend task template, just docker images • Use can define raw k8s job, more features can be enabled • Leverage all k8s resources and extensibility to fulfill various requirements, prompt innovation • New features of k8s may help enhance or improve DevOps • Help integration test - use sidecar container as dependent environment • Encapsulate API / SDK of other tools using image for better integration/ collaboration • Leverage k8s integration capabilities, such as external service registration, service catalog, integrate using CRD and controller, etc… 

13 .CI/CD Examples - Build Docker Image dockerfile DevOps using ConfigMap Operator Manage the Job Job - pod template init task • prepare code repository environment variables - volumes user build task • build the docker images image information - volumes - resources - volumes Memory / CPU / GPU Data cache pvc pvc pvc registry credential Storage Service using secret rbd / nfs / glisters 

14 .CI/CD Examples - Artifact Management user scripts DevOps using ConfigMap Operator Manage the Job Job - pod template init task • prepare code repository - volumes user build task • build the application package sidecar build task • upload files to storage service once user build completes lifecycle - preStop - volumes Query artifact data Storage APIs storage config using secret 

15 .CI/CD Examples - Human/Manual Task User Action DevOps Service DevOps DevOps Service Approve/Deny to make job successful or failed Manager Authorization check Job environment variable • send out email if needed approver list • wait for user actions approval timeout activeDeadlineSeconds volumes system email config using secret 

16 .CI/CD Examples - Continuous Deployment serviceaccount Upgrade an existing service Security policy for the build task Job • Image with kubectl or call k8s API environment variables • Image with helm client Deploy using helm template • Call Spinnaker webhook deployment strategy - Rolling Upgrade / Recreate - Deploy / Upgrade using helm，rollback if unready pods - Check tracing data, rollback if get unexpected failure rate - Call Spinnaker API to start deployment pipeline Kubernetes Cluster Kubernetes Cluster deployment pipeline Invoke Spinnaker web hook of Pod Pod Pod Pod specified deployment pipeline 

17 . CI/CD Examples - Gitlab/Sonar/Jenkins Integration Events Trigger with payload Jenkins API Image Sonar tool Image • Encapsulate API / SDK of third party tools to docker image • Pass events from other system • Gitlab webhook to build task, user can do what • On schedule they want based on the payload • Registry notification 

18 . CI/CD Examples - Gitlab/Harbor/Jira Integration Event payload can be passed to build task if needed • Call harbor API to add/remove label of image, can also lock the passed gitlab new merge image request event • Get the commits between two builds • Invoke Jira API to mark the status and add the build info git log --pretty=oneline c5eff7ea..3211901e 3211901e9b877c92ab059a6f25180469dcbf1629 Merge branch 'dev-branch' into 'dev-branch' 5d6cbb9d09a8e8a34cad8a27df036c07f3b4c291 [KK-134] Fix xxx b10192764026df3a1eac57d69019032eeb722ceb [LOT-3372] Fix xxx 7797990da64929112a90e9f6839bd3420e64c99b [KK-9] Fix xxx 7797990da64929112a90e9f6839bd3420e64c99b [LOT-3213] Fix xxx 96ce85fdecd50aafafca2eae6a2a1fe4b1aef72d Merge branch 'LOT-3033' into 'dev-branch' git commit should have naming convention 

19 .Agenda • Our DevOps Expectations • Kubernetes Capabilities and Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline/Stage/Task/Task Template/Version Control/UI generation/Volume... • Logging, monitoring, autoscaling, high availability • Extensibility/Integration • CI/CD examples • Future plan 

20 . Our Future Plan • More task templates to be added, integrate more CI/CD and project management tools • Optimize UI generation methodology • Improve development experience, such as CLI, plugin for IDE, dev on Cloud • Move forward to better DevOps under micro-service architecture • Consolidate the consistent events and data, and create k8s native ChatOps tool • Contribute to open source community Automation Integration Collaboration Quantification 

21 .Thank you !