Kustomize：针对 Kubernetes 的无模板配置自定义- Jeffrey Regan, Google

1 .kubernetes configuration customization a composable, template-free approach Nov 2018 kubecon 上海 jeff regan jregan@google monopole@github 

2 .Kubernetes 配置的用户定制 声明式的途径 Nov 2018 kubecon 上海 jeff regan jregan@google monopole@github 

3 .~60 k8s configuration tools k8s的配置工具 spreadsheet recent - automation broker maintained by Brian Grant 

4 . app app descriptor discovery package cluster manager dashboard customization lifecycle management 

5 . app app descriptor discovery Description, maintainer, version, ... package cluster manager dashboard customization lifecycle management 

6 . app app descriptor discovery Browse, search, download package cluster manager dashboard customization lifecycle management 

7 . app app descriptor discovery package cluster manager dashboard Bundling, plus dependencies customization lifecycle management 

8 . app app descriptor discovery package cluster manager dashboard What apps are running? Are they healthy? customization lifecycle management 

9 . app app descriptor discovery package cluster manager dashboard customization lifecycle management Rollouts, rollbacks, upgrades. 

10 . app app descriptor discovery package cluster manager dashboard customization lifecycle management Given config - adapt it to my needs. 

11 .kustomize Command line tool for k8s customization. k8s 用户化的命令行工具 Closes several old kubectl issues. 解决了一些kubectl的老问题 Composes with other tools. customization 可以和别的工具一起使用 sponsored by sig-CLI per this proposal 

12 .$ kustomize build helloWorld | \ kubectl apply -f - 

13 .$ tree helloWorld al helloWorld ← norm ├── configMap.yaml norm al ← ├── deployment.yaml o p p ed in r ←d ├── kustomization.yaml al └── service.yaml ← norm 

14 .service.yaml kustomization.yaml /dev/stdout kind: Service kind: Service metadata: resources: metadata: name: wordpress - service.yaml name: demo-wordpress kustomize spec: build spec: ports: ports: - port: 389 namePrefix: demo- - port: 389 selector: selector: app: wordpress app: wordpress 

15 .This is k8s-aware patching. 

16 . operands (things to include) kustomization.yaml = operations (ways to patch operands) 

17 .operands operations result service.yaml namePrefix: deployment.yaml demo- YAML Stream configMap.yaml 

18 .operands operations result YAML Stream A result can be used as an operand. 

19 . kustomize input is plain kubernetes yaml. You can kubectl apply that yaml without kustomize. To start customizing, just add a kustomization.yaml file. 

20 .Use Case #1 Variants 不同环境 dev (development, staging and production) image: hourly build staging common resources image: passed QA 1% of prod traffic r1.yaml, r2.yaml, ... production image: v2.3.8 replicas: 3000 cpu: 100 

21 .Use Case #1 Variants 不同环境 kustomization.yaml dev (development, staging and production) namePrefix: dev- bases: - ../../base kustomization.yaml kustomization.yaml staging resources: - r1.yaml namePrefix: staging- r2.yaml newTag: qa ... bases: - ../../base kustomization.yaml production namePrefix: prod- newTag: v2.3.8 bases: - ../../base 

22 .Use Case #1 Variants 不同环境 $ tree wordpress (development, staging and production) wordpress ├── base File layout: │ ├── kustomization.yaml │ ├── deployment.yaml │ ├── env.txt kustomization.yaml service.yaml deployment.yaml │ └── service.yaml └── overlays kind: Service ├── production kind: Deployment commonLabels: │ └── kustomization.yaml metadata: metadata: app: wordpress │ ├── replica_count.yaml name: wordpress name: wordpress resources: spec: │ └── cpu_count.yaml spec: - deployment.yaml ports: └── staging replicas: 1 - service.yaml - port: 389 template: ... │ ├── kustomization.yaml configMapGenerator: │ └── ... - name: wordpress-map └── dev files: ├── kustomization.yaml - env.txt └── ... 

23 .Use Case #1 Variants 不同环境 $ tree wordpress (development, staging and production) wordpress ├── base │ ├── kustomization.yaml │ ├── deployment.yaml │ ├── env.txt kustomization.yaml replica_count.yaml cpu_count.yaml │ └── service.yaml └── overlays namePrefix: prod- kind: Deployment kind: Deployment ├── production commonLabels: metadata: metadata: │ ├── kustomization.yaml variant: prod name: wordpress name: wordpress │ ├── replica_count.yaml commonAnnotations: spec: spec: │ └── cpu_count.yaml note: I’m Prod! replicas: 80 template: └── staging bases: spec: │ ├── kustomization.yaml - ../../base containers: │ └── ... patchesStrategicMerge: - name: my-container └── dev - replica_count.yaml resources: ├── kustomization.yaml - cpu_count.yaml limits: └── ... cpu: 7000m 

24 . $ tree wordpress Deploy production: wordpress ├── base │ ├── kustomization.yaml │ ├── deployment.yaml │ ├── env.txt $ kustomize build \ │ └── service.yaml wordpress/overlays/production |\ └── overlays ├── production kubectl apply -f - │ ├── kustomization.yaml │ ├── replica_count.yaml │ └── cpu_count.yaml └── staging │ ├── kustomization.yaml │ └── ... └── dev ├── kustomization.yaml └── ... 

25 . $ tree wordpress Deploy staging: wordpress ├── base │ ├── kustomization.yaml │ ├── deployment.yaml │ ├── env.txt $ kustomize build \ │ └── service.yaml wordpress/overlays/staging |\ └── overlays ├── production kubectl apply -f - │ ├── kustomization.yaml │ ├── replica_count.yaml │ └── cpu_count.yaml └── staging │ ├── kustomization.yaml │ └── ... └── dev ├── kustomization.yaml └── ... 

26 .$ kustomize build target 1 load universal k8s object descriptions 2 read kustomization.yaml from target 3 kustomize bases (recurse 2-5) 4 load and/or generate resources 5 apply target’s kustomization operations 6 fix name references 7 emit yaml 

27 .Use Case #2 Feeding customized names to containers 把用户化的名字放入容器 patch.yaml kind: Deployment /dev/stdout kustomization.yaml metadata: name: wordpress apiVersion: v1 vars: spec: kind: Deployment - name: MYSQL_SERVICE template: ... objref: spec: spec: kind: Service initContainers: initContainers: name: mysql - name: init-command - command: apiVersion: v1 image: debian fieldref: - curl demo-mysql command: fieldpath: metadata.name containers: - "curl $(MYSQL_SERVICE)" patchesStrategicMerge: - env: - patch.yaml containers: - name: WORDPRESS_DB_HOST - name: wordpress env: value: demo-mysql - name: WORDPRESS_DB_HOST value: $(MYSQL_SERVICE) 

28 .Use Case #3 ConfigMaps generated from multiple sources 归并属性 base production overlay /dev/stdout kustomization.yaml kind: ConfigMap kustomization.yaml bases: metadata: configMapGenerator: - ../../base name: prod-myCMap-b5m75cxc - name: myCMap namePrefix: prod- data: files: configMapGenerator: color=blue - common.properties - name: myCMap height=10m behavior: merge dbpassword=foo files: common.properties - secret.properties color=blue height=10m secret.properties … so on for staging and development variants. dbpassword=foo Properties can be owned by different teams. 

29 .It’s all patching. kustomize is just a means to manage k8s-targeted patching.