强化多云 Kubernetes 集群的服务

1 .Hardening Multi- Cloud Kubernetes Clusters as a Service 

2 .Who We Are Alban Crequy Dirk Marwinski CTO Kinvolk Development Architect SAP Gardener Team Love Kubernetes, Containers, Linux 

3 .Agenda Project Gardener Warm-up: Traditional Pen-Testing The race: A Second Opinion Cool-down: Security Add-on Summary & Conclusion 

4 .Project “Gardener” 

5 .

6 .

7 .Gardener Cluster Setup Seed Cluster (Off-)Shoot Clusters „A side shoot or branch on a plant“ or Worker/ Worker/ Worker/ “A thing that develops from something Master Minion Minion Minion else“ Worker/ Worker/ Worker/ Master Minion Minion Minion Worker/ Worker/ Master Minion Minion HA Worker/ Minion Worker/ manages Minion Worker/ Worker/ Worker/ Inside a Seed Cluster Minion Minion Minion Worker/Minion Worker/ Minion ETCD API Server ETCD Worker/ Worker/ Machine De-/Provisioning, Zooming into the Seed Minion Minion Self-Healing, Auto-Update and Cluste reveals… Scheduler Controller Mgr Worker/ Minion Auto-Scaling via our own Machine Controller Manager Multiple Shoot Cluster Controller Mgr Scheduler Control Planes API Server ETCD … 

8 . Garden Cluster Seed Cluster Shoot Cluster Administrator End-User R R R R R R R R Kubernete Gardener Cloud Cockpit Kubernetes Kubify kubectl s gardenctl Dashboard UI kubectl Dashboard Dashboard R R R R R R R R HTTPS Garden Cluster Seed Cluster Shoot Cluster Garden Cluster API LB Ingress LB Seed Cluster API LB Shoot Cluster API LB Shoot Cluster VPN LB R R R R Garden Cluster Storage Seed Cluster Storage [K8s] DS, RS, SS, J, ... [K8s] DS, RS, SS, J, ... Control Plane [CRD] Shoot, Seed, ... Control Plane [CRD] Machine Deployment R R R R R R Worker W Kubelet + Container Runtime Worker Kubelet + Container Runtime Worker Kubelet + Container Runtime etcd Main Backup SS Main PV Gardener Dashboard D VPN D etcd Events SS Events PV D R Gardener API Server D Calico S API Server VPN D D R R Kube Proxy Gardener Controller Manager D R S ... Scheduler D Kube DNS D R Controller Manager D New Shoot Clusters can be created via the Gardener Optional Addons dashboard or by uploading a new Shoot resource to the Garden Cluster. The Gardener picks it up and Addon Manager D starts a Terraform job to create the necessary IaaS components. Then it deploys the Shoot Cluster Machine Controller D R Actual Workload Control Plane into the Seed Cluster and required add- ons into the Shoot Cluster. Update or delete operations are handled by the Gardener fully Terraformer Shoot Cluster Control Plane J R ... automatically as well. Monitoring VPN ... Logging R SCP ... ... ... IaaS IaaS 

9 .An Analogy … The shoot cluster is like your apartment, you get full access to it and can do what you want. The seed cluster is like an apartment building with a master key to all apartments within. The Garden cluster is the apartment company with full access to all apartment buildings. (All images CC0 Creative Commons) 

10 .Challenges You are about to invite guests into your apartment but you don’t want them to enter your private bedroom or open your safe. You need to ensure that your neighbors don’t penetrate or sneak stuff from your apartment. You don’t want apartment tenants or guests to get control of the apartment building (which would provide them with full access to all apartments). Somebody who could take over the apartment company can do everything. 

11 .Warm up: The “Traditional” Approach 

12 .Warm-up We have asked our established pen-testers to attack the Gardener setup. They unpacked the big guns like Rapid7… … however they openly admitted not to have Kubernetes know-how (Wikipedia) 

13 .Results This is what they found: • Shoot cluster can be breached • Hey, its your own place, you are perfectly entitled to do that • Other minor issues So, are we safe? CC0 Creative Commons 

14 .The Race: A Second Opinion 

15 .A Second Opinion … This was too easy. With the Kubernetes expertise from Alban and Michael from Kinvolk, we performed penetration tests. They came in with filigree tools to try and breach the apartment, apartment building, building company. CC BY-SA 3.0 Here is what they found … 

16 .#1 Shared Account: Full Access to your Neighbour's Appartment (1) IaaS account secrets on worker nodes IaaS account API Server API Server secrets Controller … Controller … Seed cluster Manager Manager (IaaS account 1) Control Plane 1 Control Plane 2 Worker/ Worker/ Worker/ Worker/ Minion Minion Minion Minion kubelet kubelet kubelet kubelet Shoot clusters (IaaS account 2) 

17 .#2 Shared Account: Sneak Stuff out of your Neighbour’s Apartment (2) Tell your controller manager to do things on your behalf (it has got the account credentials) IaaS account API Server API Server secrets Controller … Controller … Seed cluster Manager Manager (IaaS account 1) Control Plane 1 Control Plane 2 Worker/ Worker/ Worker/ Worker/ Minion Minion Minion Minion kubelet kubelet kubelet kubelet Shoot clusters (IaaS account 2) 

18 . #1 + #2 General recommendations (1) Do not store account credentials on the worker nodes. (2) You can add an Admission Webhook to protect apartments from each other. However, be warned that depending on the IaaS provider there are most likely more attack vectors! CC0 Creative Commons 

19 .#3 Privilege Escalation 

20 .#3 Authenticating Proxy 

21 .#3 Bad Proxy Configuration Configuring the apiserver to accept requests from proxy: --requestheader-client-ca-file=/srv/kubernetes/ca/ca.crt --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group CC BY-SA 3.0 

22 .#3 Proxy CA Certificate 

23 .#3 The Exploit By setting the HTTP headers, it is possible to impersonate another user and gain their privileges 

24 . #3 The Fixes Using different CA in the apiserver configuration: --client-ca-file=/srv/kubernetes/ca/ca.crt --requestheader-client-ca-file=/srv/kubernetes/ca-front-proxy/ca.crt Kubernetes already had a warning in the documentation, which we improved: https://github.com/kubernetes/website/pull/10093 

25 . Metadata Service Cloud providers provide a metadata service on all instances - AWS, GCP, Azure: 169.254.169.154 - AliCloud: 100.100.100.200 It contains bootstrap information, including a kubeconfig. It can contain credentials to talk the the cloud provider API. 

26 .#4 Attack via Grafana Grafana • Grafana was offered as a service to users of the shoot cluster • It lives in the control plane: another possible attack surface to explore 

27 .#4 Architecture with Grafana 

28 .#4 Use Grafana for Metadata Access 

29 .#4 Use Grafana for Metadata Access