强化多云 Kubernetes 集群的服务

SAP 大规模运用 Kubernetes 集群。我们与合作伙伴协作,在多个云提供商之间和内部添加新功能和工作负载。 这项活动向我们提出了一个独特的挑战,即,在考虑云提供商的最佳安全实践的同时,保护多个租户的集群安全。在这次演讲中,我们将分享 Kinvolk 的安全之旅,并展示令人惊奇的发现,例如,信息泄露和泄密。我们还将展示和讨论我们在 SAP 运营 Kubernetes 而实施的强化活动和对策。 本次演讲对所有运营 Kubernetes 的与会者而言将妙趣横生。这将有助于他们从不同的方向思考,识别特定于他们环境、可能未知的攻击媒介,以及如何规避。
展开查看详情

1.Hardening Multi- Cloud Kubernetes Clusters as a Service

2.Who We Are Alban Crequy Dirk Marwinski CTO Kinvolk Development Architect SAP Gardener Team Love Kubernetes, Containers, Linux

3.Agenda Project Gardener Warm-up: Traditional Pen-Testing The race: A Second Opinion Cool-down: Security Add-on Summary & Conclusion

4.Project “Gardener”

5.

6.

7.Gardener Cluster Setup Seed Cluster (Off-)Shoot Clusters „A side shoot or branch on a plant“ or Worker/ Worker/ Worker/ “A thing that develops from something Master Minion Minion Minion else“ Worker/ Worker/ Worker/ Master Minion Minion Minion Worker/ Worker/ Master Minion Minion HA Worker/ Minion Worker/ manages Minion Worker/ Worker/ Worker/ Inside a Seed Cluster Minion Minion Minion Worker/Minion Worker/ Minion ETCD API Server ETCD Worker/ Worker/ Machine De-/Provisioning, Zooming into the Seed Minion Minion Self-Healing, Auto-Update and Cluste reveals… Scheduler Controller Mgr Worker/ Minion Auto-Scaling via our own Machine Controller Manager Multiple Shoot Cluster Controller Mgr Scheduler Control Planes API Server ETCD …

8. Garden Cluster Seed Cluster Shoot Cluster Administrator End-User R R R R R R R R Kubernete Gardener Cloud Cockpit Kubernetes Kubify kubectl s gardenctl Dashboard UI kubectl Dashboard Dashboard R R R R R R R R HTTPS Garden Cluster Seed Cluster Shoot Cluster Garden Cluster API LB Ingress LB Seed Cluster API LB Shoot Cluster API LB Shoot Cluster VPN LB R R R R Garden Cluster Storage Seed Cluster Storage [K8s] DS, RS, SS, J, ... [K8s] DS, RS, SS, J, ... Control Plane [CRD] Shoot, Seed, ... Control Plane [CRD] Machine Deployment R R R R R R Worker W Kubelet + Container Runtime Worker Kubelet + Container Runtime Worker Kubelet + Container Runtime etcd Main Backup SS Main PV Gardener Dashboard D VPN D etcd Events SS Events PV D R Gardener API Server D Calico S API Server VPN D D R R Kube Proxy Gardener Controller Manager D R S ... Scheduler D Kube DNS D R Controller Manager D New Shoot Clusters can be created via the Gardener Optional Addons dashboard or by uploading a new Shoot resource to the Garden Cluster. The Gardener picks it up and Addon Manager D starts a Terraform job to create the necessary IaaS components. Then it deploys the Shoot Cluster Machine Controller D R Actual Workload Control Plane into the Seed Cluster and required add- ons into the Shoot Cluster. Update or delete operations are handled by the Gardener fully Terraformer Shoot Cluster Control Plane J R ... automatically as well. Monitoring VPN ... Logging R SCP ... ... ... IaaS IaaS

9.An Analogy … The shoot cluster is like your apartment, you get full access to it and can do what you want. The seed cluster is like an apartment building with a master key to all apartments within. The Garden cluster is the apartment company with full access to all apartment buildings. (All images CC0 Creative Commons)

10.Challenges You are about to invite guests into your apartment but you don’t want them to enter your private bedroom or open your safe. You need to ensure that your neighbors don’t penetrate or sneak stuff from your apartment. You don’t want apartment tenants or guests to get control of the apartment building (which would provide them with full access to all apartments). Somebody who could take over the apartment company can do everything.

11.Warm up: The “Traditional” Approach

12.Warm-up We have asked our established pen-testers to attack the Gardener setup. They unpacked the big guns like Rapid7… … however they openly admitted not to have Kubernetes know-how (Wikipedia)

13.Results This is what they found: • Shoot cluster can be breached • Hey, its your own place, you are perfectly entitled to do that • Other minor issues So, are we safe? CC0 Creative Commons

14.The Race: A Second Opinion

15.A Second Opinion … This was too easy. With the Kubernetes expertise from Alban and Michael from Kinvolk, we performed penetration tests. They came in with filigree tools to try and breach the apartment, apartment building, building company. CC BY-SA 3.0 Here is what they found …

16.#1 Shared Account: Full Access to your Neighbour's Appartment (1) IaaS account secrets on worker nodes IaaS account API Server API Server secrets Controller … Controller … Seed cluster Manager Manager (IaaS account 1) Control Plane 1 Control Plane 2 Worker/ Worker/ Worker/ Worker/ Minion Minion Minion Minion kubelet kubelet kubelet kubelet Shoot clusters (IaaS account 2)

17.#2 Shared Account: Sneak Stuff out of your Neighbour’s Apartment (2) Tell your controller manager to do things on your behalf (it has got the account credentials) IaaS account API Server API Server secrets Controller … Controller … Seed cluster Manager Manager (IaaS account 1) Control Plane 1 Control Plane 2 Worker/ Worker/ Worker/ Worker/ Minion Minion Minion Minion kubelet kubelet kubelet kubelet Shoot clusters (IaaS account 2)

18. #1 + #2 General recommendations (1) Do not store account credentials on the worker nodes. (2) You can add an Admission Webhook to protect apartments from each other. However, be warned that depending on the IaaS provider there are most likely more attack vectors! CC0 Creative Commons

19.#3 Privilege Escalation

20.#3 Authenticating Proxy

21.#3 Bad Proxy Configuration Configuring the apiserver to accept requests from proxy: --requestheader-client-ca-file=/srv/kubernetes/ca/ca.crt --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group CC BY-SA 3.0

22.#3 Proxy CA Certificate

23.#3 The Exploit By setting the HTTP headers, it is possible to impersonate another user and gain their privileges

24. #3 The Fixes Using different CA in the apiserver configuration: --client-ca-file=/srv/kubernetes/ca/ca.crt --requestheader-client-ca-file=/srv/kubernetes/ca-front-proxy/ca.crt Kubernetes already had a warning in the documentation, which we improved: https://github.com/kubernetes/website/pull/10093

25. Metadata Service Cloud providers provide a metadata service on all instances - AWS, GCP, Azure: 169.254.169.154 - AliCloud: 100.100.100.200 It contains bootstrap information, including a kubeconfig. It can contain credentials to talk the the cloud provider API.

26.#4 Attack via Grafana Grafana • Grafana was offered as a service to users of the shoot cluster • It lives in the control plane: another possible attack surface to explore

27.#4 Architecture with Grafana

28.#4 Use Grafana for Metadata Access

29.#4 Use Grafana for Metadata Access