eBay 使用 Kubernetes 进行数据中心建模和操作

Kubernetes 提供很棒的应用程序模型抽象功能。Kubernetes 还会运行一组控制器来处理日常任务,以确保应用程序的理想状态与观察到的状态相匹配。但是,您认为 Kubernetes 能够大规模地为数据中心建模并通过控制器操作数据中心吗?这在 eBay 已经实现。我们正面对着在成千上万台机器上运行的大规模集群。我们需要一个中央服务,一个运行基础架构和应用程序的统一接口。 我们利用 Kubernetes CRD 为基础架构建模,例如 Rack、L2Domain、NetworkScope、Asset 和 Flavor 等。此外,我们还使用 Kubernetes CRD 为 K8sCluster 建模。在本次演讲中,我们将分享如何从 3 个层次为数据中心建模: - 机群管理 - 配置管理 - K8S 集群管理
展开查看详情

1.Model and Operate Datacenter by Kubernetes at eBay 辛肖刚, Cloud Engineering Manager, ebay 梅岑恺, Senior Operation Manager, ebay

2.Agenda About ebay Our fleet Kubernetes makes magic at ebay How we model our datacenter Model + Controller Operation in large scale Q&A

3. About ebay 177M $22.7B Active buyers worldwide Amount of eBay Inc. GMV $2.6B Reported revenue 1.1B 81% Live listings Sold are new 88% $11B Fixed price Mobile 62% International revenue

4.Our fleet 3 15 200K+ US Data Centers POPs Managed Vms 100K 4K 4.5PB Managed BMs Applications Managed Storage

5.All of us know that... It’s not easy to manage fleet and infrastructure at scale

6.Infrastructure should play magic

7.Way to Kubernetes 20 No 15- w 2 - 1 20 ow 20 N No 10- w Bare metals ...- 0 1 20

8.Kubernetes plays magic Search Grid Hadoop PoP Database Frontend VM

9. Kubernetes Core concept of Kubernetes - Declarative magic control container loop kind: api metadata: spec: control VIP loop etcd control volume loop WISB: What it should be Converge & Reconcile WIRI: What it really is

10.Kubernetes models applications What is an application looks like? Config Volume LoadBalancer Docker Image RBAC Replica Rolling Quota Update

11. How about Kubernetes itself? How about the fleet Kubernetes running on?

12.Our thinking of datacenter modeling by extending Kubernetes Onboard Provision Configuration Kubernetes • Rack • OS • Kernel params • Core components • NetworkDevice • Flavor • Environment config • Addon • ComputeAsset • ComputeNode • Network • Taint • Region/AZ • NetworkScope Operations

13.Let’s model a datacenter running Kubernetes Onboard Provision Configuration Kubernetes You need onboard something from nothing!

14.Let’s model a datacenter running Kubernetes Onboard Provision Configuration Kubernetes After you define your fleet, you want a accessible compute node: Asset + Flavor + OS = ComputeNode

15.Let’s model a datacenter running Kubernetes Onboard Provision Configuration Kubernetes After you define your fleet, you want a accessible compute node: Asset + Flavor + OS = ComputeNode

16.Let’s model a datacenter running Kubernetes Onboard Provision Configuration Kubernetes After you define your fleet, you want a accessible compute node: Asset + Flavor + OS = ComputeNode

17.Let’s model a datacenter running Kubernetes Onboard Provision Configuration Kubernetes You have your compute node now, all you need is to configure it by a configuration management orchestration. We use SaltStack.

18.Let’s model a datacenter running Kubernetes Onboard Provision Configuration Kubernetes It’s time to spin up a Kubernetes cluster!

19.Easy operation

20. How to flex up some nodes? Step 1. Find some assets not used Step 2. Create a ComputeNode Step 3. Relax and have a cup of coffee

21. What if salt master down? Step 1. Find some compute nodes Step 2. Create a SaltMaster Step 3. Relax and have a cup of coffee

22.How to upgrade a cluster? Upgrade Kubernetes core components Upgrade addons

23.Recap ● Kubernetes is amazing on its simple architecture ● Model + Controller is the key concept of Kubernetes ● It’s easy to extend Kubernetes API and write your controller based on list/watch ● ebay uses Kubernetes to model and operate it’s datacenter KafkaCluster, HadoopCluster, MongoDB, ESCluster …… Application Service K8sCluster, K8sAddons, K8sDeployment Infrastructure Service SaltMaster, SaltMinion, SaltDeployment Configuration Management Region, AvailabilityZone, NetworkZone, L2Domain Fleet (Compute, Network, Rack, NetworkDevice, ComputeAsset Storage)

24.We are hiring! xnxin@ebay.com cmei@ebay.com

25.Q&A