VMware Michael Chen《Kubernetes-+面向未来的开发和部署》

1 .Kubernetes – Software Platform for the Future Michael Chen VMware 

2 . • Hello World! • Containerize Applications • Micro Services Applications • Scalable Kubernetes Applications • Scalable Infrastructure for Applications Developer 4 

3 .Platform Application Operating System Physical Infrastructure 

4 .Containers as Enabler Fast Lightweight Portable Boot Environments Minimal Resources Ability to Move Rapidly Needed Containers Freely 

5 . Containers and VMs - A Practical Comparison Containers virtualize Application The hypervisor Containers the operating system virtualizes limiting the the number the hardware limiting of application the number of Operating System dependencies that hardware you need to install on dependences the OS. Virtualization Physical that you need to Infrastructure install on the OS 

6 .2 technologies with 2 different objectives Application Abstract the App Allows you to run multiple Containers From the OS applications on the same OS Docker Operating System Abstract the VMs Allows you to run multiple OS OS from the VMware Hypervisor on the same hardware hardware Physical Infrastructure 

7 .Containers User Cases Developer Sandbox Application Repackaging Cloud Native • Ready-to-go development • Simplify app maintenance • New application development • Self-service portal • Improve developer workflow • 12-factor apps, PCF 9 

8 .The need for containers and containers orchestrators Application Journey v v Platform Evolution Legacy Application Application Modernized Application Repackaging Refactoring Application App packaged in VM App packaged in Container 

9 . Kubernetes in 5 min Docker and Kubernetes https://youtu.be/PH-2FfFD2PU ​Docker Kubernetes Orchestrating Multiple Containers ​Run One Container at a Time • Core docker functionality provides the tooling to • Scheduling, provisioning, and resource create and run single containers management of multiple containers – Very manual, no fault tolerance, hard to scale, – Docker, Mesos à Kubernetes Support etc – AWS, Azure, Google à Kubernetes Services $docker run container1 $kubectl create –f App.yaml Wanted: $docker run container2 Container $docker Orchestrator! run container3 $docker run container4 The “App” The Kubernetes Cluster “App” Confidential │ ©2018 VMware, Inc. 11 

10 .What is Kubernetes? Docker Host App 1 App 2 App 3 Bins/Libs Bins/Libs Bins/Libs Container Engine OS 12 

11 .What is Kubernetes? Docker Host Docker Host Docker Host App 1 App 2 App 3 App 1 App 2 App 3 App 1 App 2 App 3 Bins/Libs Bins/Libs Bins/Libs Bins/Libs Bins/Libs Bins/Libs Bins/Libs Bins/Libs Bins/Libs Kubernetes Slave Kubernetes Slave Kubernetes Slave Container Engine Container Engine Container Engine OS OS OS Kubernetes Master 13 

12 . https://youtu.be/PH-2FfFD2PU Kubernetes 101 at the Highest Level • Container Cluster = “Desired State Management” P1R1 P2R1 P2R1 P1R1 – Kubernetes Cluster Services (w/API) K Worker Node • Node = Container Host w/agent called “Kubelet” VM • Application Deployment File = Configuration File of desired state P1R2 P2R1 P1R1 • Container Image = Runs in a Pod (~1:1) • Replicas = QTY of Pods that must be running K Worker Node VM App_X.yaml P1R3 P2R2 P2R2 P1R2 ContainerImage1 Replicas: 3 App_Y.yaml K Worker Node Kubernetes VM ContainerImage2 ContainerImage1 API Master Node Replicas: Replicas: 1 2 (Master & etcd nodes) ContainerImage2 Replicas: 2 VM 14 

13 .Kubernetes Trend 

14 .The Kubernetes Master Node Basic Components • Key/Value Store Master Node Worker Node • Leader based clustering ETCD • Can be clustered across Master Nodes • Kubernetes Front-end Control Plane • Contains all state known about cluster • Provides RESTful interface kube-apiserver • Returns state objects as JSON • Provides core control loops for platform kube-controller-manager •• Watches shared Policy-based state through workload apiserver scheduler •• Makes changes Topology aware from current to desired kube-scheduler • Assists with availability, performance and capacity • Affinity/Anti-Affinity Capable 

15 .The Kubernetes Worker Node Basic Components • Master Node Container Runtime Interface Worker Node • Responsible for downloading images ETCD CRI-containerd • Runs containers • Load-balance interface for Pods • Can use other runtimes such as rkt • Creates virtual IP forkube-apiserver external access Kubeproxy • Interfaces withinterface Load-balance local iptables for Pods • Creates virtual kube-controller-manager IP for external access Kubelet • Interfaces with local iptables kube-Scheduler 

16 .The Kubernetes Runtime Components Component Description • Runtime Objects Pods A grouping of one or more containers as an atomic unit Namespaces A way to organize items in a cluster Labels, Annotations & Selectors Tags for component grouping and methods to access them Service Discovery An object associated to a label selector to provide a LB and Service DNS ReplicaSets A cluster wide Pod manager providing Pod scaling DaemonSets A Pod manager to ensure a Pod is scheduled across a Cluster Node set StatefulSets Replicated Pods where each Pod gets an indexed hostname Jobs A Pod which runs until the process returns a successful termination Deployments Manages the release rollout of new versions of Pods Singletons A single instance of a Pod which is not replicated or scaled 

17 .Developers Admin 19 

18 .Kubernetes & PKS Deploys and Operates $kubectl create –f App.yaml the Kubernetes Clusters The “App” Kubernetes Cluster 

19 .The difference between PKS and Kubernetes Open Source Project – Google/Pivotal/VMware Container scheduling, scale, resiliency, and Day 2 Desired state of Application Kubernetes cluster scheduling, scale, resiliency, and Day 2 Desired state of Kubernetes 21 

20 .VMware PKS Value Proposition A turnkey solution for enterprises and service providers to provision, operate and manage production grade Kubernetes clusters Fully supported Kubernetes On-Demand Provisioning Built for Day 2 Operations PKS uses the latest stable PKS deploys Kubernetes PKS simplifies Day 2 operations with distribution of Kubernetes—with clusters on-demand on built-in network security—powered by no proprietary extensions and vSphere or GCP NSX, high availability, logging, constant compatible with GKE monitoring, analytics, and automated health checks. 22 

21 .Container Service for Cloud-Native Apps Rapidly deliver and operationalize next generation apps 

22 . End User Experience pks create cluster A • Installation and clusters creation pks create cluster B Operator Developer Deploy Worker OpsMgr API OVA Kubernetes Worker Deploy Master BOSH Cluster A Worker Upload PKS tile Worker API Kubernetes Worker Master Cluster B Worker 

23 . #pks resize K8s-3 n=5 create-cluster K8s-1 n=3 K8s-2 K8s-3 Architecture PKS Controller API Worker API PKS Controller Kubernetes Worker Cluster Services PKS Worker • Includes K8s-1 • PKS Controller, NSX-T • CFCR, Harbor, Broker Kubernetes CNI API Worker (As a Bosh Release) NSX-T • Deploys & Configures Kubernetes Worker - CFCR Cluster Services - vSphere BOSH CPI vSphere IaaS Worker - NSX-T Integration (Deploys/Manages VMs) K8s-2 - Harbor • Manages Cluster Day 2 API Worker - Scaling Kubernetes Worker - Patching Cluster Services - Upgrades Worker - Failures Harbor GCP K8s-3 Cluster3 Worker Private Container Service Worker Registry Broker Worker 25 

24 .Architecture T1 Bosh PKS NSX-T Admin Network T1 POD POD POD POD 2 3 5 6 T0 Namespace ‘demo’ PODs – Logical Switch T1 POD 1 POD 4 Namespace ‘foo’ PODs – Logical Switch T1 NCP kube-system PODs – Logical Switch NSX NSX NSX Manager Controllers Edge Cluster API Pod Pod Pod VMware vSphere etcd NCP Srv 1 3 5 Pod Pod Pod 2 4 6 Master ‘VM’ Worker ‘VM’ Worker ‘VM’ Worker ‘VM’ T1 Cluster Management Nodes – Logical Switch 

25 .Architecture Network Container Plugin (NCP) NSX-T • NSX Container Plugin: NCP is a software component provided by VMware in form of a Kubernetes Master NSX container image, runs in K8s as a Pod NSX Container Plugin (NCP) etcd Manager NSX API-Server Kubernetes NSX Manager • Kubernetes Adapter: NCP is built in a Adapter Infra API Client modular way, so that individual adapters can Scheduler be added for different CaaS and PaaS systems • NSX Infra layer: Implements the logic that creates topologies, attaches logical ports, etc. based on triggers from the Adapter layer • NSX API Client: Implements a standardized interface to the NSX API 

26 .Architecture Namespace Creation Workflow 1. NCP creates a ‘watch’ on K8s API for any Namespace events • NSX-T Kubernetes Master NSX 2. A user creates a new K8s Namespace NSX Container Plugin (NCP) etcd Manager NSX 3. The K8s API Server notifies NCP of the Kubernetes NSX API-Server Adapter Infra Manager change (addition) of Namespaces API Client Scheduler 4. NCP creates the network topology for the Namespace : a) Requests a new subnet from the pre- NSX / Kubernetes Topology configured IP block in NSX NS: foo NS: bar b) Creates a logical switch c) Creates a T1 router and attaches it to the pre-configured global T0 router d) Creates a router port on the T1 router, attaches it to the LS, and assigns an C C C C IP from the new subnet 

27 . K8s API Architecture Storage Kubernetes Worker (VM) Create Storage Class Stateful K8s vSphere Pod Kubelet Pod Cloud provider Create Persistent Vol Claim Redis DB Create Pod and Mount Volume Tools, Libs, SW K8s Vol RedisDB Name: thin-disk Podspecvolume-claim includes: Provisioner: Storage vSphere class: Persistent Volume thin-disk volume claim Diskformat: Accessmode: thin Filesystem mountreadwrite point Storage: 2GB vCenter vSphere dataVol.vmdk Datastore1 

28 . Architecture Sign Scan Harbor Build Push Trusted Image For kubectl Image Image Image CVEs apply Developer Project RBAC Project Image Project Notary Clair Repo Replic UAA AUTH ation 

29 .vRealize Ops, Log Insight For Comprehensive Visibility Launch in VMware vRealize Context VMware vRealize Operations Log Insight Capacity, Performance and Log analytics, aggregation, Configuration Management Events and search Structured Data Unstructured Data Metrics Alerts Events Logs Messages Virtual Applications