深度学习：SIG Cluster生命周期 - Di Xu, Ant Financial 及 Alexander Kanevskiy, Intel

1 .SIG Cluster Lifecycle: Deep Dive Di Xu & Alexander Kanevskiy 2018-11-15 

2 .Who Are We? Di Xu Alexander Kanevskiy Kubernetes Member Kubernetes Member Top 50 Code Contributor to K/K Open Source Technology Center Ant Financial Intel @dixudx @kad 

3 .Our Mission SIG Cluster Lifecycle’s objective is to simplify creation, configuration, upgrade, downgrade, and teardown of Kubernetes clusters and their components. 

4 .SIG Cluster Lifecycle Projects • kubeadm • cluster-api • kops • kubespray • minikube • bootkube • kubeadm-dind-cluster • ... 

5 .Agenda • kubeadm • Best practices: install and fine-tuning • How it works • Current status and plans • Cluster API • Join and contribute! 

6 . kubeadm Best Practices 

7 .Installing kubeadm Ways of installing kubeadm: What is actually needed on the node: • Supported distributions • kubeadm • Container Linux • kubelet • DEBs and RPMs: • kubectl Ubuntu, Debian, Hypriot, RHEL, CentOS, Fedora • CNI plugins • There are other distros • cri-tools • OpenSuSE, ArchLinux, … For distros without official support you can use Container Linux section for manually install binaries https://kubernetes.io/docs/setup/independent/install-kubeadm/ 

8 .kubeadm: fine-tuning kubelet • systemd • /etc/systemd/system/kubelet.service.d/10-kubeadm.conf • Don’t edit it! • Override a instance-specific kubelet flag via environment file • /etc/default/kubelet or /etc/sysconfig/kubelet • KUBELET_EXTRA_ARGS=”--flag=…” • ConfigMap in “kube-system” namespace • “kubelet-config-1.X” • kubeadm-managed files • /var/lib/kubelet/kubeadm-flags.env • /var/lib/kubelet/config.yaml 

9 .kubeadm: Offline installation • Offline and air-gapped detection $ kubeadm config images list improved in v1.12 • k8s.gcr.io/kube-apiserver • k8s.gcr.io/kube-controller-manager • “Calling home” • k8s.gcr.io/kube-scheduler • --kubernetes-version • k8s.gcr.io/kube-proxy • stable, stable-1, stable-1.12, ... • k8s.gcr.io/pause • latest, latest-1, latest-1.12, ... • k8s.gcr.io/etcd • k8s.gcr.io/coredns • ci/latest-1.13 • upgrade plan $ kubeadm config images pull https://dl.k8s.io/release/${RELEASE}/bin/linux/amd64 $ gsutil ls -l gs://kubernetes-release/release/v1.12.2/bin/linux/amd64/ 

10 . kubeadm: Network and Proxies DMZ • Proxy for CRI HTTP(S) • HTTP_PROXY Internet Proxy • HTTPS_PROXY • NO_PROXY • Attention: local registries Firewall • Proxy for kubeadm • HTTP_PROXY • HTTPS_PROXY • NO_PROXY • Node IPs range • Service IPs range Node Node Node • POD IPs range • Cluster domains NO_PROXY=tld.com,192.168.0.0/16,10.0.0.0/8,cluster.local LAN 

11 .kubeadm: Using own certificates • Directory with all your custom certificates (--cert-dir) • etcd serving: etcd/ca.key and etcd/ca.crt • Cluster CA: ca.key and ca.crt • API Server serving: apiserver.key and apiserver.crt • ServiceAccount signing key: sa.pub and sa.key • API Server Kubelet client: apiserver-kubelet-client.key and apiserver-kubelet-client.crt • Frontend Proxy -- for API Aggregation • CA: front-proxy-ca.key and front-proxy-ca.crt • Client: front-proxy-client.key and front-proxy-client.crt 

12 .kubeadm: Using ComponentConfig ● Why use a config file for configuring kubeadm? ○ Tired of long complicated flags ○ Wants more advanced features, including more customized parameters ○ The API spec in v1.12 is unstable, but there’s always an upgrade path ○ The API spec is graduated to v1beta1 in v1.13 ● Supported Kinds in config file ○ InitConfiguration: kubeadm init master-local runtime config ○ ClusterConfiguration: cluster-wide settings, wide range of parameters ○ KubeletConfiguration: cluster-wide kubelet options ○ JoinConfiguration: kubeadm join node-local runtime config ● Saved to ConfigMaps for future upgrading ○ “kubeadm-config” and “kubelet-config-1.x” in “kube-system” 

13 .kubeadm: InitConfiguration kubeadm config print-default --api-objects InitConfiguration apiVersion: kubeadm.k8s.io/v1beta1 • Usage kind: InitConfiguration • “kubeadm init --config …” localAPIEndpoint: advertiseAddress: "10.100.0.1" bindPort: 6443 nodeRegistration: • Why criSocket: "/var/run/crio/crio.sock" • Custom API endpoint address kubeletExtraArgs: • Specify init bootstrap tokens cgroupDriver: "cgroupfs" • Pass custom kubelet flags bootstrapTokens: • Set node name/taints ... 

14 .kubeadm: ClusterConfiguration kubeadm config print-default --api-objects ClusterConfiguration apiVersion: kubeadm.k8s.io/v1beta1 • Usage kind: ClusterConfiguration • “kubeadm init --config …” kubernetesVersion: "v1.12.2" imageRepository: registry.example.com networking: serviceSubnet: "10.96.0.0/12" • Why dnsDomain: "cluster.local" • Fine tune cluster defaults etcd: ... • Custom args and volume apiServer: mounts to control plane extraArgs: components ... extraVolumes: ... 

15 .Kubeadm: KubeletConfiguration kubeadm config print-default --api-objects KubeletConfiguration • Usage • “kubeadm init --config …” apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cpuManagerPolicy: static • Why failSwapOn: false • Fine tuning kubelet parameters maxPods: 110 resolvConf: /etc/my-cluster-resolv.conf 

16 .Kubeadm: JoinConfiguration kubeadm config print-default --api-objects JoinConfiguration apiVersion: kubeadm.k8s.io/v1beta1 • Usage kind: JoinConfiguration • “kubeadm join --config …” nodeRegistration: criSocket: /var/run/crio/crio.sock name: k8s-node2.example.com kubeletExtraArgs: • Why cgroupDriver: "cgroupfs" • Customize discovery options discovery: • Pass custom kubelet flags bootstrapToken: • Set node name/taints apiServerEndpoint: kube-apiserver:6443 token: abcdef.0123456789abcdef 

17 .Kubeadm: KubeProxyConfiguration kubeadm config print-default --api-objects KubeProxyConfiguration • Usage • “kubeadm init --config …” apiVersion: kubeadm.k8s.io/v1beta1 kind: KubeProxyConfiguration iptables: masqueradeAll: false • Why masqueradeBit: 14 • Fine tuning kubeproxy syncPeriod: 30s parameters ipvs: syncPeriod: 30s 

18 .kubeadm: Managing “addons” • Installed and configured by kubeadm • DNS • CoreDNS has already been marked as default starting from 1.12 • kube-dns can be used via feature-gate • kube-proxy - deployed as a DaemonSet • Anything that isn’t required for meeting the Conformance criteria is outside of kubeadm scope • Dashboard • CNI plugins • Future: Cluster Bundles… 

19 . kubeadm Under the hood 

20 .kubeadm: Atomic work “phases” preflight Run pre-flight checks kubelet Commands related to handling the kubelet. certs Generates certificates for a Kubernetes cluster kubeconfig Generates all kubeconfig files for the control plane and the admin kubeconfig file controlplane Generates all static Pod manifest files necessary to establish the control plane kubeconfig Generates all kubeconfig files for the control plane and the admin kubeconfig file upload-config Uploads the currently used configuration for kubeadm to a ConfigMap mark-master Mark a node as master bootstrap-token Manage kubeadm-specific bootstrap token functions addon Installs required addons for passing Conformance tests 

21 .kubeadm init Generate Wait for static Pod Run Generate the Configure Generate Manifests Preflight KubeConfig Control Kubelet Certificates for the Checks Files Plane to Control be healthy Plane Upload Install Setup the RBAC Generate a (by Taint and kubeadm & DNS and Authorization default random) label the kubelet Proxy System Bootstrap Token master config to a Addons ConfigMap 

22 . kubeadm join --experimental-control-plane Generates Runs Checks static Pod Applies Kubeadm Preflight if the Cluster Manifests Master Init Checks Configuration for new Label and Preflight Supports Control Taints Check Plane Run Specific Fetches Bootstraps Uploading Checks Init Kubelet Currently Used base on the Configuration InitConfiguration Init Configuration --experimental-control-plane 

23 .kubeadm upgrade: Control Plane Upgrades Checks Gets the Enforces the Upgrade Preflight if the configuration all control RBAC cluster from the version Checks plane rules and is "kubeadm-config" skew Static addons healthy ConfigMap policies Pods 

24 .kubeadm upgrade: Node ● Currently kubeadm only supports upgrading the kubelet configuration, NOT the kubelet itself. ○ Manually upgrade the kubelet package using your package manager (e.g. apt, yum) ● Kubeadm will download ConfigMap “kubelet-config-1.x” in the “kube-system” namespace for global cluster kubelet settings ○ Will be stored as “/var/lib/kubelet/config.yaml” ○ Require kubelet version v1.11+ 

25 . kubeadm Status and Roadmap 

26 .kubeadm 1.12 Changelog • Config / ComponentConfig • Experimental: join master • Improved CRI support • Improved offline / air-gapped support • Deprecating of SelfHosting, HA 

27 .kubeadm 1.13 Roadmap • kubeadm is graduated to GA/stable!!! • Promote the config file from v1alpha3 to v1beta1 • Phases graduated to beta • addon, bootstrap-token, certs, control-plane, etcd, kubeconfig, kubelet-start, mark-control-plane, preflight, upload-config • Bug fixes 

28 .Cluster API 

29 .What is the Cluster API? Cluster Declarative API Machine Machine Set + Machine Deployment Machine Lifecycle (including Provisioning) Common Logic Machine Upgrade Infrastructure platform (vSphere, GCP, AWS, etc.) Pluggable Architecture Support for various Operating Systems Cluster Bootstrapping, Upgrade Tooling, Services Auto-scaling, Repair, Node Auto-provisioning