Kubernetes 应用软件的网络服务质量——Jun Du,华为技术

流量定形是网络服务质量的一个重要方面,广泛用于您想要控制流量的诸多场景。在 Kubernetes 中,若带宽相关的注释已添加到 Pod 的元数据中,就会发生针对应用的流量定形。然而,目前仅 kubenet 支持在 pod 设置期间运用基本的流量定形,而 kubelet 在其运行于 CNI 网络驱动器中时并不支持。今后随着 kubenet 将受到轻视,我们需要完善有缺陷的部分。 本介绍将说明我们的实施,使用 linux tc 添加一个简易的 CNI 插件以创建入口(包括数据串)和出口(包括数据串)速率限制器。我们已将此 CNI 插件投入 CNI 社区中,由 Kubernetes SIG 网络主导的与 Kubernetes 集成工作正在进行中。

1.Network QoS Support for Kubernetes Applications Jun Du Huawei Cloud

2.Kubernetes Networking Overview Kubernetes Master Pod • One IP for each Pod Kubelet • CRI create network namespace • CNI initialize network device CRI network driver dockershim containerd kubenet CNI 初始化pause容器 p2p bridge flannel calico 网络namespace 初始化pause容器网络设备 解决容器跨机通信 加入pause容器网 络namespace Container A Pause A eth0 eth0 Pause B Container B

3.CNI: Container Network Interface • Standard of linux container network • Configure container interfaces using JSON • Two kinds of interfaces: - configure network – invoked when create container AddNetwork(net NetworkConfig, rt RuntimeConf) (types.Result, error) - clean up network – invoked when delete container DelNetwork(net NetworkConfig, rt RuntimeConf) error

4.Why Need Network QoS Support • For Users: - Applications should have the ~SAME performance in Cloud - Do NOT want to live with the noisy neighborhood • For Cloud Providers: - Need a way to isolate different tenants or applications - Need a way to handle network flood - Charge with different QoS level • For Kubernetes: - Better support for voice/video streams, IoT scenarios etc. - Handle well even if scheduling result is not pretty good - Part of multi-tenancy - Deprecate Kubenet

5.Basic Network QoS Implementations • QoS policy, management, and accounting functions to control and administer end-to-end traffic across a network • QoS identification and marking techniques for coordinating QoS from end to end between network elements • QoS within a single network element (for example, queuing, scheduling, and traffic-shaping tools)

6.QoS within a Network Element • Congestion control • Queue management • Link efficiency • Traffic shaping and policing

7.QoS in Linux with TC and Filters Glossary: SFQ: Stochastic Fairness Queueing HTB: Hierarchical Token Bucket

8. Linux TC Sample Commands export POD_IP= export DLD_CLASS=1 tc qdisc add dev cni0 root handle 1: htb default 30 tc class add dev cni0 parent 1: classid 1:${DLD_CLASS} htb rate 10Mbit tc filter add dev cni0 protocol ip parent 1:0 prio 1 u32 match ip dst ${POD_IP}/32 flowid 1:${DLD_CLASS} cni0 1: 1:1 # 10Mb/s 1:30默认

9. CNI Bandwidth Plugin • Configure Linux's Traffic control (tc) subsystem • Configures a token bucket filter (tbf) queuing discipline (qdisc) on both ingress and egress traffic • Creates an Intermediate Functional Block device (ifb) to redirect packets from the host interface eth0 ifb veth123 • Applies traffic shaping to interfaces created by previously applied plugins

10.Sample Config of Bandwidth Plugin { "cniVersion": "0.3.1", "name": "mynet", "plugins": [ { "type": “bridge", // can be ptp as well "ipam": { "type": "host-local", "subnet": "" }, }, { "name": "slowdown", "type": "bandwidth", "ingressRate": 123, "ingressBurst": 456, "egressRate": 123, "egressBurst": 456 } ] }

11.Integrating With Kubernetes • Kubelet runs with any network driver: cni or kubenet • Configure Pod’s annotations to limit ingress/egress bandwidth rate • For CNI: - Make sure bandwidth plugin binary exists in CNI plugins directory (/opt/cni/bin) - Configure enabling traffic shaping in network plugin config file (/etc/cni/net.d/10-caclico.confilist) apiVersion: v1 { kind: Pod "name": "k8s-pod-network", metadata: "cniVersion": "0.3.0", "plugins": [ name: iperf { annotations: "type": "calico", kubernetes.io/ingress-bandwidth: 1M "datastore_type": "kubernetes", kubernetes.io/egress-bandwidth: 1M ...... spec: }, containers: { - name: iperf "type": "bandwidth", image: moutten/iperf "capabilities": {"bandwidth": true} } ] }

12.Workflow of Limit Pod’s Bandwidth Pod.yaml annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M Kubernetes API Server Call CNI API: CNI plugin: Linux AddNetowrk() bandwidth tc CNI network driver Network plugin configlist tc qdisc Kubelet { tc class "type": "bandwidth", tc filter "capabilities":{"bandwidth":true} ... }

13.K8S Support Traffic Shaping in v1.12 Huawei team will keep contributing…

14.Demo https://asciinema.org/a/L60lAcHknt9BhdsMDNAr9oJ2q

15.Future Work • Burst rate support in Kubernetes API - Given the burst rate support in CNI bandwidth plugin side • Support traffic shaping in CNI third party plugins(calico, weave…) - weave already got a plan • Explore a way to re-configure when a Pod is running • If ALL network plugins supports traffic shaping, should we move it out of annotations? • More Flexible policies instead of static configuration • Windows Container traffic shaping?