Multiple Networks and Isolation in Kubernetes

Kubernetes currently only supports one network interface per pod, and the entire cluster has one flat network plane. In this presentation, we will share our implementation on enabling multiple networks and network isolation for NFV customers. Which includes physical network abstraction to enable the ability for pods to select physical network, and logical network for users to define network namespace and isolation. In addition, we enabled multiple plugin support at runtime, Kubernetes cluster can be deployed without binding to any specific network plugin, pods can choose network plugin in their own spec.

1. Multiple Networks and Isolation in Kubernetes Haibin Michael Xie / Principal Architect Huawei

2.Agenda • CNI and network plug-ins • Multiple network use cases, design and implementation • Network multi-tenancy requirement and implementation • Demo

3.CNI and Network Plug-ins • What is CNI – Common container network interface specification and libraries for writing plugins to configure network interfaces in Linux containers. • Assign IP to pod – Kubelet startup parameter --network-plugin=cni – --pod-cidr for pod IP addresses – Network plugin assigns one IP from the CIDR to each pod • Many third party network plugins –

4.Definition of Multiple Networks  Multiple physical networks  Multiple logical networks  Multiple network interfaces per container  Multiple network address spaces per cluster  Multiple network tenants per cluster  … Kubernetes Cluster Physical Network 1 Physical Network 2 Logical Network Logical Network Logical Network Logical Network A B C D Pod Pod Pod Pod

5.Multiple Networks Node1 Node2 Load Balancer Load Balancer Service1 Service2 Service1 Service2 eth0 eth1 eth0 eth1 flannel0 iCan0 flannel0 iCan0 veth0 veth1 veth0 veth1 veth3 eth0 eth0 SrcPod DestPodA DestPodB eth1 eth1 eth1

6.Why Multiple Networks  Logical network abstraction • IP space, quota/speed, network policies  Multiple network tenants • Physical isolation and logical isolation  Use multiple network solutions  User scenarios: • NFV: access to control plane, data plane and monitor plane • Applications that want to separate different traffic such as video streaming application • IPV6 co-existing with IPV4 • Applications have both internal and public access • Servers that want to isolate traffic from multiple clients • Utilizing multiple physical NICs on host

7.Changes to Kubernetes • New physical network object • New logical network object • Pod object with multiple networks • Service in specific logical network • Network based scheduling • Network tenancy – isolation, bandwidth, QPS limiting etc

8.Multiple Network Workflow Physical network Logical network LogicalNetwork.yaml apiVersion: v1 PhysicalNetworks.yaml kind: Network { ETCD metadata: "provider": [ name: management { name: phy_net0, labels: description: ***}, { name: phy_net1, 1 1 spec: physicalNet: phy_net0 description: ***} plugin: Flannel|iCan ApiServer subnet: ] } 2 Service.yaml 3 pod.yaml kind: Service 2 spec: apiVersion: v1 containers: metadata: Scheduler - image: test-webserver name: my-service name: test-container annotations: metadata: network: management CNS Master (iCan master, annotations: spec: flannel master …) networks: selector: 4 management:eth0 data:eth1 app: MyApp ports: Master 2 Register node Slave HostPhysicalNetwork.yaml kubelet 5 CNI Genie { "provider": [{ AND AND AND alias: phy_net0, ref_nic: eth1, },{ alias: phy_net1, Network plugins ref_nic: eth2, } }] iCan flannel …

9.Network Tenancy Requirements • Network isolation among tenants – Limit access to other tenants’ containers/services – Limit access to host network – Limit access to other tenant’s network resources like load balancers and DNS records • Network connectivity – Containers have internet access – Allow services to have external IP for ingress – Access other tenants’ containers/services

10.Network Tenancy How • Logical network, Kubernetes namespace and tenant mappings • Network isolation: – Physical isolation – IPTables – VLAN/VXLAN • DNS isolation – access control, dedicated DNS • Gateway for ingress/egress • Misc: – NodePort? – Support multiple namespaces and/or multiple logical networks in one tenant – Network based scheduling – Network quota allocation – Tenancy in federated clusters, cross data center or region

11.CNI-Genie • Multiple physical and logical networks • Adaptor to any network plug-in • Network isolation with policy • Admission control: validation, access control, scheduling • SLA monitoring and enforcement

12. Example Usage List of slave nodes Node description List of Physical Networks List of Logical Networks

13.Example Usage Deploy pod

14.Example Usage Query pod

15.More? Code repository: PaaS/CNI-Genie/ Watch demo videos: - Physical network and logical network: - Multiple IPs per pod: - - Co-existence of multiple plugins: - - CNI-Genie admission control: - Network policy controller:

16. Thank you Haibin Michael Xie wechat: 153346957