基于 Kube-router 和 VPP-DPDK 的云原生网络解决方案——Hongjun Ni，英特尔

1 .A Cloud Native Networking Solution Based on Kube-router and VPP-DPDK Hongjun Ni Acknowledgement: Intel Ray Kinsella, Steve Liang @Intel Pierre Pfister, Jerome Tollet @Cisco Email: hongjun.ni@intel.com Rastislav Szabo @Cisco Foundation 

2 .About me • Focus on FD.io/VPP and Cloud Native Networking • VPP Maintainer (Load Balancer, VxLAN-GPE, NSH, GPTU, PPPoE) • Sweetcomb Project Lead • NSH_SFC Project Lead • Hc2vpp Committer • VPPSB Router Maintainer Foundation 

3 .Agenda • Cloud Native Networking and Challenges • Proposed Architecture • Why Choosing FD.io • Option 1: Load Balancer and Service Proxy • Option 2: Pod Networking and DSR • Option 3: Pod Ingress Firewall • Key Takeaway • Promote A New Project: Sweetcomb Foundation 

4 .What’s in Cloud Native Networking? Control Plane: • Assigns IPs (from a pool given to each workload) • Distributes routing information (i.e. how to get to this workload) • Distributes policy (e.g. who can connect to whom) Data Plane: For each packet to/from the workload: • Enforces policy • Forwards it to the right destination Reference: https://www.cncf.io/wp-content/uploads/2017/11/CNCF-Networking-Webinar-final-1-1.pdf / Foundation 

5 .Challenges on Present Solution Linux kernel solution: • Watches service and endpoints • Installs iptables/IPVS rules • Captures traffic and selects pod • Redirects traffic to chosen pods Problems: • Uses load balancing on iptables/IPVS • Uses NAT on iptables/IPVS • Communication via VETH • Performance degrades when iptables entries increase. Reference: https://kubernetes.io/docs/concepts/services-networking/service/ Foundation 

6 .Proposed Architecture • A turnkey solution on Kube- router, replacing Linux kernel’s networking stack. • Load Balancer and Service Proxy Running on VPP-DPDK • Policy based on VPP ACL • Integration with GoBGP • Routing based on VPP FIB Foundation 

7 .Existing Solution vs. Proposed Solution Existing Solution Proposed Solution Solution Stack Linux Kernel Stack Kube-router, VPP-DPDK Policy Enforcement Iptables + ipset VPP ACL Node Load Balancing Iptables, IPVS VPP kube-proxy Connection Tracking Iptables, IPVS VPP kube-proxy DNAT and SNAT Iptables, IPVS VPP kube-proxy Communication between Via VETH Via vhost-user or memif Host and Container External Load Balancer Via CSP’ Via VPP Load Balancer Load Balancer Routing Linux Kernel Networking GoBGP and VPP vRouter Foundation 

8 . Orchestration & Ligato Controller Data Plane Data-plane Management Analytics Networking trex Management Agent Honeycomb Sweetcomb Contiv-VPP -VPP Packet Processing Testing NSH_SFC PMA Tools Sandbox /Support Packet Processing CSIT DMM P4VPP GoVPP VPP trex Network IO OSV Packaging Operating Systems deb_dpdk rpm_dpdk Foundation 

9 .Option 1: Load Balancer and Service Proxy • Kube-router is deployed on each node and run as a service proxy, replacing Linux kernel kube-proxy. • Router, Load Balancer and Service Proxy could be implemented on VPP. • They run as typical K8s networking. Foundation 

10 .Option 1: Load Balancer on VPP-DPDK Load Balancer K8s Node 1 • Distributes traffic among K8s nodes Node IP: a.a.a.a • Consistent Hashing ensures e resilience to K8s node changes. Load Connection t Tunnel balancing Tracking h • Connection Tracking supports connection persistence. K8s Node N Node IP: • Supports two encapsulation types VPP-DPDK b.b.b.b  GRE tunnel  IPIP tunnel Foundation 

11 .Option 1: Service Proxy on VPP-DPDK K8s-node Backend Pod 1 Backend Pod 2 • Distributes traffic among Pods Pod IP: x.x.x.x Pod IP: y.y.y.y • Supports two interface types virtio-user memif virtio-user memif  vhost and virtio-user vhost memif vhost memif  memif • Supports three service types DNAT  ClusterIP  NodePort Connection  External LoadBalancer SNAT VPP-DPDK Tracking Load balancing eth Foundation 

12 .Option 1: Service Proxy Data Plane Internals • Ingress IPv4 traffic goes through lb4-nodeport, lb4-nat4 graph nodes. • Egress IPv4 traffic goes through nat4-in2out graph node. • IPv6 traffic has similar data path as IPv4. Foundation 

13 .Option 1: Flow Path • VPP Router enables ECMP feature. • VPP Load Balancer distributes traffic and encapsulates packets via GRE tunnels. A specific flow will be sent to the same K8s node. • On K8s node, it removes GRE tunnel and goes through Service Proxy and performs DNAT, and then distributes traffic to selected pods. • Return traffic will also pass through Service Proxy performing SNAT. • Pod IPs are not visible to VPP LB. Foundation 

14 .Option 2: Pod Networking and DSR • Kube-router is deployed on each node and run as a vRouter. • Each K8s node runs iBGP and peers with LB and rest K8s nodes. • Pod IPs are visible to VPP LB and rest K8s nodes. Foundation 

15 .Option 2: Flow Path • VPP Router enables ECMP feature. • VPP Load Balancer selects pod as per Service IP and encapsulates packets through IPIP tunnels. • On K8s node, vRouter just routes IPIP traffic to selected Pod. • On each Pod, it removes IPIP tunnel, manipulates packets, swaps source IP and destination IP, and then sends packets directly to clients (Direct Server Return). Foundation 

16 .Option 3: Pod Ingress Firewall K8s-node Pod 1 Pod 2 Pod 3 Pod N memif memif memif memif • Leverage VPP ACL feature. memif memif memif memif • Convert ingress and egress rules into Local ACL Local ACL Ingress+egress policy Local ACL per-pod ingress and egress ACLs Ingress+egress policy Ingress+egress policy common for Pod2, Pod3 (local tables), each assigned to a memif interface connecting to a pod. Packet Processing DNAT Routing • A single egress ACL (global table) assigned to interfaces connecting VPP-DPDK Global ACL Host VPP1 rest cluster node, the tunnel and the Egress policy for memif memif interface connecting other node-outbound traffic VPP instance. Cluster Internet GRE Tunnel SNAT • Pods with the same policy configuration share the same ACL. Eth interface NIC Foundation 

17 .Key Takeaway • A solution to enable high performance K8s networking. • Kube-router provides operational simplicity and high performance. • VPP-DPDK implements Load Balancer and Service Proxy. • Kube-router handles Pod networking with direct routing on VPP-DPDK. • Converts Network Polices to ingress and egress rules on local and global ACL. Foundation 17 

18 .Promote a New Project: Sweetcomb Foundation 

19 . Thank you ! Q&A Email : hongjun.ni@intel.com Foundation 19