1. x Achieve stable high performance DPDK Application on modern CPU TAO YANG XUEKUN HU INTEL 

2. Agenda 1. Modern CPU Architecture 2. Performance impact on shared resource 1. Shared EU and L1/L2 Cache 2. Shared L3 Cache 3. Shared Core Power 3. Summary 2 INTEL CONFIDENTIAL Doc #xxxxx 

3.Server CPU in Hot Chips 2017 •https://www.hotchips.org/wp-content/uploads/hc_archives/hc29/HC29.22-Tuesday-Pub/HC29.22.90-Server-Pub/HC29.22.921-EPYC- Lepak-AMD-v2.pdf •https://www.hotchips.org/wp-content/uploads/hc_archives/hc29/HC29.22-Tuesday-Pub/HC29.22.90-Server-Pub/HC29.22.930-Xeon- Skylake-sp-Kumar-Intel.pdf •https://www.hotchips.org/wp-content/uploads/hc_archives/hc29/HC29.22-Tuesday-Pub/HC29.22.90-Server-Pub/HC29.22.942-Centriq- 2400-Wolford-Qualcomm%20Final%20Submission%20corrected.pdf 3 

4.Shared Execution Engine and L1/L2 Cache •Hyper-Threading Technology •Intel® 64 and IA-32 Architectures Software Developer Manuals 4 

5.Linux tool for hyper-thread •Hyper-thread and cores in the system [root@wolfpass-6230n ~]# lscpu CPU(s): 80 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 [root@wolfpass-6230n ~]# lscpu -e CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ 0 0 0 0 0:0:0:0 yes 3900.0000 800.0000 1 0 0 1 1:1:1:0 yes 3900.0000 800.0000 ...... 40 0 0 0 0:0:0:0 yes 3900.0000 800.0000 ...... •Binding application to core • taskset -c CORE-ID DPDK-APP 5 

6. Hyper-thread performance impact •DPDK L3fwd on 1 core/1 Thread (Intel Xeon 6230N 2.30GHz) with 2*10G port •Stress workload running on the other hyper thread DPDK L3fwd performance impact on hyper thread workload DPDK 1 Core 1 Stress-ng L3fwd 0.90 0.90 0.83 Thread2 0.81 Thread1 Intel Xeon 6230N 0.77 Intel Xeon 6230N 10GbE NIC Port Port 1 2 IXIA* Traffic Generator No hyper thread compute workload L1 Cache workload L2 Cache workload Socket workload io workload workload 6 

7.Shared L3 Cache •Intel® 64 and IA-32 architectures optimization reference manual 7 

8.Shared Memory Bandwidth •https://www.hotchips.org/wp-content/uploads/hc_archives/hc29/HC29.22-Tuesday- Pub/HC29.22.90-Server-Pub/HC29.22.930-Xeon-Skylake-sp-Kumar-Intel.pdf 8 

9.Intel® Resource Director Technology •Intel® 64 and IA-32 Architectures Software Developer Manuals •https://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html 9 

10.Linux Tools for Resource Control •Intel RDT Kernel Interface Documentation https://www.kernel.org/doc/html/latest/x86/resctrl_ui.html •Intel RDT Reference Software Package（Direct access CPU register） https://github.com/intel/intel-cmt-cat 10 

11. RDT Test configuration Memory “CAT with Cache Allocation Scheme Bandwidth Stress-ng VPP vRouter VPP vRouter Stress-ng Aggressors” Case VM3 Physical Allocation Scheme VM1 VM2 VM4 Process/VM Core Capacity Port Port Port Port Bit Mask 11 bit CBM representation 1 2 CoS 1 2 (CBM) Open vSwitch with DPDK 33,35,37 Other App 3 0xC 10 9 8 7 6 5 4 3 2 1 0 10% 60,73,75,77 CentOS 7.6 20 ovs-vswitchd 3 0xC 10 9 8 7 6 5 4 3 2 1 0 10% 21,26,27, OVS-DPDK 1 0x7F0 10 9 8 7 6 5 4 3 2 1 0 100% Intel Xeon 6230N 28,29,36 PMD 22,23,24 VM1 - SUT 1 0x7F0 10 9 8 7 6 5 4 3 2 1 0 100% Intel 710 Intel 710 10GbE 10GbE 30,31,32 VM2 - SUT 1 0x7F0 10 9 8 7 6 5 4 3 2 1 0 100% VM3 - Noisy Port Port Port Port 25,34,65,74 2 0x3 10 9 8 7 6 5 4 3 2 1 0 10% 1 2 1 2 Neighbor VM4 - Noisy 38,39,78,79 2 0x3 10 9 8 7 6 5 4 3 2 1 0 10% Neighbor IXIA* Traffic Generator 0-19,40-59 OS on CPU 0 0 0x7FF 100% pqos -e "llc:0=0x7ff;llc:1=0x7f0;llc:2=0x3;llc:3=0xc;" pqos -e "mba:0=100;mba:1=100;mba:2=10;mba:3=10;“ pqos -a "llc:0=0-19,40-59" pqos -a "llc:1=21,26,27,28,29,36,22,23,24,30,31,32;llc:2=25,34,38,39;llc:3=20,33,35,37" pqos -a "llc:1=61,66,67,68,69,76,62,63,64,70,71,72;llc:2=65,74,78,79;llc:3=60,73,75,77" 11 

12.RDT Test data 12 

13.Performance data with RDT OVS-DPDK/VPP vRouter performance throughput Mpps 28.2% 24.1% 23.5% 18.8% 23.4% 17.1% 18.9% 14.9% 11.9% 9.8% 64 128 256 512 1024 1280 1518 No Stress memory stress memory stress + cat + mba 13 

14.Shared Power •IntelSpeed Select Technology - Base Frequency •https://builders.intel.com/docs/networkbuilders/intel-speed-select-technology-base-frequency- enhancing-performance.pdf 14 

15. Intel® SST-BF Enabled CPU SKUs Base Config Intel SST-BF Configuration Base Configuration Intel SST-BF High Intel SST-BF Low Priority Parameters Priority Cores Cores SSE Base Freq SSE Base Freq SSE Base Freq Cores (GHz) Cores (GHz) Cores (GHz) Intel® Xeon® Gold 6252N Processor 24 2.3 8 2.8 16 2.1 Intel® Xeon® Gold 6230N Processor 20 2.3 6 2.7 14 2.1 Intel® Xeon® Gold 5218N Processor 16 2.3 4 2.7 12 2.1 Intel Network Builders NDA Partner Training 15 1 

16.Linux tool for ISS-BF • Enable the Intel® SST-BF feature in the BIOS. • OS can determine high priority cores by enumerating ACPI _CPC object’s “guaranteed perf” value for each core for scheduling purposes • Linux kernel v5.0.8+ exposes /sys/devices/system/cpu*/cpufreq/base_frequency • User space script to enable High/Low priority cores • https://github.com/intel/CommsPowerManagement 16 

17.SR-IOV Performance with ISS-BF L3fwd/SR-IOV Thoughtput DPDK 9.5% 6230N wo BF(1C, 2.3G) 6230N w BF(1C, 2.7G) •1 core in Xeon 6230N, 4 * 10G, SR-IOV passthrough, DPDK l3fwd in VM, packet size 64B 17 

18.OVS-DPDK Performance with ISS-BF •2*Intel Xeon 6230N + 6*10G in a system •Only used 16 cores in CPU 1 6 cores for OVS-DPDK data plane Low priority Core ID: ovs-vswitchd Core ID: 20 High priority Core ID: ovs-pmd: 21,26,27,33,34,36 3 cores for every VPP vRouter VM Low priority Core ID: VM 1: 22,23,24 VM 2: 28,29,30 VM 3: 32,35,37 •VPP VM core configuration: VM Core 0: for control plane VM Core 1,2: VPP data plane 18 

19.OVS-DPDK Performance with ISS-BF •Enable ISS-BF(VPP vRouter 2.1Ghz/OVS-DPDK 2.7Ghz) vs Disable ISS-BF(All Core 2.3Ghz ) OVS-DPDK/VPP vRouter Throughput (Mpps) 8.0% 9.5% 9.3% 8.6% 6.2% 64B 128B 256B 512B 1024B 1518B 6230N w BF(20C, 2.3G) w 3VM Total 16C 6230N w BF(20C, 2.7/2.1G) w 3VM Total 16C 19 

20.Summary •Many resources are shared in multi-core CPU. •Application running on different core compete for the shared resources. •Shared resource partition can reduce the competition and achieve stable high performance. 20 

21.Thank You !