Benchmark Results for Linux 4.19 Kernel Spectre Mitigation on Various CPUs

Many people have been wondering the exact performance costs of the latest Meltdown and Spectre mitigations built into the Linux 4.19 kernel, which has been in development since earlier this month – with a lot of Spectre and Meltdown protection being added. Even though Spectre attacks still aren’t a huge threat due to their incredibly slow data extraction speeds, the Linux kernel devs aren’t taking any chances.

In any case, they’ve been adding Spectre and Meltdown mitigations to the Linux 4.19 kernel for x86_64, POWER, S390, and ARM architectures where they can be applicable. Since a lot of people have been wondering exactly how the mitigations will affect the performance of each architecture, we have some benchmarks for you.

For an overall look at the performance cost of the Spectre and Meltdown mitigation techniques used in Linux kernel 4.19, three Intel Xeon systems and two AMD EPYC systems, as well as a virtual machine on each side for seeing the average default performance of the Linux 4.19 kernel with mitigations applied, compared to an unmitigated kernel.

The Linux 4.19-rc1 development kernel released just this past weekend is the basis of these benchmarks across all the systems. For Intel, the relevant mitigations included page table isolation (PTI/KPTI) for Meltdown, as well as the various Spectre speculative execution mitigations, which include __user pointer sanitization, the full generic Retpolines via IBPB IBRS_FW, speculative store bypass disable via prctl and seccomp, and for L1TF/Foreshadow is PTE inversion and conditional cache flushes for VMs.

The Linux 4.19 kernel does not enforce its full mitigation via disabling Intel HT/SMT support, which is worth remembering if you run VMs and you feel like shady code/users have access to the VM. If you opt for the entire mitigation where SMT is disabled, the performance impact will be felt a lot more because it halves the number of threads available. Public cloud providers currently seem to be adjusting their scheduler to ensure no SMT threads will go across users, which avoids the need to disable Hyper Threading (and its subsequent huge performance cost).

As far as AMD EPYC goes, the default mitigations are only for their relevant vulnerabilities which utilize __user pointer sanitization for Spectre V1, the AMD Retpoline IBPB (Indirect Branch Predictor Barrier) for Spectre V2, and a speculative store bypass disable (SSBD) for Spectre V4.

In the benchmarks we’re showing, all of the configurations on the stock Linux 4.19-rc1 kernel were tested, followed by a repeat of the tests using the various run-time switches that disable the mitigations. The systems were tested with Ubuntu 18.04.1 LTS x86_64 with the Linux 4.19-rc1 kernel via the Ubuntu Mainline Kernel PPA, with the latest updated microcodes and BIOS, GCC 7.3, and the EXT4 file-system.

Here is a list of all the system configurations tested in the comparisons:

• The Intel Xeon E3-1280 v5 Skylake processor on an MSI Z170A SLI PLUS motherboard, 16GB DDR4, and 256GB Toshiba RD400 NVMe SSD.
• The Intel Xeon E5-2687W v3 Haswell processor on an MSI X299 SLI PLUS motherboard, 32GB DDR4, and a 80GB Intel 530 SATA 3.0 SSD.
• A dual Intel Xeon Gold 6138 setup running within a Tyan 1U chassis and having 96GB of RAM and backed by a Samsung 970 EVO NVMe SSD 256GB.
• A single KVM-based virtual machine running on the above-mentioned dual Xeon Gold server. This VM was the only active process on the system and configured to access 80% of the CPU cores/threads (64 threads), 48GB of RAM, and a 118GB virtual disk. During the VM testing when running mitigations disabled both the guest and host kernels had the mitigation switches disabled.
• The AMD EPYC 7601 processor on a Tyan 2U server with 128GB of RAM and a 280GB Intel Optane 900p NVMe SSD.
• A single KVM-based virtual machine running on the above-mentioned AMD EPYC 7601 server. The VM had access to 80% of the CPU cores/threads (52 threads), 48GB of RAM, and 120GB virtual disk.
• An AMD EPYC 7551 server using a Gigabyte MZ31-AR0 motherboard, 32GB of RAM, and a Samsung 960 EVO 256GB NVMe SSD.

The benchmarker noted that all of the hardware is quite different and is not intended to be compared in raw performance between these systems, but is instead meant to look at a broad spectrum of systems to gain an idea of the overall impact of the CPU vulnerability mitigations in the Linux 4.19 kernel.

Being so, the results of all the data was normalized against each system’s unmitigated performance for an easy look at the relative cost across the board. The source of these benchmarks is the Phoronix Benchmark Suite.

Mitigation Impact on Compilation

For these benchmarks, the focus is on relevant mitigations to Spectre and Meltdown where I/O is involved, or heavy kernel interactions which show the overall cost of the mitigation techniques. Workloads that run on the CPU and are not relying heavily upon the CPU cache, context switching, etc. really don’t show too much of a performance difference with workloads that involve CPU-based rendering in gaming, etc.

“As of the Linux 4.19 kernel, the default mitigations on the Intel CPUs cause the performance to come in 7~16% lower than an unmitigated kernel using the run-time switches. On the AMD side with having to worry about just the relevant Spectre vulnerabilities, its default configuration runs at 96~97% the speed of an unmitigated kernel rather than 84~93%.”

“It’s a similar story with the sub-test of reading the compiled tree where the Intel CPUs were coming in at 85~86% the speed of the unmitigated kernel while the AMD EPYC systems were at 94~96%.

In real-world compilation workloads like building out the Linux kernel, there ends up being about a ~2% performance cost.” 

Mitigation Impacts on Hackbench, PostgreSQL, and GIMP

“The Hackbench Linux kernel scheduler benchmark tends to show an impact from these mitigations as well. On the Intel hardware was a performance hit of around 20% except for an outlier in the bare metal dual Xeon Gold results. On the AMD EPYC systems there was no real performance penalty incurred from its relevant Spectre mitigations.

The PostgreSQL database server is one of the real-world workloads we’ve seen incurring performance hits this year as the various speculative execution vulnerabilities are mitigated. In this particular test run today, the Intel CPUs with mitigations applied were running at 93~95% the speed of an unmitigated kernel while the EPYC CPUs were at 99%.

The GIMP image manipulation program is another one of the programs we’ve seen where performance penalties can be incurred due to Spectre/Meltdown. On the Intel side it generally meant a 5~10% drop in performance while on the AMD EPYC systems was 0~2%.” 

Mitigation Impacts on Redis, NGINX, and Apache

For those running the Redis in-memory database, on Linux 4.19 with the Intel hardware there is a performance hit of up to 11% observed on the Intel Skylake E3 v5 system while for the other hardware was 5~7% lower. On the AMD EPYC systems this was 1~5%.

When running the NGINX web server, the performance impact on the tested Xeon systems with the Linux 4.19 kernel was up to 20% while on the AMD EPYC systems was just 1~2% of 6% on the EPYC virtual machine.

Likewise, with the Apache HTTP web server was also noticeably lower out-of-the-box performance on the Xeon hardware while the EPYC bare metal performance was unchanged and the EPYC VM just a few percent lower in its default configuration.

Mitigation Impacts on Thread, File, and Process Creation

“With the OSBench test of file creation speed, the Intel Xeon performance out-of-the-box was 84~87% the speed of an unmitigated kernel while the EPYC systems were at 91~94%.

With the thread creation test, there was also a measurable difference between the Intel and AMD processors.

Or in the time to launch programs and create processes, there was a smaller but still evident difference with the default Linux 4.19 kernel against the unmitigated configuration.

That’s where things stand today with the out-of-the-box performance cost of these CPU vulnerability mitigations with the Linux 4.19 kernel. Keep in mind if your system(s) are exposed to untrusted users/code, particularly in VMs, additional steps like l1tf=full may be needed where you may end up disabling SMT/HT or always enforce the L1 cache flushing rather than the conditional behavior, which will further lower the performance of the systems.”

The post Benchmark Results for Linux 4.19 Kernel Spectre Mitigation on Various CPUs appeared first on Appuals.com.