GCP Narrowly Edges Past AWS, Azure in Cockroach Labs 2021 Report

Cockroach Labs’ 2021 cloud report found that competition between cloud computing’s three industry heavyweights  — Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) — is no longer an equal playing field. Cockroach Labs’ 2021 cloud report found that competition between cloud computing’s three industry heavyweights  — Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) — is no longer an equal playing field. In the 2020 report, all three major cloud providers showed relatively equal performance. The 2021 report reveals that each provider comes out on top for at least some of the benchmark criteria tested. “Declaring a winner was much harder to declare than in years past,” according to the authors of the report, as the gap on most metrics was razor thin. Where GCP received top marks for running online transactional processing workloads, AWS is considered to be the best value for the money, while Azure ultra disks stood out as best in storage. Testing was conducted on both the newest and previously top-performing machine types available from AWS, Azure, and GCP on three axes: CPU, network, and storage I/O and over 1,000 benchmark test runs across 54 machine types including CPU, network throughput, network latency, storage read performance, and storage write performance. Performance was measured through the industry standard online transaction processing (OLTP) database, specifically focusing on the TPC-C OLTP benchmark tool that simulates e-commerce business. The TPC-C benchmark tool also provides a measure of objectivity by simulating transactional database applications.  What’s inside each machine matters. Therefore, each cloud’s CPU performance was evaluated using CoreMark, an open-source, cloud-agnostic, benchmark that tests against various real-world workloads like list sort and search. Cockroach Labs reported the average number of iterations for both the single-core and the 16-core results. On the CPU front, Cockroach Labs found Intel processors had the best performance on single-core runs, but wavered on 16-core tests. AWS’ custom-built Graviton2 Processor, which uses a 64-bit ARM architecture, reigned supreme. Network benchmark testing was split into two tests: network throughput and network throughput latency. GCP for the third year in a row won network throughput by a landslide insomuch that it delivered nearly triple the throughput of either AWS or Azure. The kicker: GCP’s worst performing machines outpaced each network’s top performing machines despite not having an advanced disk option available. AWS took home the top spot in network latency for three years running. Its top-performing machine’s 99th percentile network latency was 28% and 37% lower than Azure and GCP, respectively. However, Cockroach Labs noted that there is possible randomness in the physical distance between instances. Unlike years past, Cockroach Labs opted to omit the local solid-state drives (SSDs) and make network-attached disks the focus of the report. The authors of the report note that the primary motivation for the focus relates to its usage of network-attached disks in CockroachCloud. Each of the clouds offers what Cockroach Labs designated an “advanced disk” – a more expensive disk for applications and workloads that require higher performance: AWS’s io2, Azure’s ultra disk, and GCP’s extreme-pd – which was not available at the time of testing. At the top end of the storage race, Azure’s storage I/O performance narrowly edged past GCP and AWS when running what Microsoft calls “ultra disks,” SSDs specialized in high IOPS and low-latency performance.  Azure’s write IOPS made up enough ground from its performance last year to take the top spot from AWS. However, Azure was the least cost-efficient of all providers. That being said, “Azure’s ultra-disks are worth the money,” the authors concluded.  TPC-C was also measured for throughput, throughput-per-minute (tpm) performance, and the total number of warehouses supported. According to the report: “On all things throughput, GCP is king.”  In the benchmark simulation of real-world OLTP workloads, Cockroach Labs concluded that for overall OLTP performance, advanced disks may not be necessary as cheaper machines with general purpose disks won for both AWS and GCP.  In fact, GCP’s top-performing machine for storage I/O read and write achieved only 5% fewer read IOPS than Azure and AWS’s top-performing machines.