Gap Narrows Between Azure, AWS, GCP in Cockroach Labs 2020 Report

Cockroach Labs’ 2020 cloud report found that competition in the cloud is alive and well between the three major public cloud providers — Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) — with the playing field more equitable than ever before.  Cockroach Labs’ 2020 cloud report found that competition in the cloud is alive and well between the three major public cloud providers — Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) — with the playing field more equitable than ever before.  The report measured AWS, GCP, and Azure, using a series of tests and customer-like workloads to highlight the performance trade-offs associated with each provider. Aided by new and improved machine types, GCP closed in on AWS. “GCP has made noticeable improvements in the TPC-C benchmark such that all three clouds fall within the same relative zone for top-end performance,” Cockroach Labs noted in a blog post. Testing was conducted on both the newest and previously top-performing machine types available from AWS, Azure, and GCP on three axes: CPU, network, and storage I/O and over 1,000 benchmark test runs across 26 machine types including CPU, network throughput, network latency, storage read performance, storage write performance, and TPC-C. Performance was measured through the industry standard online transaction processing (OLTP) database, specifically focusing on the TPC-C OLTP benchmark tool that simulates e-commerce business. The TPC-C benchmark tool also provides a measure of objectivity by simulating transactional database applications.  Cockroach Labs shared the initial results of its study with all three cloud providers prior to publication and incorporated their respective feedback to help improve Cockroach Labs’ methodology and benchmarks, Andy Woods, product manager at Cockroach Labs, explained in an interview with SDxCentral.  “This is like a peer review process and a scientific paper,” Woods added. “We worked with them to get the most fair and accurate benchmarks possible.” While the providers run many of the same benchmarking tests internally to measure and monitor improvements, each public cloud provider is still incentivized to show themselves off in the best possible light. That’s why, as Woods explained, “having somebody as a third party to make those [benchmarking] choices and apply them equally across all three clouds, helps consumers have trust that they can actually make these comparisons fairly.“ And because these companies are constantly adopting new hardware, it’s important to note that cloud providers’ performance can change drastically and quickly as a result. On the CPU front, the best performing Azure machines achieved significantly better results on the CPU micro benchmark, according to Cockroach Labs. “The largest difference between the CPUs on each cloud was that even though all machines have 16 vCPUs, the top performing Azure machines use 16 cores with 1 thread per core. The other clouds use hyperthreading across all instances and use 8 cores with 2 threads per core to achieve 16 vCPUs,” the company wrote in its report. Network benchmark testing was split into two tests: network throughput and network throughput latency. GCP’s network throughput outperformed AWS and Azure, insomuch that, “not only do [GCP’s] top performing machines beat each network’s top performing machines but so do their bottom performing machines.”  Although GCP significantly improved its network latency since the last version of this report, the authors noted, AWS still reigned supreme in network throughput latency. And as for Azure, even its top performing machines were more than five times worse than AWS or GCP on latency. Cockroach Labs also tested locally attached storage and network attached storage to measure throughput and latency. I/O was tested using a Sysbench configuration — an open-source benchmarking tool — to measure how much data could be written from storage per second. Read and write capabilities were measured separately.  AWS was victorious in the write round with “superior write storage performance with the i3en machine type.” Whereas AWS and GCP hit a bottleneck at four threads, Azure continued to increase write input/output operations per second (IOPS) until 16 threads, giving it an advantage over the other two providers in its ability to manage threads.  Despite excelling at managing applications with more threads, Azure write IOPS came up short behind AWS, according to the report. Azure’s read throughput was also similar to its write throughput and couldn’t outperform AWS on read throughput due to an extremely variable read latency. Thus, the report found that AWS won the combined storage read comparison across all categories with its i3en machine type. TPC-C, which was measured for throughput, throughput-per-minute type C (tpmC) performance, and the total number of warehouses supported. Cockroach Labs concluded that all three clouds were within 5% of each other but AWS came out on top again, although with a narrowed lead. In last year’s report, AWS outperformed GCP by about 40% thanks in part to AWS’s Nitro System. “GCP shows dramatic improvement in the 2020 Cloud Report edging out AWS and Azure on price per performance of TPC-C but slightly underperforming AWS and Azure on max tpmC available on a three node cluster,” Cockroach wrote in the report. “You really can get a good performance on Azure, AWS and GCP,” Woods said. “It was really exciting for us to see the new machine types performed so well because it really validated our hypothesis that there is real value in not being locked into a specific vendor.”