Más contenido relacionado La actualidad más candente (20) Similar a Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure) (20) Más de Revelation Technologies (20) Compute Cloud Performance Showdown: 18 Months Later (OCI, AWS, IBM Cloud, GCP, Azure)1. Chicago Oracle Users Group
April 15, 2021
7:00pm EST
Compute Cloud Performance Showdown: 18 Months Later
Oracle Cloud, Amazon Web Services, IBM Cloud, Google Cloud, Microsoft Azure
3. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 3 of 58
@Revelation_Tech
About Me
• Master’s degree in Computer Science from George Mason University
• Recent emphasis on cloud, DevOps, middleware, and security in current projects
• Oracle ACE, OCE, OCA
• Author, Blogger, Presenter
• @Ahmed_Aboulnaga
Sponsor
5. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 5 of 58
@Revelation_Tech
Disclaimer
• The results of the test results documented in this presentation should not be considered
definitive
• A lot of details surrounding setup, configuration, and assumptions regarding the test cases
are not documented in this presentation
• Results vary with repeated testing:
‒ Ongoing/unknown backend and hardware changes at each provider
‒ Varying load on backend hardware due to multitenancy
‒ Lack of backend access makes interpreting results difficult at times
• Testing limitations exist (see next slides)
6. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 6 of 58
@Revelation_Tech
Objective
• Conduct 2nd round of tests to compare compute cloud (i.e., IaaS) performance across 5
cloud providers against comparable medium-sized instance types:
‒ Amazon Web Services (AWS)
‒ Oracle Cloud Infrastructure (OCI)
‒ IBM Cloud
‒ Google Cloud Platform (GCP)
‒ Microsoft Azure
• Compare performance of the following:
‒ Linux Host
‒ Oracle WebLogic Server 14c
‒ Oracle Database 19c
• Compare results to 2019 tests
7. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 7 of 58
@Revelation_Tech
Testing Limitations
• Tested on single virtual machine (i.e., did not recreate instances and retest)
• Did not test different instance types
• Did not test across different data centers
• All virtual machines are multitenant-based (i.e., no dedicated hardware)
• All instances and software configured exactly identically*, and no tuning performed
8. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 8 of 58
@Revelation_Tech
Summary of Results | January 2019
• Similar tests were conducted in January 2019
‒ At the time, difficult to perform apples-to-apples comparison of cloud providers
‒ Nothing alarming in the performance results; more powerful CPUs yielded better performance
• Linux Host performance:
‒ Amazon Web Services had the slight processing edge due to newer and higher-end CPU model
• Oracle WebLogic Server performance:
‒ Microsoft Azure slightly underperforms compared to the other providers
• Oracle Database performance:
‒ Microsoft Azure consistently had the poorest throughput and performance
9. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 9 of 58
@Revelation_Tech
Findings from Performance Tests | October 2020
▪ IBM and Azure have slight edge on CPU
▪ OCI completely dominates on I/O (again)
▪ Unacceptable I/O from AWS, IBM, and GCP
▪ OCI has the best performance-to-cost value
▪ Azure slightly outperformed the others
▪ OCI slightly underperformed relative to the others
▪ GCP has the slight edge here
▪ AWS dips compared to 2019 and freezes towards end of test
▪ Azure demonstrates notable improvements, but still lags
behind the others
Interpret results with caution!
10. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 10 of 58
@Revelation_Tech
Concluding Our Performance Tests | What We Learned
Difficult to conclusively determine a leader
Specifications (expectations) ≠ Performance (reality)
Inconsistency in performance is likely attributed to multitenancy
IBM Cloud has improved in terms of stability and service offerings
Google Cloud Platform, Microsoft Azure, and IBM Cloud have improved in
performance
No competition to Oracle Cloud Infrastructure in terms of cost and overall value
Amazon Web Services remains the gold standard that all other CSPs are trying to catch up
to, but there are compelling reasons to consider other providers (specifically Oracle Cloud
Infrastructure and Google Cloud Platform).
11. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 11 of 58
@Revelation_Tech
COMPARING THE 5 CLOUD PROVIDERS
17. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 17 of 58
@Revelation_Tech
AWS Experiences
• EC2 instance startup/shutdown second slowest to OCI
• Remains the gold standard that all other CSPs are trying to catch up to
18. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 18 of 58
@Revelation_Tech
OCI Experiences
• Took much longer to startup/shutdown than other providers
– Never experienced this with other tenancies
• Remains the lowest cost of all providers and has the best overall value
19. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 19 of 58
@Revelation_Tech
IBM Experiences
Improvements
• No longer forces you to use pay for a hardware firewall using PayPal
• Now allows full firewall management via Security Groups
Odd
• Consistently takes 12 seconds for Linux password prompt to appear
• VMs default to America/Chicago CDT timezone despite data centers in
Northern Virginia (all others default to UTC or GMT)
• Large number of interfaces on instance
Cons
• Consistently overcharges in monthly invoices; was informed this is due to
merging of billing systems (but they never undercharge)
• Restart required when attaching a new Security Group to a VM*
• Only provider that does not disable ‘root’ login
20. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 20 of 58
@Revelation_Tech
GCP Experiences
• Only provider to always warns you about data loss before shutting down (ask yourself why)
• Firewall rule names can only have hyphens and lowercase letters
• Only provider that recommended increasing hardware after a series of stress tests
21. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 21 of 58
@Revelation_Tech
Azure Experiences
• Offers Oracle Linux images
• Frequently renamed disks (alternated /dev/sdb with /dev/sdc between reboots)
22. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 22 of 58
@Revelation_Tech
Other Observations – Shutdown Times
• AWS and OCI take much longer to shutdown the OS and poweroff the VMs than the others
• Results have been relatively reproducible
• Have not experienced these long shutdown times in other accounts or in 2019
OS Shutdown
Duration
Full VM Poweroff /
Deallocation
AWS 4:05 minutes 4:31 minutes
OCI 15:40 minutes 15:40 minutes
IBM 0:09 minutes 8:08 minutes
GCP 0:05 minutes 0:38 minutes
Azure 0:05 minutes 4:22 minutes
23. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 23 of 58
@Revelation_Tech
CPU/Memory Cost Comparison
Instance Type vCPUs RAM
Cost
(per hour)
m5.4xlarge 16 64 GB
$0.8980
$0.7680
VM.Standard2.8
VM.StandardE2.8
VM.Standard2.8
16
16
16
120 GB
64 GB
120 GB
$0.5104
$0.2400
$0.5104
B1.16x64 16 64 GB
$0.7722
$0.7680
(Custom) 16 64 GB
$0.9183
$0.6037
D16s_v3
D16as_v4
16
16
64 GB
64 GB
$0.8980
$0.7680
* CPU/memory cost only (excludes storage, firewall, static IPs, load balancers, etc.)
$647
$367
$556
$661 $647
$553
$173
$553
$435
$553
AWS OCI IBM GCP Azure
Cost Per Month
CPU/memory only, based on official pricing sheets, not actuals
2019 2020
* Values with strikethrough were previously used in 2019 tests
24. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 24 of 58
@Revelation_Tech
Virtual Machines Specifications
• Some variation in instances configuration compared to 2019 tests
AWS OCI IBM GCP Azure
Region N. Virginia US-ASHBURN-AD-1
NA East
(WDC01)
us-east4
(Northern Virginia)
East US
Profile / Type / Shape m5.4xlarge
VM.Standard2.8
VM.Standard.E2.8
VM.Standard2.8
B1.16x64 (custom)
D16s_v3
D16as_v4
vCPU 16 16 16 16 16
Memory 64 GB
120 GB
64 GB
120 GB
64 GB 64 GB 64 GB
Kernel
3.10.0-957.el7.x86_64
3.10.0-1127.10.1.el7.x86_64
4.14.35-1818.5.4.el7uek.x86_64
4.14.35-1902.301.1.el7uek.x86_64
3.10.0-957.1.3.el7.x86_64
3.10.0-1127.8.2.el7.x86_64
3.10.0-957.1.3.el7.x86_64
3.10.0-1127.10.1.el7.x86_64
3.10.0-957.1.3.el7.x86_64
4.14.35-
1902.302.2.el7uek.x86_64
Operating System
RHEL 7.6
RHEL 7.7
OL 7.6
OL 7.8
RHEL 7.6
RHEL 7.7
RHEL 7.6
RHEL 7.8
RHEL 7.6
OL 7.7
Model Intel Xeon Platinum 8175M CPU
Intel Xeon Platinum 8167M CPU
AMD EPYC 7551 32-Core
Intel Xeon Platinum 8167M CPU
Intel Xeon CPU E5-2683 v3
Intel Xeon Gold 6130
Intel Xeon CPU
Intel Xeon CPU E5-2673 v3
AMD EPYC 7452 32-Core
MHz 2.50 GHz
2.00 GHz
2.00 GHz
2.00 GHz
2.10 GHz
2.20 GHz
2.40 GHz
2.35 GHz
25. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 25 of 58
@Revelation_Tech
Test Cases
• In 2019, tests were taken on separate days:
– Test 1 on weekday (Jan 2019)
– Test 2 on weekend (Jan 2019)
• In 2020, tests were also taken on separate days:
– Test 3 on weekday (Sep 2020)
– Test 4 on weekend (Sep 2020)
– Test 5 on weekday (Oct 2020)
– Test 6 on weekend (Oct 2020)
Legend
26. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 26 of 58
@Revelation_Tech
Evaluating the Chosen OCI CPU Models
• OCI tests were repeated on both VM.Standard.E2.8 (AMD) and VM.Standard2.8 (Intel)
instances.
• This is due to the skewed results compared to 2019 when switching the CPU model.
0
500
1,000
1,500
2,000
2,500
3,000
VM.Standard.E2.8
AMD EPYC 7551
VM.StandardE2.8
Intel Xeon
Platinum 8167M
CPU Stress Test (2020)
(higher is better)
AMD EPYC 7551
Intel Xeon
Platinum 8167M
Release Date Q2 2017 Q3 2017
Frequency 2.0 GHz 2.0 GHz
Turbo Frequency 3.0 GHz 2.4 GHz
L1 Cache 2048 KB 832 KB
Cores 32 26
Threads 64 52
Multiprocessing 2 8
https://www.cpu-world.com/Compare/359/AMD_EPYC_7551_vs_Intel_Xeon_8167M.html
▪ Impossible to identify
which of the selected
AMD or Intel CPUs
on OCI are more
powerful based on
specifications alone.
▪ We began OCI tests
with the AMD EPYC
7551 CPU model first.
▪ The OCI AMD model
is 53% lower in cost
and 20% lower in
CPU performance
than the Intel model.
OCI CPU Challenge
OCI CPU Findings
Processor Specifications Actual Performance Test Results
27. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 27 of 58
@Revelation_Tech
Comparing CPU Marks
• Should research CPU specifications in conjunction with instance shapes.
• The “CPU Mark” is a measure of the CPU’s performance (higher is better).
• Table below is based off of comparisons from www.CPUBenchmark.net.
• Google Cloud processor specifications remain unknown.
https://www.cpubenchmark.net/compare/Intel-Xeon-Platinum-8175M-vs-AMD-EPYC-7551-vs-Intel-Xeon-Gold-6130-vs-AMD-EPYC-7452/3311vs3089vs3126vs3600
Intel Xeon
Platinum
8167M @
2.00 GHz
▪ Azure should
outperform other
providers based on the
CPU Mark ratings.
▪ IBM Cloud should
underperform relative
to the other providers
based on CPU Mark.
▪ OCI AMD model is 53%
lower in cost than its
Intel model, so likely will
perform worse.
▪ Keep in mind that AWS,
IBM, and Azure are
priced identically.
▪ Other factors have
impact on overall
performance, but no
details are provided;
controller,
motherboard, etc.
Our Expectations
?
?
?
29. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 29 of 58
@Revelation_Tech
Testing Tool
• stress-ng
– yum install stress-ng
• Simple workload generator that will stress test a server for the following features:
– CPU compute
– Cache thrashing
– Drive stress
– I/O syncs
– VM stress
– Socket stressing
– Context Switching
– Process creation/termination
– Much more
stress-ng: info: [12157] successful run completed in 322.04s (5 mins, 22.04 secs)
stress-ng: info: [12157] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
stress-ng: info: [12157] (secs) (secs) (secs) (real time) (usr+sys time)
stress-ng: info: [12157] cpu 637597 309.01 5134.20 0.00 2063.32 124.19
bogo
Bogus operations per second, are not comparable between different stressors.
bogo ops/s (real time)
Total bogo operations per second based on wall clock run time. The wall clock time reflects the apparent run
time. The more processors one has on a system the more the work load can be distributed onto these and
hence the wall clock time will reduce and the bogo ops rate will increase. This is essentially the “apparent”
bogo ops rate of the system.
30. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 30 of 58
@Revelation_Tech
Stress Tests
• Stress Test:
– Number of Tests: 4
– Types: CPU, Memory, I/O, Large File Copy
– Duration: 15 minutes
31. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 31 of 58
@Revelation_Tech
CPU Stress Test
• Findings:
– AWS and GCP tested with same specification as 2019, but…
– IBM and Azure benefitted notably from a change in CPU model
– OCI consistent with 2019 tests (compare Tests #1 and #2 with #5 and #6)
• Command:
stress-ng --cpu 2000 --timeout 15m --verbose --metrics-brief
0
1,000
2,000
3,000
4,000
AWS OCI IBM GCP Azure
CPU Stress Test
(higher is better)
Switch to
AMD
Upgrade
to AMD
Different
CPU model
For all host, application, and
database performance testing:
▪ AWS tests in 2019 and 2020 are
using exactly identical shapes.
▪ OCI tests in 2020 were
performed against AMD (Tests 3
& 4) and Intel (Tests 5 & 6)
CPUs.
▪ OCI tests conducted in 2019
(Tests 1 & 2) use the identical
CPU model as 2020 (Tests 5 &
6).
▪ IBM Cloud tests conducted in
2019 (Tests 1 & 2) had an older
Intel CPU model compared to
2020 (Tests 3, 4, 5, & 6).
▪ GCP tests conducted in 2019
and 2020 are on identically
sized VMs, but it is unknown if
the underlying CPU has
changed.
▪ Azure tests conducted in 2019
were on Intel-based CPUs (Tests
1 & 2) vs stronger AMD CPUs in
2020 (Tests 3, 4, 5, & 6).
Important Note About All Tests
32. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 32 of 58
@Revelation_Tech
Memory Stress Test
• Findings:
– Nothing alarming; comparable results
• Command (other variations tested too):
stress-ng --vm 8 --vm-bytes 6G --timeout 15m --metrics-brief
0
50,000
100,000
150,000
200,000
250,000
AWS OCI IBM GCP Azure
Memory Stress Test
(higher is better)
33. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 33 of 58
@Revelation_Tech
I/O Stress Test
• 3 tests conducted in 2019 due to wildly inconsistent results among many providers
• Findings:
– Inconsistent results, but OCI generally on top
– AWS and IBM consistently worse than the rest
• Command (other variations tested too):
stress-ng --io 16 --timeout 15m --verbose --metrics-brief
0
10,000
20,000
30,000
AWS OCI IBM GCP Azure
I/O Stress Test
(higher is better)
34. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 34 of 58
@Revelation_Tech
Large File Copy Stress Test
• Findings:
– OCI generally on top and outperforms all others by huge margin
• Command (other variations tested too):
stress-ng --hdd 8 --hdd-bytes 2G --timeout 15m --metrics-brief
0
50,000
100,000
150,000
200,000
250,000
AWS OCI IBM GCP Azure
Large File Copy Stress Test
(higher is better) Default Root
Volume Size
AWS 10 GB
OCI 40 GB
IBM 25 GB
GCP 20 GB
Azure 30 GB
▪ Involves lots of writes to the
root volume.
▪ All providers, with the
exception of OCI, gave low
disk space warnings.
Large File Copy Test
35. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 35 of 58
@Revelation_Tech
Interpreting the Results
▪ Performance varied up to
22% on the same VM.
▪ Multitenancy may have
unforeseen and unexpected
impact on performance.
▪ Best performance-to-cost
value.
▪ The VM.Standard AMD
model is 20% poorer in
performance than its Intel
counterpart, but 53% lower
in cost.
▪ Did not change the cost of
their instance shape
(B1.16x64), but upgraded
their backend CPU models.
▪ Switched out their backend
CPU model (a 24% boost in
the CPU Mark) which
delivered a 22% real-world
performance boost.
▪ Performance varied up to
22% on the same VM.
▪ Multitenancy may have
unforeseen and unexpected
impact on performance.
▪ No information on CPU
model.
▪ Benefited from a change in
shape from the Intel to the
AMD model, delivering a
17% boost in performance
at a 15% reduction in cost.
▪ Despite the performance
improvement, results are
underwhelming considering
the AMD model has a 67%
higher CPU mark than the
Intel.
▪ No comment. ▪ No comment. ▪ No comment. ▪ No comment. ▪ No comment.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
▪ Consistently outperforms all
other providers.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
▪ Inconsistent performance.
▪ Impact may be due to
multitenancy.
▪ Inconsistent performance.
▪ Impact may be due to
multitenancy.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
▪ Consistently outperforms all
other providers.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
▪ Simply unacceptable.
▪ Real-world tests may be
needed for further
validation.
CPU
Stress
Test
Memory
Stress
Test
I/O
Stress
Test
Large File
Copy
Stress Test
? ✓ ✓ ? ✓
✓
✓
✓ ✓ ✓ ✓ ✓
36. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 36 of 58
@Revelation_Tech
Final Thoughts
What impressed us…
▪ OCI’s extremely competitive pricing provides the
best performance-to-cost value.
▪ IBM is the only provider to upgrade its backend
CPU model, while offering the instance type at the
same cost.
▪ OCI destroys all other providers in I/O related
performance tests.
▪ Azure’s AMD instance type is 17% more powerful
and 15% lower in cost than its Intel equivalent.
What disappointed us…
▪ AWS and IBM I/O performance is unacceptable.
▪ Incredible delay in login time (and WebLogic
startup) in the IBM VMs is concerning.
▪ IBM hourly cost is identical to AWS and Azure yet
offers an older generation CPU model, but
surprisingly performs similarly.
▪ Based on CPU benchmarks, Azure underwhelms in
CPU tests, but at least delivers nearly identical
performance to AWS and IBM at the same price.
✓
37. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 37 of 58
@Revelation_Tech
APPLICATION SERVER LOAD TESTS
38. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 38 of 58
@Revelation_Tech
Testing Scenario
• Oracle WebLogic Server 14.1.1.0.0
• Single node (no cluster, no load balancer)
• 8 GB managed server
• Application specifics:
– Minimalistic ADF application with 3 use cases
(2 web pages, RESTful services, computation intensive operations)
– Uses standard SH (Sales History) schema in the Oracle Database 19c
• Tests conducted not identical to 2019 tests, so no direct comparison offered
39. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 39 of 58
@Revelation_Tech
Test
• Apache JMeter 5.0
– https://jmeter.apache.org
• “The Apache JMeter application is open source
software, a 100% pure Java application
designed to load test functional behavior and
measure performance.”
• Load Test:
– Parameters: ./poc-run-all.sh -Jusers=30 -Jloops=10 -Jrampup=120 -Jlthreads=64
40. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 40 of 58
@Revelation_Tech
Results – Average Response Times
• No comparison to 2019 tests
0
20
40
60
80
100
120
AWS OCI IBM GCP Azure
Average Response Time (sec)
(lower is better)
AMD
▪ Azure has a slight edge.
▪ Unclear why on OCI the inferior
AMD performed identically to Intel,
possibly indicating non-CPU and
non-I/O related.
▪ Overall all providers performed
close to each other, with the
exception of 2 of the 3 tests on
OCI.
Application Performance Thoughts
41. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 41 of 58
@Revelation_Tech
Results – Throughput (Transactions Per Second)
• Nothing notable about throughput
• OCI generally had slightly lower TPS than the others
• OCI took longer to complete than the rest (~8 minutes)
Encrypted Sales Data (Test #4)
Product Sales Report (Test #4)
Heavy Load Service (Test #4)
Product Sales Report (Test #4)
Azure vs OCI
42. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 42 of 58
@Revelation_Tech
Results – Number of Errors
• All errors were:
Non HTTP response code: javax.net.ssl.SSLHandshakeException/Non
HTTP response message: Remote host terminated the handshake
Non HTTP response code: org.apache.http.NoHttpResponseException/Non
HTTP response message: pochostX.revtech.poc:7002 failed to respond
0
20
40
60
AWS OCI IBM GCP Azure
Errors
(lower is better)
Number of Transactions
(total)
Error Rate
AWS 12,000 0.25%
OCI 12,000 0.03%
IBM 12,000 1.18%
GCP 12,000 0.33%
Azure 12,000 0.62%
▪ OCI experienced considerably
less errors, indicating its ability
to handle application load
better (but spread out
longer?).
▪ IBM ranks worst in terms of its
ability to handle load relative
to the other providers.
Thoughts on Error Rates
43. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 43 of 58
@Revelation_Tech
Interpreting the Results
▪ Generally comparable
response times between
AWS, IBM, and GCP.
▪ Cannot explain the identical
Average Response Times
between the AMD and Intel
tests.
▪ Unclear why higher Average
Response Times in general
in 2 of the 3 tests.
▪ Generally comparable
response times between
AWS, IBM, and GCP.
▪ Generally comparable
response times between
AWS, IBM, and GCP.
▪ Slight lead over all other
providers.
▪ Generally comparable
throughput between AWS,
IBM, GCP, and Azure.
▪ Generally lower TPS than
the others on average.
▪ Longer times to completion
than the others (but also
least errors).
▪ Generally comparable
throughput between AWS,
IBM, GCP, and Azure.
▪ Generally comparable
throughput between AWS,
IBM, GCP, and Azure.
▪ Generally comparable
throughput between AWS,
IBM, GCP, and Azure.
▪ Not as bad as IBM, not as
good as OCI.
▪ Most tests ran error-free.
▪ Had the lowest error rate of
all providers (0.03%).
▪ Possibly tied to throughput.
▪ Highest error rate of all
providers (1.18%).
▪ Not as bad as IBM, not as
good as OCI.
▪ Not as bad as IBM, not as
good as OCI.
Average
Response
Times
Throughput
(TPS)
Number
of Errors
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
✓
? ? ?
45. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 45 of 58
@Revelation_Tech
Testing Tool
• SwingBench 2.6
– http://www.dominicgiles.com/swingbench.html
• “Swingbench is a free load generator (and benchmarks) designed to stress test an Oracle
database (11g,12c)”
46. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 46 of 58
@Revelation_Tech
Test
• Oracle Database 19.0.0.0
• Stress Test:
– Number of Tests: 4
– Users: 100
– Duration: 48 minutes
– Load Ratio: Select (40%)
Insert (15%)
Update (30%)
Delete (10%)
– Database Setup: Single node (no RAC)
File system datafiles (no ASM)
All testing against CDBROOT
Default DBCA configuration
47. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 47 of 58
@Revelation_Tech
Results – Throughput
• AWS dips a bit compared to 2019 tests
• OCI results are difficult to interpret as it pertains to the different CPU models
• IBM consistent, despite improved CPU model
• GCP shows minor gains compared to 2019 tests
• Azure demonstrates notable improvements, but still lags behind others
0
20,000,000
40,000,000
60,000,000
80,000,000
AWS OCI IBM GCP Azure
Total Completed Transactions
(higher is better)
AMD
▪ Azure benefited from change
in instance type, but still lags
behind the others.
▪ Last test (Test #6) on OCI is
lower than expected, indicating
potential backend
infrastructure loads.
Interpreting Throughout Results
48. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 48 of 58
@Revelation_Tech
Results – Throughput Breakdown
0
10,000,000
20,000,000
30,000,000
AWS Oracle IBM Google Azure
Total SELECT Transactions
(higher is better)
0
5,000,000
10,000,000
15,000,000
AWS Oracle IBM Google Azure
Total INSERT Transactions
(higher is better)
0
10,000,000
20,000,000
30,000,000
AWS Oracle IBM Google Azure
Total UPDATE Transactions
(higher is better)
0
2,000,000
4,000,000
6,000,000
8,000,000
AWS Oracle IBM Google Azure
Total DELETE Transactions
(higher is better)
▪ AWS, OCI, and GCP are
generally comparable.
▪ IBM slightly lower.
▪ Azure, despite improving,
still underwhelms.
Interpreting Throughout Results
49. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 49 of 58
@Revelation_Tech
Results – Average Response Times
0.0
1.0
AWS Oracle IBM Google Azure
Average SELECT Response Times in
Seconds
(lower is better)
0
10
20
30
40
50
AWS Oracle IBM Google Azure
Average INSERT Response Times in
Seconds
(lower is better)
0
10
20
30
AWS Oracle IBM Google Azure
Average UPDATE Response Times in
Seconds
(lower is better)
0
10
20
30
AWS Oracle IBM Google Azure
Average DELETE Response Times in
Seconds
(lower is better)
50. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 50 of 58
@Revelation_Tech
Results – Number of Errors
• Cause of errors are unknown
• Errors are spread across the DML operations
• No conclusions ascertained from findings
Total Number of
Transactions
Total Number of
Errors
AWS 183,743,512 100
OCI 182,586,407 98
IBM 154,872,575 0
GCP 217,700,922 0
Azure 129,590,715 96
51. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 51 of 58
@Revelation_Tech
Interpreting the Results
▪ 183 million transactions.
▪ Generally comparable
throughput to OCI.
▪ Less throughput compared
to 2019 tests, despite
identical specifications.
▪ Freezes towards end of test
(see next slide).
▪ 182 million transactions.
▪ Generally comparable
throughput to AWS.
▪ Unclear why Intel
underperformed compared
to AMD.
▪ 154 million transactions.
▪ Ranked 4th place.
▪ Results identical to 2019,
despite improved CPU
model.
▪ 217 million transactions.
▪ Highest number of
completed transactions.
▪ Generally comparable
throughput between AWS,
OCI, and GCP.
▪ More throughput compared
to 2019 tests.
▪ 129 million transactions.
▪ Lowest number of
completed transactions.
▪ Generally comparable
average response times
between AWS, OCI, IBM,
and GCP.
▪ Best SELECT response times.
▪ Generally comparable
average response times
between AWS, OCI, IBM,
and GCP.
▪ Worst SELECT response
times.
▪ Generally comparable
average response times
between AWS, OCI, IBM,
and GCP.
▪ Generally comparable
average response times
between AWS, OCI, IBM,
and GCP.
▪ Much better than 2019
results.
▪ Still lags between all others.
▪ Small, but negligible.
▪ Spread across DML
operations.
▪ Small, but negligible.
▪ Spread across DML
operations.
▪ No errors. ▪ No errors. ▪ Small, but negligible.
▪ Spread across DML
operations.
Throughput
Average
Response
Times
Number
of Errors
?
✓ ✓ ✓ ✓ ✓
✓ ✓
✓ ? ✓ ✓
52. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 52 of 58
@Revelation_Tech
Point-In-Time Performance Captures
Transactions Per Minute
100 Users Logged On
25 Minute Mark
Transactions Per Minute
100 Users Logged On
45 Minute Mark
▪ Screenshots captured “best”
values within that time
period.
▪ AWS begins tests incredibly
fast, but shockingly and
consistently comes to a
standstill the last ¼ of the
test.
▪ AWS TPS may be tied to
limits on service thresholds.
Interpreting Throughout Results
53. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 53 of 58
@Revelation_Tech
Final Thoughts
What impressed us…
▪ OCI’s AMD model is 53% lower in cost, yet
performed near the top.
▪ GCP demonstrated overall best performance with
the highest throughput, comparable response
times, and zero errors.
What disappointed us…
▪ AWS comes to a complete halt in the last quarter of
the test.
▪ Despite notable improvements compared to 2019,
Azure continues to underperform in database
related operations relative to the other providers.
▪ AWS experienced a small dip compared to 2019.
✓
55. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 55 of 58
@Revelation_Tech
Cloud Service Provider Comparison
• Chart ranked on each metric, and provides weighted and unweighted
totals
• Support, service offerings, and other factors not included in these tests
are not considered
• Highest weights given to: cost, CPU performance, application response,
database throughput, and database errors
Host Application Database Rank
Cost
CPU
Mark
CPU Memory I/O
App
Response
App Errors
DB
Response
DB
Throughput
DB Errors Unweighted Weighted
AWS 3 2 4 5 1 4 4 5 4 4 36 25
OCI 5 4 2 5 5 2 5 4 4 4 40 24
IBM 4 5 5 5 1 4 1 4 3 5 36 24
GCP 3 2 3 5 1 3 3 5 5 5 36 26
Azure 3 1 4 5 1 5 2 3 2 4 30 22
Weight 1.00 0.05 1.00 0.05 0.90 1.00 0.50 0.80 1.00 1.00
▪ AWS dips compared to
2019 despite identical
hardware specifications.
▪ Azure ranked top in
application response times
but lagged in database
response times.
▪ Azure has highest CPU
mark, but is not reflected in
the performance tests.
▪ OCI reported 20% higher
performance on CPU tests
on Intel vs. AMD, but
performed similarly on
application and database
tests.
Inconsistent Results
56. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 56 of 58
@Revelation_Tech
• Only 7 providers covered, those that
qualify as “hyperscale cloud providers”
• Cloud lock-in is real
• There is a strong case for stating that
multi-cloud is the wrong answer
Gartner Magic Quadrant for Cloud Infrastructure & Platform Services
https://www.gartner.com/en/documents/3989743/magic-quadrant-for-cloud-infrastructure-and-platform-ser
https://www.theregister.com/2020/09/09/gartner_on_cloud_aws_fails/
57. © Revelation Technologies Group, Inc. 2021 | All rights reserved. Slide 57 of 58
@Revelation_Tech
Gartner Magic Quadrant for Cloud Infrastructure & Platform Services
◦ “Very strong performer in most Gartner use cases”
◦ “Customers continue to believe incorrectly that AWS reduces prices broadly; however, the decreases are often not universally
applied across all services”
◦ Storage “has not experienced a price reduction since 2014” despite greatly reduced prices for storage components
◦ AWS will continue to fail to “reduce prices broadly”
◦ “Thoughtfully architected, hyperscale cloud architecture”
◦ Ideal for Oracle’s E-Business Suite (EBS)
◦ Microsoft Partnership “can overcome shortfalls in Oracle’s cloud products via Azure services”
◦ Best suited for lift-and-shift and hybrid scenarios
◦ Benefits from Red Hat Technology
◦ “10 years late to market”
◦ “Negligible worldwide market share in the application PaaS category”
◦ “Diminishing mind share among developers who perceive IBM as a provider of legacy technologies”
◦ Strong for “every use case”
◦ “Closed a number of critical capability gaps between GCP and Azure”
◦ “Number of GCP outages during the last year, with devastating impact on customers”
◦ “Strong in all use cases”
◦ Particularly good for “extended cloud and edge” and for “Microsoft-centric organizations”
◦ “The lowest ratio of availability zones to regions of any vendor in this Magic Quadrant”
◦ “Does not provide any form of guaranteed capacity to customers”
https://www.gartner.com/en/documents/3989743/magic-quadrant-for-cloud-infrastructure-and-platform-ser
https://www.theregister.com/2020/09/09/gartner_on_cloud_aws_fails/