This document discusses using both "sledgehammer" and "fine brush" approaches to quality assurance (QA). It describes how the AdoptOpenJDK Quality Assurance (AQA) project has progressively refined its QA process from initial brute force techniques to more targeted testing. The AQA project aims to ensure quality across the broadest range of OpenJDK platforms through an open, community-driven approach that evolves alongside the JDK.

1. From
Sledgehammer to
Fine Brush for QA
@ShelleyMLambert
shelley.lambert@adoptopenjdk.net
August 2020

2. About Me
• AdoptOpenJDK TSC
• AQA Lead
• Eclipse OpenJ9 Committer
• IBM Runtimes Global QA Lead
• Deep Learning Aficionado
• Edible Food Forests
• FCF Director (tree planting!)
• Yoga teacher

3. Progressive Refinement
Start with brute force
End with finesse
Both needed in QA
• Sledgehammer – ruthless, insensitive, unnecessary force
• Fine brush – detail work

4. The Steps
ASSESS WHERE
YOU ARE
DETERMINE
WHAT MATTERS
DESIGN PROTOTYPE
APPROACH
REFINE

5. Assess Where You Are
• What resources you have
• Team/people
• Existing tests
• Dove-tailing into existing solution? Machines / CI
• Measure
• Code coverage (sledgehammer blow)
• Functional coverage (brush stroke)
• Other metrics (PR list, change-based testing)
• Issues list
• What your users tell you

6. Determine What Matters
• Requirements
• Users
• Prioritize
• functionality, security, perf, scalability
• Risk map, meh? (related to Issues list)

7. Design
• Design your QA approach like it is a software solution (because it is)
• Understand where your tool choices will constrain you
• Select tools for prototype, but design to avoid lock-in… (portability)
• Online tutorials focus on ‘test with IDE xxxx’ or ‘use framework yyyy’

8. Prototype Approach
• Bare minimum needed to ‘plug in’ testing

9. Refine
• Keep the good, toss the bad
• Friction is bad
• Testing evolves alongside your code base
• Always ask, how can we improve?

10. Specific Example: AQA
• AdoptOpenJDK Quality Assurance (AQA)
• Community driven, open-source project
• Broadest set of platforms of any OpenJDK distributor
• Testing a wide criteria representing actual business
requirements to identify binaries ready for
production usage

11. AQA: Assess
Current state Post-assessment Roadmap
Functional correctness Security
OpenJDK regression (open) Passes known vulnerability tests
Functional correctness
Compliance OpenJDK regression
Oracle JCK (closed) Eclipse functional
Usability
Builder-specific testing (unknown) Application & framework tests
Performance
Published metrics
Achieves minimum throughput scores
Scalability & durability
Load & stress testing

12. AQA: Determine What Matters
• “Make quality certain to happen”
• AQA Manifesto
• Open and transparent and community driven
• Diverse and robust enough to cover enterprise requirements
• Evolving alongside JDK
• Portable (run on laptop to support development, run in CI servers)
• Tagged/tracked and published to rerun tests and reproduce results

13. AQA: Design

14. AQA: Scope
OpenJ9 Hotspot SAP Corretto
8 11 13 14 15 +
openjdk functional perf system
external
(3rd
party app)
Red Hat
Branch
X
JDK Implementations
Platforms
JDK Versions
Test Categories
osx aix win xlinux plinux
6 versions
aarch
58 impl_platform
250000 tests
87,000,000 Tests Impl_platform x testLevels x testGroups x versions
58 x 2 x 3 x 6 x 10M = 20G+ test output per nightly build
Plus PR builds,
promotion builds and
personal builds &
Docker image testing
Built in-house Built upstream
scala
scala
scala
scala
scala
camel
scala
scala
scala
derby
scala
scala
scala
scala
elasticsearch
scala
scala
scala
scala
openliberty
scala
scala
scala
scala
jenkins
scala
scala
scala
scala
kafka
scala
scala
scala
scala
quarkus

15. AQA: Design
AQA AUTOMATION: 3 LAYER CAKE
testkitgen (TKG)
Test Results Summary Service (TRSS)
CI System (Jenkins / Tekton / AzDO / Github Actions, etc.)
Responsibilities:
- Categorize logically, Generates test targets based on playlist (level/platform/version/impl specific)
- Execute tests via common command line / make target patterns
- Test addition via auto-discovered directories, Standardize exclusion
- Parameters: BUILD_LIST, EXTRA_OPTIONS, JVM_OPTIONS, TEST_FLAG, ITERATION
- Generate dynamic test playlists based on input (smart parallelization)
Responsibilities:
- Schedule regular builds
- Multiplex tests across multiple nodes of all platforms
- Basic GUI view of test results
- Basic forms of parallelization
Responsibilities:
- Monitor multiple CI servers
- Graphical, Aggregate summary, Deep history
- Search/sort/filter
- Pluggable parsers
- Basis for deeper analytics and deep learning services
Layer 1: Standalone
- Execution
- Exclusion
- Test target report
summary (TAP)
- Reproducibility
Layer 2: Requires L1
- More nodes
- Scheduling
- GUI
- enhanced reporting for
tests that support Junit
output
Layer 3: Requires L1 & L2
- Database queries
- Basis for extras, search /sort
/filter /analyze across DB
entries
- Monitor anything

16. AQA: Prototype Approach
1st layer of the cake

17. AQA: Refine
• Past 2.5 years have been a shocking amount of refinement
• Refer to tests repo history (openjdk-tests, TKG, openjdk-test-tools,etc)
• Building blocks for further refinement

18. AQA 2020 Plans – Keep on Refinin’
• Performance
• compare from release to release
• Multi-machine
• EXTERNAL Test group expansion/clean-up
• External_custom, ‘_with_tag’ targets
• Smarter/Better/Faster
• Smart Parallelism
• Automatically test and reinclude excluded tests
• Change-based testing
• TRSS enhancements
• BY ‘started_by’ view, By ‘Version_info’ view
• Enhanced Analytics services to aid triage, Bug prediction, Core analytics
• Collaboration with research
• Deepsmith – deep learning model to generate fuzzers
• Feedback to open benchmark teams

19. The Long List of Brush Strokes
In no particular order and definitely not the full list:
• Bug prediction
• Combinatorial Test Design / Modeling
• Improve your Triage tools
• Change-based testing (combine codecov with PR list)
• Verify your Tests (defect injection)
• Apply deep learning to help with certain QA tasks

20. Functional vs Code Coverage
static Integer mathyJunkFunction(int num1, Integer num2, String num3) {
return new Integer(num1) / (num2 + new Integer(num3));
}
Integer result = mathyJunkFunction(1, new Integer(1), “1”);
TESTED, 100% code coverage
hazaa!
• Defect 1: What if num3 has a value of null (throws NumberFormatException)?
• Defect 2: What if num3 is “1” and num2 is -1, or “0” and 0, or any other combination that sum to
0 (throws ArithmeticException)?
• Defect 3: What if num2 is null (throws NullPointerException)?

21. Bug Prediction
• SCORES PER FILE BASED ON ‘RECENT’ CHANGES DUE TO DEFECTS (GITHUB PRS/ISSUES), PREDICT
BASED ON CHANGE & DEFECT HISTORY, OTHER FEATURES?

22. Hear from More AdoptOpenJDK Folks
https://blog.adoptopenjdk.net/2020/07/
adoptopenjdk-virtual-roadshow

23. Extra Info
• adoptopenjdk.net
• trss.adoptopenjdk.net
• Bi-weekly AQAvit meetups (open, transparent evolution)
• https://github.com/AdoptOpenJDK/openjdk-tests/issues?q=+label%3A%22AQAvit+Meeting%22+

24. Demo TRSS