1. ODTUG
Value-Based Testing
Your mileage may vary, void where prohibited 1

2. ODTUG
Wikipedia.com:
Quality assurance (QA) refers to the planned and
systematic activities implemented in a quality system
so that quality requirements for a product or service
will be fulfilled. It is the systematic measurement,
comparison with a standard, monitoring of processes
and an associated feedback loop that confers error
prevention.
Your mileage may vary, void where prohibited 2

3. ODTUG
What do they really do?
• Ensure proper functionality
• Look for “big picture” issues
• Large-scale testing (hopefully)
• Identify edge cases
• Advocate for user experience
• Environment is volatile
Your mileage may vary, void where prohibited 3

4. What do they really do?
Old school process:
• Throw code over the
wall
DEV QA OPS • And run
• Its Ops problem now
Your mileage may vary, void where prohibited 4

5. ODTUG
What do they really do?
DEV • Responsibility lines blurred
• Collaboration required
QA • Dev should be in escalation
path
OPS
Your mileage may vary, void where prohibited 5

6. ODTUG
Ops changes also need QA
• New tools, scripts, jobs must
follow release procedures
• Patches, PSUs, environment
settings
• Can combine to test all at once
• Monitoring must be tested
(verified) too
Your mileage may vary, void where prohibited 6

7. ODTUG
But how?
• Make identical environments
(i.e. not a laptop VM!)
• Automation ensures
repeatable results, fewer
mistakes
• Two-way communications
– Ops takes input from QA
– QA takes input from Ops
Your mileage may vary, void where prohibited 7

8. ODTUG
Conducting Tests…Better
• QA needs *real* info about
production
• Session counts, workload
stats, response time
• Apply load identical to prod
load.
• When identical isn’t
possible, scale back, note
risks
Your mileage may vary, void where prohibited 8

9. ODTUG
More Tests
• Operational tests – HA
failures, backups, stats
gathering
• Avg load, peak load,
crash load
• Load test with missing
nodes, disks, NICs
Your mileage may vary, void where prohibited 9

10. ODTUG
What did you do?
• Record tests completed,
especially parameters
• Keep performance reports,
AWR, capture ASH data,
key exec plans
• Baselines = key, especially
around upgrades
Your mileage may vary, void where prohibited 10

11. ODTUG
Typical Challenges, Issues
• Spend all the money on
prod, no $$$ for QA
• Our staff is so skilled, we
don’t need a “real” QA
• Our schedule slipped, so
we cut QA time down to
10% of the original plan
Your mileage may vary, void where prohibited 11

12. ODTUG
Typical Challenges, Issues
• We don’t have
requirements
• Can’t you test it faster?
• We don’t have a way to
simulate 500 users
Your mileage may vary, void where prohibited 12

13. ODTUG
Answer to Typical Challenges
• Me: What is the cost of downtime?
• Mgmt: We aren’t sure…
• Me: Let me take production down, then we can
measure.
• Mgmt: On second thought, I think I can probably
estimate it pretty closely without an outage.
Your mileage may vary, void where prohibited 13

14. ODTUG
War Stories
Your mileage may vary, void where prohibited

15. ODTUG
Q&A
• Any questions or comments, please contact:
– Gwen Shapira
• Email: cshapi@gmail.com
• Twitter: @gwenshap
– Dan Norris
• Email: dannorris@dannorris.com
• Twitter: @dannorris
Your mileage may vary, void where prohibited 15

16. Your mileage may vary, void where prohibited