In the February Fortune magazine, eBay made the cover with the title “eBay is Back!” The article cited improvements in the look and feel of the site, strategic investments in fulfillment, and technology partnerships with retailers to establish it as more than just an online auction service. Jon Bach joined just as eBay was making big bets to make notable and visible gains with this strategy. Jon recounts his two and a half years as a quality engineering director and introduces a concept he calls Live Site Quality. It means the value your customers get while experiencing different activity flows through your online product or service. It’s the impression they’re left with as they try to solve a problem or meet a need—whether it’s a simple task or a complex exploration of your site’s capabilities. Jon’s main idea is selling stakeholders on different Live Site Quality perspectives and getting eBay colleagues from different parts of the organization to buy into the idea that bugs that might most affect their Live Site Quality may lie between their team and another.

1. K4
Keynote
10/3/2013 8:30:00 AM
"Selling (and Buying) "Live Site
Quality" at eBay"
Presented by:
Jon Bach
eBay
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Jon Bach
eBay, Inc.
With more than eighteen years of experience in software testing, Jon Bach has held technical
and managerial positions in companies including Hewlett-Packard and Microsoft. In his current
role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests
(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.
He is most notable for creating, with his brother James, Session-Based Test Management, a
method to manage and report exploratory testing.

3. 9/23/2013
Selling (and Buying)
Live Site Quality) at
Jon Bach
QE Director, Live Site Quality
jobach@ebay.com
STAR West 2013
February issue
1

4. 9/23/2013
What’s the quality right NOW?
Selling and Buying Live Site Quality
Is it good enough?
Too Much Quality
Further time and effort
is a waste of resources.
floating line
Good enough quality bar
Further time and effort
is crucial.
Unacceptable Quality
Selling and Buying Live Site Quality
2

5. 9/23/2013
Heuristics of quality
Bad Buyer Experiences
Gross Merchandise Volume (+)
Net Promoter Score
Earnings Per Share
Social Media chatter
Morning Mission Report
eWatch’s Most Wanted
Re-rolls and Rollbacks
Call volume
Mean Time to Detect
Mean Time to Mitigate
Selling and Buying Live Site Quality
5
Heuristics of quality
Overdue bug fixes
Survey results
Tell Us What You Think
Metrics: DoD / Wow / MoM / YoY
Key Flow Test Results
eCommerce Bytes
#ebaysux
“Be The Customer”
Discussion Forum topics
Calls to eBay Radio
Emails from VOICES
Selling and Buying Live Site Quality
6
3

6. 9/23/2013
What I do
(“Director of Live Site Quality”)
Selling and Buying Live Site Quality
7
“Director of Live Site Quality”
Jon the Bug Hunter on eBay Radio
Call customers who participate in our NPS callback program
Speak at VOICES meet-ups
Selling and Buying Live Site Quality
8
4

7. 9/23/2013
“Director of Live Site Quality”
Live Site Ledger
P1 Bugs found this week (with Root Cause Analysis)
Overdue bugs that need fixing
What bad things happened in the last 24 hours?
Teach New Hires How to Explore
Selling and Buying Live Site Quality
9
“Director of Live Site Quality”
Make sure eWatch is in the loop on issues
Find out who *really* owns this bug
Drive issues that threaten the notion of “site readiness”
Selling and Buying Live Site Quality
10
5

8. 9/23/2013
“Director of Live Site Quality”
Coordinate teams in end-to-end testing
Host bug bashes that focus on exploration
Manage crowdsourced testing efforts in production
Selling and Buying Live Site Quality
11
What WE do … especially after release
Monitoring
Experimentation
Surveys
Scenario
Follow-up
Investigation
Competitive
Events
Reputation
Dogfooding
Selling and Buying Live Site Quality
12
6

9. 9/23/2013
Monitoring (Checking)
Selling and Buying Live Site Quality
13
Experimentation (A/B testing)
A
B
Selling and Buying Live Site Quality
14
7

10. 9/23/2013
Surveys
When site visitors click this…
Selling and Buying Live Site Quality
15
Selling and Buying Live Site Quality
16
…we see this:
8

11. 9/23/2013
Selling and Buying Live Site Quality
17
Follow-up on “UFO reports”
------------------------------------------------------------------------------“I have attached the screen shot for you. Try as I might, I can't get
the selling reminders to go to the top. The arrows don't work, but I
can move sold, unsold and active selling. I even set the default to
summary, but no luck. FYI- I am using Safari on my trusty Mac.
Also, the default font size is too big. When I zoom it out, it's too
small. It was a better size the way it was before. I sent a message
to the team through the 'what do you think' for that page.”
Selling and Buying Live Site Quality
18
9

12. 9/23/2013
Investigative Testing
What questions would you have to troubleshoot a dip in metrics like
this?
Selling and Buying Live Site Quality
19
Topics for investigation (FORCOSTTTIP)
•Feature – Do we know what happens when people bid?
•Origin -- When did this start happening?
•Recent – Did we just push code to Production?
•Consistency / Correlation -- What happened yesterday, tomorrow?
Are there other graphs and metrics?
•Overlap – Is there any other process happening?
•Security – Is it malicious? Intentional?
•Terminology -- What kinds of “bids”? US, UK, International?
•Third-Party / Silent Partner – What’s not in the room? What sort of
information have we NOT heard from?
•Truth is out there – does someone already know about this?
•Instrumentation – is the graph wrong? Can I get more data?
•Prediction – can we test a conjecture with a prediction
Selling and Buying Live Site Quality
20
10

13. 9/23/2013
Competitive Testing
Testing… After You’ve Finished Testing
21
“Site Readiness”
A large seller on eBay wants to have a “Flash Sale”.
Are we ready?
Selling and Buying Live Site Quality
22
11

14. 9/23/2013
Reputation
Selling and Buying Live Site Quality
23
Dogfooding
Video of a flow test:
Selling and Buying Live Site Quality
24
12

15. 9/23/2013
Ask my brother to round up a posse…
Rapid Testing Intensive
July 2012
• 80 testers from around the world – in-person and online
* 4 days of testing eBay Motors: Tire Finder / Wheel Center
•How to jump into a product and find bugs while you learn all about
it, while working toward "deep coverage" testing.
•Charter-based exploration, modeling, usability testing, intense
coverage to deliver notions of quality and risk
Selling and Buying Live Site Quality
25
What I’ve learned
Once they see it, they may want it
There is no sale without a buyer
A buyer can return an item being sold
Selling and Buying Live Site Quality
26
13

16. 9/23/2013
Most Important Message…
JON BACH
I AM LIVE SITE QUALITY
(AND SO CAN YOU!)
Selling and Buying Live Site Quality
Selling and Buying Live Site Quality
27
Back at your desk on Monday
•Run Experimentation???
•Create monitoring???
•What to investigate…
•Make surveys???
•How to make scenarios…
•What’s our reputation???
•Follow-up on what???
•Dogfood a medical device?
•Capabilities? What???
•Competitive what now?
•Usability, but no lab?
Try crowdsourced charters
Take an Ops guy to lunch
Customer Support’s Most Wanted
Interview a Business Analyst
Try a flow exercise with legos
Scan user groups
Talk to an early adopter
Watch a live study or trial
See a marketing roadmap
Explore a competitor URL
Try Open-Book Testing
Selling and Buying Live Site Quality
28
14