1. Optimizing eBay
Improving customer experience at the world’s online marketplace
Elissa Darnell,
Director, User Experience Research
Deepak Nadig,
eBay Principal Architect
eMetrics Marketing Optimization Summit
May 06, 2008

2. In the beginning…
“I started eBay as an experiment”
— Pierre Omid
2
eBay Inc. confidential

3. An Amazing Story
2008
Registered eBay
users: 276 Million
1996
Registered eBay
users: 41,000
eBay Inc. confidential

4. The eBay context
eBay manages …
– Over 276,000,000 registered users
– Over 1 Billion photos
– eBay users worldwide trade more than $2039 worth of goods every second
An SUV sells every 2 seconds
A sporting goodis sold every 5 minutes
– eBay averages well over 1 billion page views per day
– At any given time, there are over 113 million items for sale on
the site in more than 50,000 categories
– eBay stores over 2 Petabytes of data – over 200 times the size of the Library of Congress!
– eBay analytics processes over 25 Petabytes of data on any day
– The eBay platform handles 4.4 billion API calls per month
In a dynamic environment
– 300+ features per quarter
– We roll 100,000+ lines of code every two weeks
In 39 countries, in seven languages.
>44 Billion SQL executions/day!
eBay Inc. confidential
Over ½ Million pounds of
Kimchi are sold every year!

5. Site Statistics: in a typical day…
June
Q1
1999
2007
Outbound Emails
1M
41 M
41x
Total Page Views
54 M
>1 B
19x
16 Gbps
59x
0
150 M
N/A
~97%
99.94%
50x
Peak Network Utilization
API Calls
Availability
268 Mbps
43 mins/day
5
eBay Inc. confidential
50 sec/day
Growth

6. Meet the Buyers and Sellers
Some people buy on eBay and other people sell,
and some do both
eBay users trade in more than 50,000 categories
eBay’s buyers want a fun shopping experience that
provides a great deal
Approximately 1.3M people around the world make
all or part of their living selling on eBay
6
eBay Inc. confidential

7. Who is eBay’s Community?
“eBay Community” is everyone who has a relationship with eBay, Inc.
Collectors
New Buyers
Frequent buyers
Casual Sellers
Business Sellers
Hobby Sellers
Store Sellers
One-time buyers
Top Buyers
Power Sellers
Small Businesses
New Sellers
Experienced Buyers
In-active buyers
Buy online
Merchant Accounts
Pay a friend
7
eBay Inc. confidential
Personal Accounts

8. What we mean by the whole user experience
My overall feeling about a brand/
product (in the abstract) is good.
It starts by being useful…
I like the way the product looks and feels.
Functionally, people must
be able to use it…
The way it looks must
be pleasing…
This helps create an
overall brand experience
Executing well on all of these
areas is what creates a great
user experience. Research is
needed for each.
8
eBay Inc. confidential
I am able to use the product easily
(I can perform the tasks it supports).
It is useful to me. It meets my needs.

9. Assortment of User Research methods
Lab Testing
Surveys
9
eBay Inc. confidential
Field Visits
Eye Tracking
Participatory Design
Card sorting

10. Lab-based Usability Testing
• Usability or “lab-testing”
– We bring in representative users
individually to our usability labs
– Observe them while they perform
assigned tasks
– Use prototypes or the live site
• Enables direct observation of target users
as they interact with our web site or a
design prototype
– Observe what users ACTUALLY do, rather
than relying on what they SAY they do
– Understand WHY people do what they do, not just
what they do
• Identify areas that are confusing and potential fixes
• Usability testing done iteratively, throughout the design process
10
eBay Inc. confidential

11. Variations on the standard lab study
Some variations on standard lab tests
• Low Fidelity (Paper) Lab Testing
– Designs are shown on paper
– Researcher or Designer acts as the computer
– Participant uses their finger as the mouse
• RITE Lab Testing
– Stands for Rapid Iterative Test and Evaluation
– Focuses more on rapidly iterating and re-testing
the design based on very small sample
– Can be performed on lo-fi (paper) or high fidelity
interactive prototypes
11
eBay Inc. confidential

12. Visits
Seeing through the eyes of our customers
12
eBay Inc. confidential

13. Visits: What is it?
• Also known as a Field Study or Ethnography
• Involves going into people’s home, office, where ever they use eBay
• Spending two to three hours with them, observing them use eBay
(or shop online) and listening to them, and sometimes conducting
structured interview
• Unlike Lab Testing, typically formal tasks are not given
• Data is collected through videotaping and taking extensive notes
• Findings are summarized across participants
13
eBay Inc. confidential

14. Visits allow us to observe how people use eBay in a natural context
14
eBay Inc. confidential

15. We get to see people use eBay…
• On their own computers (PCs, laptops, old, new)
• With various connection speeds (some dial-up)
• To perform their own tasks (such as selling a camera or book)
• With their own cameras and workspace (living room, office)
• With life’s normal interruptions (talking parrot, cluttered desk)
They tend to let their guard down and we learn
things we might not otherwise learn online or in the lab
15
eBay Inc. confidential

16. The focus of Visits
• Questions we focus on in Visits are:
– “What is the larger context of use?”
– “What issues exist, and WHY?”
– “What can we do to address the issues?”
– Visits are NOT about the numbers, or the question,
“How many users experienced that?”
– If pervasiveness of a particular issue is of interest, we
supplement with survey or analytics data to help quantify the issue
16
eBay Inc. confidential

17. Research methods span the Product Development
Strategic research to inspire
Understand
Field Visits
Participatory Design
Competitive Evaluation
17
eBay Inc. confidential
Conceive
Design research to inform
Design
Lab Testing
RITE Testing
Paper Prototypes
Eye Tracking
Card sorting
Assessment research to track
Develop
Launch
Surveys
Live-to-site Testing
Longitudinal Studies
Diary Studies

18. Research methods
Self-reported
(stated)
Focus Groups / “Voices”
Desirability studies
Intercept Surveys
Cardsorting
Attitudes | What people believe Email Surveys
Phone Interviews
Diary/Camera Study
Message Board Mining
DATA SOURCE
(Onsite interviews)
Mixture
Why
How to fix
What
How much
“Visits” / Ethnographic Field Studies
Usability Lab Studies (task-based)
(Extended observation)
/
Quantitative user experience assessments
(e.g., Keynote/Vividence)
Usability benchmarking (in lab)
Observed
Behavior
Eyetracking
Behaviors | What people do
Data mining
Experimentation
Clickstreams
Qualitative
APPROACH
KEY – Context of data collection with respect to product use
eBay Inc. confidential
De-contextualized / not using product
Natural use of product
Scripted or lab-based use of product
Combination / hybrid
Quantitative

19. Case Study
Using Qualitative and Quantitative Research to create a New “View Item” Page
Why redesign the “View Item” Page?
• Increase BID/BIN efficiency
• Improve the user experience: Reduce complexity and clutter and inspire buyers to convert
Research Approach
• Qualitative and quantitative research to understand user needs and inform design decisions
• Incorporate User Experience Research, Market Research, and Analytics
• Institute research as an integral component of the redesign process (from start to finish)
View ItemInc. confidential
2000
eBay
19
View Item 2003
View Item 2006
View Item 2008

20. Research Overview
Understanding User Needs (Qualitative)
• “Compelled” lab study (US) to understand the current experience and how it may be improved
• Participatory design (US, UK, DE) to understand the “ideal” experience to gather user requirements and inform
product design and innovation
Concept Testing (Qualitative)
• Focus groups (US) to gauge user reaction to early design concepts and isolate aspects of alternative designs
that resonate with users
Iterative Design (Qualitative & Quantitative)
• Rapid Iterative Testing (RITE) (US, DE, IT) to gauge user reaction to an early View Item prototype and make
rapid improvements to the design based on user feedback and behavior
• Maximum Differential Survey (US) to determine the relative perceived usefulness of features to Buyers in an
effort to streamline and simplify the experience
• Analytics (US) to understand current usage of existing features
Visual Design Research (Quantitative)
• Desirability study (US) to determine which visual design approach best conveys target brand attributes (e.g.
convenient, clean, safe, fun)
• Tab Visual Design Eyetracking study (US) to explore different visual design treatments for View Item page tabs
and determine which best captures and maintains attention as measured by eyetracking data
Longitudinal Research (Q2 2008) (Qualitative)
• Longitudinal Diary Study (US) with functional prototype to understand “real world” usage of View Item over time
eBay Inc. confidential
20
and determine areas for improvement prior to launch

21. Compelled Lab Study
Objective
Surface usability issues and perceived strengths and weaknesses
of the current View Item page and larger transaction flow
Approach
A “compelled” lab study with users genuinely interested in
purchasing items of interest on eBay. Users were asked to find
and purchase items of interest on eBay with their real account and
payment information in the lab setting
Participants
12 - Mix of experienced and less experienced eBay buyers
All participants were pre-screened to ensure they had a genuine
interest in purchasing specific items of interest on eBay in the
coming days
21
eBay Inc. confidential

22. Feature Prioritization
Maximum Differential Survey
Background and Objectives
• A key goal was site simplification: Provide buyers with the most important information they need to make
their purchase decision
• Traditional surveys focused on understanding feature usefulness typically do not effectively differentiate
among features (e.g. majority of features considered useful by 90% or more of buyers)
• A Max Differential survey was conducted in the U.S. to determine the relative usefulness of View Item
features to understand which features to include and the relative prominence that should be assigned
• Approximately 5000 buyers and 5000 sellers completed the survey which involved asking users to
choose the most useful and least useful feature among different combinations of features to yield a score
for each feature
EXAMPLE
Total Buyers
Current bid
Auction countdown
End time
Time left
Real-time updating of bid
Questions and Answers
Your maximum bid
Number of unique bidders
22
eBay Inc. confidential
Top 30% Buyers
Lower 70% Buyers

23. Tab Visual Design Eyetracking Study
Goal: To explore different visual design treatments for View Item page tabs and
determine which best captures and maintains attention as measured by
eyetracking data
On
Hover
Off
Sample Heat map summarizing
overall viewing pattern
On
Hover
Off
On
Hover
Off
Sample scanning pattern
23
eBay Inc. confidential

24. How do we know the new “View Item” page will be a
success?
Research was conducted throughout the product lifecycle to evaluate the current
strategy and design at each stage (prior to further dedication of design or engineering
resources)
• Focus groups on early concepts
• Rapid Iterative Testing of early prototype
• Max Differential survey (and Analytics data) to ensure that reduction of page complexity
targeted the least useful features
• Quantitative “Desirability” study to ensure that the chosen design approach best conveyed
target brand attributes (such as “fun” and “unique”)
• Quantitative Eyetracking study of tab design alternatives to ensure the chosen treatment
would best capture and maintain user attention
• A longitudinal diary study (planned in Q2 2008) to gather qualitative real-world usage
feedback to ensure a good user experience prior to launch (to be
interpreted in
conjunction with A/B Test data)
24
eBay Inc. confidential

25. Experimentation
• Metrics
• Reporting
• Idea (!)
• Learning
7. Analysis &
Results
1. Hypothesis
• Tracking
• Monitoring
5. Measurement
eBay
Experimentation
Platform
2. Experimental
Design
• DOE
• Define Samples,
Treatments, Factors
4. Launch
Experiment
• User  (Experiment,
Treatment)
• Serve Treatment
25
eBay Inc. confidential
3. Setup
Experiment
• Setup Experiment Samples
Treatments, Factors
• Implementation

26. Automation
• Dynamically adapt experience
– Choose page, modules, and inventory which provide best experience for that user and
context
– Order results by combination of demand, supply, and other factors (“Best Match”)
• Feedback loop enables system to learn and improve over time
– Collect user behavior
– Aggregate and analyze offline
– Deploy updated metadata
– Decide and serve appropriate experience
• Best Practices
– “Perturbation” for continual improvement
– Dampening of positive feedback
26
eBay Inc. confidential

27. Experimentation Platform
Access
Access
Design
Results
Observations
Experiment
Metadata
Results
Observations
Metrics /
Experience
27
eBay Inc. confidential
Page,
Module
Experience
Metrics /
Experience
Experience
Response

28. What we think about
Fidelity of
Experiments
Extent to which the model and its conditions represent the final feature or
product
Cost of Experiments
The total cost of designing, building, running, and analyzing an
experiment
Iteration time
The time from hypothesis to when the analyzed results are available for planning the
next iteration
Concurrency
Number of experiments that can be run at the same time
Signal/Noise Ratio
The extent to which the metrics are obscured by experimental noise
Type/Level of Experiment
Supporting different types (A/B, 1-FAT, DOE) and levels (page, module, page flow)
of experiments
28
eBay Inc. confidential

29. Challenges …
• Sticky-ness to user
• What, not why
• Duration and long term effects
• Minor vs. major differences
• Extent of generalization
29
eBay Inc. confidential

30. The Power of Combining Data
Qualitative research such as lab tests and field visits give us rich
data about
•
•
•
•
•
Usability problems
Discoverability
Navigation
Terminology
More complex problems
Quantitative research such as surveys, usage studies, and log
analysis tells us
• How pervasive a problem is
• What features on the site are actually being used
The combination of qualitative and quantitative data give us a more
complete picture that is most powerful
30
eBay Inc. confidential

31. How we get our data differs…
Analytics
Research
Measure what customers
do through serving
experiences and tracking
responses
Work directly with
customers by…
• Talking to them
• Observing them
• Surveying them
The real power is from looking at our user’s
31
eBay Inc. confidential
from both sides!

32. Summary
• Understanding the customer experience requires insights into
– What they do
– Why they do it
– Attitudes, motivations and behaviors
• A variety of research methods, both qualitative and quantitative, can
be used.
• Each of the methods – qualitative and quantitative - have their
advantages and limitations.
• Using them together continues to help eBay gain a holistic
understanding of the user experience and aspects that need to be
improved.
32
eBay Inc. confidential

33. Questions
Elissa Darnell
edarnell@ebay.com
Deepak Nadig
dnadig@ebay.com
33
eBay Inc. confidential