This document summarizes Liz Graefe and Matt Nicole's presentation on selling taxonomy as a product at Etsy. They discuss their goals of adding filters to Etsy like other ecommerce sites, but finding their initial gatekeepers to be Etsy sellers who did not categorize or attribute listings well. They then worked to better communicate with and support sellers. While making progress, they continue to face challenges around data and analytics. Their lessons are to focus on user needs, start small, and prioritize understanding product data.
20240507 QFM013 Machine Intelligence Reading List April 2024.pdf
Know Your Gatekeepers: How to Sell Taxonomy as a Product (IAC 2019)
1. 1
Liz Graefe & Matt Nicole
March 16, 2019
Know Your Gatekeepers
How to Sell Taxonomy
as a Product
Agenda
1 - Intros
2 - What were we trying to do?
3 - Finding our gatekeepers
4 - Appeasing our gatekeepers
5 - Where are we now?
6 - What would we do different?
3. Etsy is a marketplace
Etsy is home to a universe of special, extraordinary items,
from unique handcrafted pieces to vintage treasures.
Our mission is to Keep Commerce Human. We believe that
our sellers are the experts about their wares, and we
empower them with control over their content.
Translation: Anyone can list anything* and they can call it
whatever they want.
3
*Almost. Contrary to that Facebook meme, you can’t actually list
your children.
5. 5
What were we
trying to do?
● Add filters to Etsy
● Just like every other ecommerce website
6. Filters
● Filters are built on Attributes
● Attributes are built on Categories
● Categories & Attributes are populated by
Listings
● Listings are created by Sellers
8. 8
How We
Taxonomy
at Etsy
We don’t enforce categorization.
We provide a hierarchy for the seller to classify
their products, but classification can happen at
almost any level, and accuracy is not enforced.
Until recently, it was even possible to classify a
product at the top level.
We don’t require attributes.
We provide category-specific structured attributes,
but we rarely require sellers to fill them out.
We allow sellers to vary by anything.
While we do offer structured variation attributes, sellers
have the ability to build their own variation attribute
using their own terms.
9. Family Llama T-shirt by
Good4theGoose
If our structured metadata isn’t meeting a seller’s
needs, in many cases they have the power to create
their own attributes and values.
Sellers will find a way.
10. 10
We can’t turn on filters if sellers don’t fill out
attributes.
But sellers won’t fill out attributes if they don’t see
what we’re using them for.
Chicken, meet Egg.
Any plan to launch filters was going to need
communication to sellers so they would understand
the intent behind our actions.
11. 11
Change
Communication
It all begins with data.
In order for any taxonomy change to have a chance
at success, the data has to be there to support it.
1. Products have to be classified into
categories.
2. Categories need attributes that meet the
specifications of the products and the
needs of the users
3. Attributes have to be filled out correctly.
If the data powering your taxonomy comes from
other people, make sure they know how to use it
correctly. If something is changing, make sure
they’re prepared.
Changes need to be
communicated.
12. 12
Our team ● Product Manager
● Dedicated Designer
● Fully Staffed Engineering Team
● Dedicated Analyst
● Taxonomist
17. 17
Be the product
manager ● There are Different kinds
○ 6 types
○ 3 different types
○ 7 other kinds
● Product management is about
building the right things for the right
people.
○ Lower risk, effort, friction
● Many myths
31. “We”
Humans are messy
"Regardless of what we discover, we understand
and truly believe that everyone did the best job
they could, given what they knew at the time,
their skills and abilities, the resources available,
and the situation at hand."
Norm Kerth, Project Retrospectives: A Handbook for
Team Review
33. 33
Selling your product
manager
● Talk about the Product Triangle
○ Users (personas)
○ Business (canvas)
○ Technology (lots of links here)
● Talk in impact
○ “We can move this metric” vs
○ “I want to build this thing”
○ Some starter metrics
● Know what a good one looks like
○ SVPG
● Create specific user stories
○ As a taxonomist (or other
persona)
○ I’d like to be able to (some
capability)
○ In order to (make some impact)
37. Listen to the users.
“I come to Etsy when I have
time to kill.”
“I will literally click through
250 pages of results in order
to find what I’m looking for.”
38. “I want shoes in the most
popular size.”
Listen to the users.
39. No one has ever said this. “I want shoes in the most
popular size.”
40. 40
Issues with
Past
Experiments
Traffic
In order to achieve significance in a timely fashion,
we created filters that were broad and applied to all
categories. And when that didn’t work, we tried
adding filters to top-level categories.
42. 42
Issues with
Past
Experiments
Traffic
In order to achieve significance in a timely fashion,
we created filters that were broad and applied to all
categories. And when that didn’t work, we tried
adding filters to top-level categories.
Ordering
Values displayed in order of highest inventory.
Filters displayed in the order that they were created
in. We had no control over how our data was being
displayed.
Listing Adoption
In order for filters to succeed, the values need to meet a
user’s needs, but so does the assortment that those
values return. If we build a new attribute, it can take
months for sellers to update existing listings or create
new ones that fill them out.
43. 43
If our overall goal was just “proving that filters are a win”
or just “getting filters defined everywhere regardless of
what they are,” then we’d created a vanity metric that
would lead us to develop products that aren’t useful.
Chasing a “win” was only ever going to lead us to failure.
44. 44
An Experiment
that Worked
The Right Filters Everywhere
At this point we’d been creating attributes for a few
years. Some of our highest-trafficked verticals had
category-specific filters with good coverage at all
levels of the taxonomy.
Filters vs. No Filters
This time we didn’t try to test the experience on a
single vertical. We went full A/B. Our users would
either see all of our filters, or they wouldn’t see any.
Old Filters vs. New Filters
The experiment closely followed the release of a
redesign of the Clothing vertical that included a lot of
brand new attributes. We took the opportunity to
include these filters, knowing that they might not have
the best adoption.
47. In Need of
Analytics
● How many of our products use the value April Fools’?
● How many active listings are in Unisex T-shirts?
● How many people use the filters on Wall Art?
● What are the top search queries for Slime?
● How many people browse vs. search?
In the past, we had analytics from
experiments, which weren’t that
helpful. If we wanted other insights
we had to ask an analyst and wait
until they had time to send us a one-
day snapshot of the data we needed.
49. 49
Where are we now?
Questions we still can’t answer:
● When users click on filters, which values
are they choosing most often?
● Is there any impact to filter engagement
or seller attribute adoption if we reorder
our attributes?
● How do our listings change over time?
● How often are our listings recategorized?
Problems we still face:
● The code that classifies search
queries to our taxonomy still
doesn’t see all categories
● We don’t have the ability to
migrate listings, which means we
can’t delete obsolete categories.
● Browse sessions are often logged
as search sessions, giving us an
incomplete view of how our
Buyers are looking for products.
50. 50
Things we wished
we’d done sooner. ● Securing access to Analytics
● Manually ordering attributes and
values
● Being able to QA changes on the site
before they go live.
● Being able to QA changes in the tools
before launching them to the site.