For e-commerce applications, matching users with the items they want is the name of the game. If they can't find what they want then how can they buy anything?! Typically this functionality is provided through search and browse experience. Search allows users to type in text and match against the text of the items in the inventory. Browse allows users to select filters and slice-and-dice the inventory down to the subset they are interested in. But with the shift toward mobile devices, no one wants to type anymore - thus browse is becoming dominant in the e-commerce experience. But there's a problem! What if your inventory is not categorized? Perhaps your inventory is user generated or generated by external providers who don't tag and categorize the inventory. No categories and no tags means no browse experience and missed sales. You could hire an army of taxonomists and curators to tag items - but training and curation will be expensive. You can demand that your providers tag their items and adhere to your taxonomy - but providers will buck this new requirement unless they see obvious and immediate benefit. Worse, providers might use tags to game the system - artificially placing themselves in the wrong category to drive more sales. Worst of all, creating the right taxonomy is hard. You have to structure a taxonomy to realistically represent how your customers think about the inventory. Eventbrite is investigating a tantalizing alternative: using a combination of customer interactions and machine learning to automatically tag and categorize our inventory. As customers interact with our platform - as they search for events and click on and purchase events that interest them - we implicitly gather information about how our users think about our inventory. Search text effectively acts like a tag and a click on an event card is a vote for that clicked event is representative of that tag. We are able to use this stream of information as training data for a machine learning classification model; and as we receive new inventory, we can automatically tag it with the text that customers will likely use when searching for it. This makes it possible to better understand our inventory, our supply and demand, and most importantly this allows us to build the browse experience that customers demand. In this talk I will explain in depth the problem space and Eventbrite's approach in solving the problem. I will describe how we gathered training data from our search and click logs, and how we built and refined the model. I will present the output of the model and discuss both the positive results of our work as well as the work left to be done. Those attending this talk will leave with some new ideas to take back to their own business.

1. Search Logs
+ Machine Learning
= Automatic Tagging
John Berryman

2. Hi! I'm John Berryman
@JnBrymn
This is my name
with all the
unnecessary
letters removed.
● Degree in Aerospace Engineering
● Moved into Search Technology
● Wrote a book (... well 40% of one)
I got a haircut. Life
has been different
ever since.
That's me on
the cover.
● Discovery Engineer @ Eventbrite
(Search/Recommendations)

3. What is "tagging"?
… and why would you want it?

4. First, let's talk about e-commerce search
● Search is ubiquitous.
○ Search makes the internet accessible
○ Search is the backbone of many products
○ Search is embedded in most products
● E-commerce is powered by search
● Browse is an important aspect of
the experience. You filter inventory
based upon tags.
● Mobile users prefer browse over
text search.
● Everyone is moving to mobile.
These are tags!
These are
also tags!

5. What is "tagging"?
… and why would you want it?
It's the ability to CATEGORIZE and
UNDERSTAND your inventory.
Because it powers the emerging
dominant e-commerce interaction.

6. How can you tag your inventory?
● Use curators to tag content:
○ Benefits: control over tagging, uniform tagging approach
○ Drawbacks: curation approach must be define, curators must be trained, curators are
expensive
● Require tagging from content creators:
○ Benefits: content creators know their content the best, scales well
○ Drawback: content creators may not cooperate if they see no advantage for themselves
● Encourage customers to tag content:
○ Benefits: customers are the ones buying content and their idea of tags matters most
○ Drawbacks: there's even less likelihood for customers to cooperate
… but what of nobody wants to tag your content?

7. An Interesting Observation
● Every day millions of people search for events on Eventbrite.
● They issue approximately > 500K distinct queries in a month.
● But the most common 1,000 queries accounts for 41% of all search traffic.
● The common queries look like tags!
Can we use logged searches as a
training set to built a tagging model?
○ 5k run
○ back to school
○ job fair
○ 4th of july party
○ baby
○ real estate
○ car show
○ pool party
○ golf
○ gospel
○ speed dating
○ boat party
○ photography
○ dog
○ data science
○ business
○ kids
○ networking
○ christian
○ free

8. Search Logs
+ Machine Learning
= Automatic Tagging
John Berryman

9. Initial Approach
● Given – we have 3 tables:
○ search log
○ click log
○ event table
● Step 1: Find the most common 500 queries
● Step 2: Find all the events clicked after a user search using a common query
● Step 3: Collect the name and description of those events
● Create a training set:
○ X = input = title and body text of events
○ y = output = query string used to find them a.k.a. tags
● Train a model to predict y based on new X

10. tagging_with_searches_1.ipynb
emergency backup plan

11. Problems with this Approach
● Near synonym tags:
○ memorial day
○ memorial day weekend events
○ memorial day weekend
● Small tag vocabulary
● Each event only gets 1 or 2 tags. Sometimes 0.

12. Improved Approach
● A session may contain several queries. These queries are often related:
○ Spelling corrections
○ Word synonyms
○ Query Refinements or generalizations
● Idea:
○ Let's group statistically significant query strings together.
○ Then we can train the neural network based on the query string groups

13. query_string_clusters
and
tagging_with_searches_2
emergency backup plan
emergency backup plan

14. Things to Notice
Benefits
● Much fewer near-synonyms (bitcoin, block chain, blockchai → blockchain)
● More sample data
○ v1 model - 500 most popular queries - 33% of query traffic
○ v2 model - 2649 most popular queries collapsed down to 681 - 52% of query traffic
● Broader tags
Drawback
● Some of the clusters pull in very loosely related words
○ ai → blockchain
○ ozio → rosebar

15. Tagging-Related Applications
● Power Faceted Search
● Infer relationship between tags
● Provide organizers tag
recommendations
● Better understand supply and
demand
● Apply tags to users for better
recommendation
● Search Synonyms (e.g.
misspellings)

16. Future Work
● Better coverage
○ Currently reach 50% of our traffic with 2,500 queries.
Long tail is long! > 500K distinct queries in a month
○ Model biases towards short tail labels - everything's a "day party"
○ Can't cover searches for an event that isn't in our inventory.
● Create real pipeline
● Build out all the cool ideas on the last slide

17. Questions?
… better yet, Ideas?

18. Final notes:
● My jupyter notebooks are here:
○ First implementation
○ Query collapsing
○ Second implementation
○ Third implementation
Data Nerds
● Want to learn data science with
others? You should try Data Nerds.
● Do you like spending time around
people that love learning? Penny
University is the peer-to-peer
learning community for you!
● I just shared my talk
https://twitter.com/JnBrymn

19. This slide intentionally left blank.

20. DON'T FORGET
● Tweet the slides out just before the talk
● Open the notebooks
○ do
■ cd ~/Personal/data_science/tagging_events/
■ jupyter notebook
■ open the 3 notebooks in event_tagging_strategies
○ or just use gists: one, two, three
● Bump up the font size on the notebooks
● Remove the menus
● Clear cells