ACTIVATE 2019 Keynote: Search is at the heart of modern day e-commerce. In this talk, Dr. Kamelia Aryafar, Chief Customer and Algorithms Officer at Overstock.com will share how deep learning and multimodal learning to rank methods can improve the relevancy of production scale e-commerce search engines. Kamelia will also share best practices on building successful AI teams and an experiment-driven culture, including how Overstock.com approaches search and machine learning and how she sees the field evolving.
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Multimodal Learning to Rank in Production Scale E-commerce Search
1. Where is my Sofa?! The Power of
Multimodal Learning to Rank in E-
commerce Search
Kamelia Aryafar, Ph.D.
Chief Algorithms Officer and EVP, Overstock.com
@KAryafar
1
3. What is AI?
Artificial Intelligence: Human intelligence exhibited by
machines
Machine Learning: An approach to achieve artificial
intelligence
Deep Learning: A technique for implementing machine
learning
3
4. Data Science
Extract insights from structured
and unstructured data!
Use any tool and technique that
makes sense!
A data-driven problem solver!
Diverse backgrounds make a
better data science team!
What is Data Science? Who is a Data
Scientist?
4
7. How Can AI Help your Business?
● Who are our customers?
● How can we help them best to:
○ Find what they need Discovery and Engagement
○ Find what they are inspired by Retention
● How can we engage new customers? Growth
AI can help by creating new experiences and optimizing existing experiences internally
and externally!
7
10. Personalization and Recommendations
PHOTO CAN GO HERE Various recommender
systems are used to
personalize product pages
based on user history,
actions and product
content.
Personalization
10
11. Deep Learning
PHOTO CAN GO HERE Enable users to explore
different styles through
your platform.
Style Transfer
Discovering Style Trends through Deep Visually Aware Latent Item Embeddings
Murium Iqbal, Adair Kovac, Kamelia Aryafar, CVPR 2018 11
13. Organic Search and Ranking
PHOTO CAN GO HERE
Machine learning
techniques are often used
to personalize and rerank
search results.
Search as an AI
Application
13
14. Other Applications
● Fraud detection and prevention
● Customer relationship management
● Inventory management
● Supply chain optimization
● ...
AI can be used in different places for different
products and different teams!
14
21. Approaches to Learning to Rank
Pointwise
• For an item find a score
or implicit ordering
• Possible class
imbalance
Pairwise
• Ranking transformed
into classification or
regression
• Balanced classes
Listwise
• Create a ranked list for
each query
22. Pairwise Preference Learning
Each training instance represents a pair of items from same set of search results in your logs for
each query.
Learner must learn to order item1 and item2 correctly, with respect to user preference decisions
found in your logs.
23. Label Creation for Pairwise Features
23
Transform any implicit relevance judgement pair into either a well-
ordered or non-well ordered instance.
43. Experiment-driven Culture!
Baseline
Establish a simple learning to rank
baseline that works for your
environment and is scalable!
Offline Experiments
Enable offline experimentation
early on by correlating online
performance metrics with model
KPIs.
Iterate!
Iterate on better models, more
features and continue enhancing
your ranking model!
46. Get Your Hands Dirty Early On
● Start coding early on!
● Start small and expand upon your expertise, you don’t need to know all the machine learning
foundations and tools to get started!
● Use data often and frequently to come up with hypothesis
○ Don’t fit the tool or model to the question, ask the question first!
● Data science in production
○ Learn about deployment techniques and tools and how to scale your models to large
scale data sets
● Data Science is experiment-driven, be a perfectionist and keep iterating!
● Give back to the community, share your code on github, write papers and posts, …!
● Continue learning and coding!
46
47. Interested in Leadership?
● Start working on more collaborative projects
● Define a roadmap and goal for your own projects and share it widely
● Keep track of your progress and iterate
● Leadership is not only about technical skills!
● Build relationships with your team and external teams
● Collaborate externally!
● Share your work with the data science community!
● Ask!
47
49. Pain Points
● Integrate data scientists with business to deliver value:
○ Data science team structures
● Data science in production:
○ How does data science add value to your organization?
● Data science tool set
49
50. Data Science Team Structures
Centralized Embedded
No right answer!
+
Synergy, ownership,
mentorship,...
_
Disconnect with business,
Silo effect, slower hiring,...
+
Direct business impact,
fast hiring ,...
_
Lack of mentorship,
duplicated effort,
technical debt ,...
50
52. Data Science in Production
Data
Data Clean-
up
● Data Visualization, Dashboards
● Production model and inference
○ Live
○ Offline
● Data evaluation and insight
● Automate expensive manual processes
Clear requirements defined by business!
What are you expecting from your data scientists?
80 %
52
53. Data Science Tool Set
● Cloud Offerings
● Different languages
● Deployment Pipelines
● Model Versioning
● Metrics Evaluation Tools
● Open source or not?
Use the best tool for the job!
53
54. Culture Shift: Experiment-driven Culture
● Enable teams to run valuable A/B tests
○ opportunity sizing
○ Aligning data science teams with strategic initiatives
● Establish best practices for experimentation
○ Offline validation metrics
○ Establish online/offline performance correlation
● Data science is an iterative process! Don’t expect quick wins!
54
56. Machine learning is an iterative process, always be A/B testing!
Establish a production baseline with an MVP
Establish offline metrics and online model performance correlation early on to enable offline testing
Keep iterating on everything: featurization, data source and modeling!
Gain confidence using historical data
Know what you are testing!
Takeaways
56
Notas del editor
Overstock is an online retailer that’s all about helping people create their dream home for less.