SlideShare una empresa de Scribd logo
1 de 56
Descargar para leer sin conexión
Where is my Sofa?! The Power of
Multimodal Learning to Rank in E-
commerce Search
Kamelia Aryafar, Ph.D.
Chief Algorithms Officer and EVP, Overstock.com
@KAryafar
1
What is Artificial Intelligence?
What is AI?
Artificial Intelligence: Human intelligence exhibited by
machines
Machine Learning: An approach to achieve artificial
intelligence
Deep Learning: A technique for implementing machine
learning
3
Data Science
Extract insights from structured
and unstructured data!
Use any tool and technique that
makes sense!
A data-driven problem solver!
Diverse backgrounds make a
better data science team!
What is Data Science? Who is a Data
Scientist?
4
How Does Overstock Use AI?
5
a dream home is
really what’s on
the inside
6
How Can AI Help your Business?
● Who are our customers?
● How can we help them best to:
○ Find what they need Discovery and Engagement
○ Find what they are inspired by Retention
● How can we engage new customers? Growth
AI can help by creating new experiences and optimizing existing experiences internally
and externally!
7
Customer Journey
Machine learning
techniques are often used
to know who the
customers are!
Core Optimization
Offsite Marketing
Machine learning
techniques are often used
to connect with customers
offsite.
Marketing Tech
9
Personalization and Recommendations
PHOTO CAN GO HERE Various recommender
systems are used to
personalize product pages
based on user history,
actions and product
content.
Personalization
10
Deep Learning
PHOTO CAN GO HERE Enable users to explore
different styles through
your platform.
Style Transfer
Discovering Style Trends through Deep Visually Aware Latent Item Embeddings
Murium Iqbal, Adair Kovac, Kamelia Aryafar, CVPR 2018 11
Sponsored Search and Ranking
PHOTO CAN GO HERE
12
Organic Search and Ranking
PHOTO CAN GO HERE
Machine learning
techniques are often used
to personalize and rerank
search results.
Search as an AI
Application
13
Other Applications
● Fraud detection and prevention
● Customer relationship management
● Inventory management
● Supply chain optimization
● ...
AI can be used in different places for different
products and different teams!
14
Scalable Machine Learning
in Production
Learning to Rank
17
18
Establish a Baseline
Learning to Rank as a Machine Learning Problem
20
Approaches to Learning to Rank
Pointwise
• For an item find a score
or implicit ordering
• Possible class
imbalance
Pairwise
• Ranking transformed
into classification or
regression
• Balanced classes
Listwise
• Create a ranked list for
each query
Pairwise Preference Learning
 Each training instance represents a pair of items from same set of search results in your logs for
each query.
 Learner must learn to order item1 and item2 correctly, with respect to user preference decisions
found in your logs.
Label Creation for Pairwise Features
23
Transform any implicit relevance judgement pair into either a well-
ordered or non-well ordered instance.
Model: RankingSVM
Transformation to Pairwise Classification
Production Sample Pipeline
Enable Offline Experimentation
26
Measure Performance
Iterate!
28
Multimodal Learning to Rank in Production Scale E-commerce Search
Multimodal Learning to Rank in Production Scale E-commerce Search
© Overstock
31
Multimodal Learning to Rank in Production Scale E-commerce Search
33
34
35
Model: RankingSVM
Transformation to Pairwise Classification
37
Production Pipeline
38
Model Offline Validation Metrics
Production Pipeline Sample:
41
Key Takeaways!
42
Experiment-driven Culture!
Baseline
Establish a simple learning to rank
baseline that works for your
environment and is scalable!
Offline Experiments
Enable offline experimentation
early on by correlating online
performance metrics with model
KPIs.
Iterate!
Iterate on better models, more
features and continue enhancing
your ranking model!
Kamelia Aryafar, Ph.D. | Chief Algorithms Officer and EVP
Overstock.com
@KAryafar
Thank you!
Appendix
45
Get Your Hands Dirty Early On
● Start coding early on!
● Start small and expand upon your expertise, you don’t need to know all the machine learning
foundations and tools to get started!
● Use data often and frequently to come up with hypothesis
○ Don’t fit the tool or model to the question, ask the question first!
● Data science in production
○ Learn about deployment techniques and tools and how to scale your models to large
scale data sets
● Data Science is experiment-driven, be a perfectionist and keep iterating!
● Give back to the community, share your code on github, write papers and posts, …!
● Continue learning and coding!
46
Interested in Leadership?
● Start working on more collaborative projects
● Define a roadmap and goal for your own projects and share it widely
● Keep track of your progress and iterate
● Leadership is not only about technical skills!
● Build relationships with your team and external teams
● Collaborate externally!
● Share your work with the data science community!
● Ask!
47
Building Successful
Data Science
Teams
48
Pain Points
● Integrate data scientists with business to deliver value:
○ Data science team structures
● Data science in production:
○ How does data science add value to your organization?
● Data science tool set
49
Data Science Team Structures
Centralized Embedded
No right answer!
+
Synergy, ownership,
mentorship,...
_
Disconnect with business,
Silo effect, slower hiring,...
+
Direct business impact,
fast hiring ,...
_
Lack of mentorship,
duplicated effort,
technical debt ,...
50
Data Science as a Platform Team
51
Data Science in Production
Data
Data Clean-
up
● Data Visualization, Dashboards
● Production model and inference
○ Live
○ Offline
● Data evaluation and insight
● Automate expensive manual processes
Clear requirements defined by business!
What are you expecting from your data scientists?
80 %
52
Data Science Tool Set
● Cloud Offerings
● Different languages
● Deployment Pipelines
● Model Versioning
● Metrics Evaluation Tools
● Open source or not?
Use the best tool for the job!
53
Culture Shift: Experiment-driven Culture
● Enable teams to run valuable A/B tests
○ opportunity sizing
○ Aligning data science teams with strategic initiatives
● Establish best practices for experimentation
○ Offline validation metrics
○ Establish online/offline performance correlation
● Data science is an iterative process! Don’t expect quick wins!
54
Key Takeaways
55
 Machine learning is an iterative process, always be A/B testing!
 Establish a production baseline with an MVP
 Establish offline metrics and online model performance correlation early on to enable offline testing
 Keep iterating on everything: featurization, data source and modeling!
 Gain confidence using historical data
 Know what you are testing!
Takeaways
56

Más contenido relacionado

La actualidad más candente

Build Customise Or Buy? Share South Africa
Build Customise Or Buy? Share South AfricaBuild Customise Or Buy? Share South Africa
Build Customise Or Buy? Share South AfricaRichard Harbridge
 
Richard Harbridge: 7 SharePoint Success Factors
Richard Harbridge: 7 SharePoint Success FactorsRichard Harbridge: 7 SharePoint Success Factors
Richard Harbridge: 7 SharePoint Success FactorsSharePoint Saturday NY
 
Using flow based road mapping and options
Using flow based road mapping and optionsUsing flow based road mapping and options
Using flow based road mapping and optionsLeanDog
 
Succeeding With SharePoint In Seven Steps - Share Atlanta
Succeeding With SharePoint In Seven Steps - Share AtlantaSucceeding With SharePoint In Seven Steps - Share Atlanta
Succeeding With SharePoint In Seven Steps - Share AtlantaRichard Harbridge
 
SharePoint Live - Steps to Effective Governance
SharePoint Live - Steps to Effective GovernanceSharePoint Live - Steps to Effective Governance
SharePoint Live - Steps to Effective GovernanceRichard Harbridge
 
Generating Insights: Using Competitive Assessment to Shape Your Content Strategy
Generating Insights: Using Competitive Assessment to Shape Your Content StrategyGenerating Insights: Using Competitive Assessment to Shape Your Content Strategy
Generating Insights: Using Competitive Assessment to Shape Your Content StrategyLaura Blaydon
 
Webinar: Business Intelligence From The Inside Out
Webinar: Business Intelligence From The Inside OutWebinar: Business Intelligence From The Inside Out
Webinar: Business Intelligence From The Inside OutCorSourceTechPDX
 
The Accessible Office: Making Your Documents More Accessible to Users with Di...
The Accessible Office: Making Your Documents More Accessible to Users with Di...The Accessible Office: Making Your Documents More Accessible to Users with Di...
The Accessible Office: Making Your Documents More Accessible to Users with Di...Sharon Rosenblatt
 
The CPO's New Role Within the Global Organization
The CPO's New Role Within the Global OrganizationThe CPO's New Role Within the Global Organization
The CPO's New Role Within the Global OrganizationLionbridge
 
10 steps to be successful with search
10 steps to be successful with search10 steps to be successful with search
10 steps to be successful with searchAgnes Molnar
 
Best Practices - The Seven Most Important Success Factors
Best Practices - The Seven Most Important Success FactorsBest Practices - The Seven Most Important Success Factors
Best Practices - The Seven Most Important Success FactorsRichard Harbridge
 
SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...
SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...
SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...Richard Harbridge
 
Data Science at LinkedIn - Data-Driven Products & Insights
Data Science at LinkedIn - Data-Driven Products & InsightsData Science at LinkedIn - Data-Driven Products & Insights
Data Science at LinkedIn - Data-Driven Products & InsightsYael Garten
 
Quantify your impact: how to measure the ROI of your talent brand | Talent Co...
Quantify your impact: how to measure the ROI of your talent brand | Talent Co...Quantify your impact: how to measure the ROI of your talent brand | Talent Co...
Quantify your impact: how to measure the ROI of your talent brand | Talent Co...LinkedIn Talent Solutions
 
PTW2014 Digital Strategy / Marketing Center of Excellence Presentation
PTW2014 Digital Strategy / Marketing Center of Excellence PresentationPTW2014 Digital Strategy / Marketing Center of Excellence Presentation
PTW2014 Digital Strategy / Marketing Center of Excellence PresentationBill Rattner
 
Spca2014 yammer share point_what_next_moneypenny
Spca2014 yammer share point_what_next_moneypennySpca2014 yammer share point_what_next_moneypenny
Spca2014 yammer share point_what_next_moneypennyNCCOMMS
 
ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...
ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...
ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...Mahendra Chauhan
 
SharePoint Fest Denver - The Seven Most Important SharePoint Success Factors
SharePoint Fest Denver - The Seven Most Important SharePoint Success FactorsSharePoint Fest Denver - The Seven Most Important SharePoint Success Factors
SharePoint Fest Denver - The Seven Most Important SharePoint Success FactorsRichard Harbridge
 
15 tips for bullet proof requirements analysis on SharePoint projects
15 tips for bullet proof requirements analysis on SharePoint projects15 tips for bullet proof requirements analysis on SharePoint projects
15 tips for bullet proof requirements analysis on SharePoint projectsDocFluix, LLC
 

La actualidad más candente (20)

Build Customise Or Buy? Share South Africa
Build Customise Or Buy? Share South AfricaBuild Customise Or Buy? Share South Africa
Build Customise Or Buy? Share South Africa
 
Richard Harbridge: 7 SharePoint Success Factors
Richard Harbridge: 7 SharePoint Success FactorsRichard Harbridge: 7 SharePoint Success Factors
Richard Harbridge: 7 SharePoint Success Factors
 
Using flow based road mapping and options
Using flow based road mapping and optionsUsing flow based road mapping and options
Using flow based road mapping and options
 
Succeeding With SharePoint In Seven Steps - Share Atlanta
Succeeding With SharePoint In Seven Steps - Share AtlantaSucceeding With SharePoint In Seven Steps - Share Atlanta
Succeeding With SharePoint In Seven Steps - Share Atlanta
 
Introduction to Analytics
Introduction to AnalyticsIntroduction to Analytics
Introduction to Analytics
 
SharePoint Live - Steps to Effective Governance
SharePoint Live - Steps to Effective GovernanceSharePoint Live - Steps to Effective Governance
SharePoint Live - Steps to Effective Governance
 
Generating Insights: Using Competitive Assessment to Shape Your Content Strategy
Generating Insights: Using Competitive Assessment to Shape Your Content StrategyGenerating Insights: Using Competitive Assessment to Shape Your Content Strategy
Generating Insights: Using Competitive Assessment to Shape Your Content Strategy
 
Webinar: Business Intelligence From The Inside Out
Webinar: Business Intelligence From The Inside OutWebinar: Business Intelligence From The Inside Out
Webinar: Business Intelligence From The Inside Out
 
The Accessible Office: Making Your Documents More Accessible to Users with Di...
The Accessible Office: Making Your Documents More Accessible to Users with Di...The Accessible Office: Making Your Documents More Accessible to Users with Di...
The Accessible Office: Making Your Documents More Accessible to Users with Di...
 
The CPO's New Role Within the Global Organization
The CPO's New Role Within the Global OrganizationThe CPO's New Role Within the Global Organization
The CPO's New Role Within the Global Organization
 
10 steps to be successful with search
10 steps to be successful with search10 steps to be successful with search
10 steps to be successful with search
 
Best Practices - The Seven Most Important Success Factors
Best Practices - The Seven Most Important Success FactorsBest Practices - The Seven Most Important Success Factors
Best Practices - The Seven Most Important Success Factors
 
SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...
SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...
SharePoint Saturday New Hampshire - The Seven Most Important SharePoint Succe...
 
Data Science at LinkedIn - Data-Driven Products & Insights
Data Science at LinkedIn - Data-Driven Products & InsightsData Science at LinkedIn - Data-Driven Products & Insights
Data Science at LinkedIn - Data-Driven Products & Insights
 
Quantify your impact: how to measure the ROI of your talent brand | Talent Co...
Quantify your impact: how to measure the ROI of your talent brand | Talent Co...Quantify your impact: how to measure the ROI of your talent brand | Talent Co...
Quantify your impact: how to measure the ROI of your talent brand | Talent Co...
 
PTW2014 Digital Strategy / Marketing Center of Excellence Presentation
PTW2014 Digital Strategy / Marketing Center of Excellence PresentationPTW2014 Digital Strategy / Marketing Center of Excellence Presentation
PTW2014 Digital Strategy / Marketing Center of Excellence Presentation
 
Spca2014 yammer share point_what_next_moneypenny
Spca2014 yammer share point_what_next_moneypennySpca2014 yammer share point_what_next_moneypenny
Spca2014 yammer share point_what_next_moneypenny
 
ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...
ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...
ChatBot Employee Query and Service Manager HR HCM Machine Learning AI Artific...
 
SharePoint Fest Denver - The Seven Most Important SharePoint Success Factors
SharePoint Fest Denver - The Seven Most Important SharePoint Success FactorsSharePoint Fest Denver - The Seven Most Important SharePoint Success Factors
SharePoint Fest Denver - The Seven Most Important SharePoint Success Factors
 
15 tips for bullet proof requirements analysis on SharePoint projects
15 tips for bullet proof requirements analysis on SharePoint projects15 tips for bullet proof requirements analysis on SharePoint projects
15 tips for bullet proof requirements analysis on SharePoint projects
 

Similar a Multimodal Learning to Rank in Production Scale E-commerce Search

How the Analytics Translator can make your organisation more AI driven
How the Analytics Translator can make your organisation more AI drivenHow the Analytics Translator can make your organisation more AI driven
How the Analytics Translator can make your organisation more AI drivenSteven Nooijen
 
AI Maturity Levels and the Analytics Translator
AI Maturity Levels and the Analytics TranslatorAI Maturity Levels and the Analytics Translator
AI Maturity Levels and the Analytics TranslatorGoDataDriven
 
FROM BI TO APPLIED AI
FROM BI TO APPLIED AIFROM BI TO APPLIED AI
FROM BI TO APPLIED AILior Sidi
 
Crafting a Compelling Data Science Resume
Crafting a Compelling Data Science ResumeCrafting a Compelling Data Science Resume
Crafting a Compelling Data Science ResumeArushi Prakash, Ph.D.
 
Webinar: A Communicator’s Guide To Metrics And Measuring Success
Webinar: A Communicator’s Guide To Metrics And Measuring SuccessWebinar: A Communicator’s Guide To Metrics And Measuring Success
Webinar: A Communicator’s Guide To Metrics And Measuring SuccessSocialChorus
 
Product Management in the Era of Data Science
Product Management in the Era of Data ScienceProduct Management in the Era of Data Science
Product Management in the Era of Data ScienceMandar Parikh
 
AI Orange Belt - Session 3
AI Orange Belt - Session 3AI Orange Belt - Session 3
AI Orange Belt - Session 3AI Black Belt
 
Level Up Your Tech Skills to Build Better Products by Upwork PM
Level Up Your Tech Skills to Build Better Products by Upwork PMLevel Up Your Tech Skills to Build Better Products by Upwork PM
Level Up Your Tech Skills to Build Better Products by Upwork PMProduct School
 
How to Increase Your Product Sense by ServiceNow Senior PM
How to Increase Your Product Sense by ServiceNow Senior PMHow to Increase Your Product Sense by ServiceNow Senior PM
How to Increase Your Product Sense by ServiceNow Senior PMProduct School
 
Adopting Data Science and Machine Learning in the financial enterprise
Adopting Data Science and Machine Learning in the financial enterpriseAdopting Data Science and Machine Learning in the financial enterprise
Adopting Data Science and Machine Learning in the financial enterpriseQuantUniversity
 
Predictive Analytics for Non-programmers
Predictive Analytics for Non-programmersPredictive Analytics for Non-programmers
Predictive Analytics for Non-programmersOlalekan Fuad Elesin
 
How to be a Digital Products Ninja by ServiceNow Sr. PM
How to be a Digital Products Ninja by ServiceNow Sr. PMHow to be a Digital Products Ninja by ServiceNow Sr. PM
How to be a Digital Products Ninja by ServiceNow Sr. PMProduct School
 
Your Roadmap, Your Product Story & Datadriven Product Management
Your Roadmap, Your Product Story & Datadriven Product ManagementYour Roadmap, Your Product Story & Datadriven Product Management
Your Roadmap, Your Product Story & Datadriven Product ManagementProduct School
 
5 Steps to Create a High Impact Learning
5 Steps to Create a High Impact Learning5 Steps to Create a High Impact Learning
5 Steps to Create a High Impact LearningTuan Yang
 
How to Use Artificial Intelligence by Microsoft Product Manager
 How to Use Artificial Intelligence by Microsoft Product Manager How to Use Artificial Intelligence by Microsoft Product Manager
How to Use Artificial Intelligence by Microsoft Product ManagerProduct School
 
How to Navigate Being a Product Manager by Spotify PM
How to Navigate Being a Product Manager by Spotify PMHow to Navigate Being a Product Manager by Spotify PM
How to Navigate Being a Product Manager by Spotify PMProduct School
 
How to Master Product Management Case Studies by fmr Groupon PM
How to Master Product Management Case Studies by fmr Groupon PMHow to Master Product Management Case Studies by fmr Groupon PM
How to Master Product Management Case Studies by fmr Groupon PMProduct School
 
Intro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product ManagerIntro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product ManagerProduct School
 
What does-x api-mean-for-your-learning-data and analytics-strategy-slideshare
What does-x api-mean-for-your-learning-data and analytics-strategy-slideshareWhat does-x api-mean-for-your-learning-data and analytics-strategy-slideshare
What does-x api-mean-for-your-learning-data and analytics-strategy-slideshareJames Stack
 
Practical model management in the age of Data science and ML
Practical model management in the age of Data science and MLPractical model management in the age of Data science and ML
Practical model management in the age of Data science and MLQuantUniversity
 

Similar a Multimodal Learning to Rank in Production Scale E-commerce Search (20)

How the Analytics Translator can make your organisation more AI driven
How the Analytics Translator can make your organisation more AI drivenHow the Analytics Translator can make your organisation more AI driven
How the Analytics Translator can make your organisation more AI driven
 
AI Maturity Levels and the Analytics Translator
AI Maturity Levels and the Analytics TranslatorAI Maturity Levels and the Analytics Translator
AI Maturity Levels and the Analytics Translator
 
FROM BI TO APPLIED AI
FROM BI TO APPLIED AIFROM BI TO APPLIED AI
FROM BI TO APPLIED AI
 
Crafting a Compelling Data Science Resume
Crafting a Compelling Data Science ResumeCrafting a Compelling Data Science Resume
Crafting a Compelling Data Science Resume
 
Webinar: A Communicator’s Guide To Metrics And Measuring Success
Webinar: A Communicator’s Guide To Metrics And Measuring SuccessWebinar: A Communicator’s Guide To Metrics And Measuring Success
Webinar: A Communicator’s Guide To Metrics And Measuring Success
 
Product Management in the Era of Data Science
Product Management in the Era of Data ScienceProduct Management in the Era of Data Science
Product Management in the Era of Data Science
 
AI Orange Belt - Session 3
AI Orange Belt - Session 3AI Orange Belt - Session 3
AI Orange Belt - Session 3
 
Level Up Your Tech Skills to Build Better Products by Upwork PM
Level Up Your Tech Skills to Build Better Products by Upwork PMLevel Up Your Tech Skills to Build Better Products by Upwork PM
Level Up Your Tech Skills to Build Better Products by Upwork PM
 
How to Increase Your Product Sense by ServiceNow Senior PM
How to Increase Your Product Sense by ServiceNow Senior PMHow to Increase Your Product Sense by ServiceNow Senior PM
How to Increase Your Product Sense by ServiceNow Senior PM
 
Adopting Data Science and Machine Learning in the financial enterprise
Adopting Data Science and Machine Learning in the financial enterpriseAdopting Data Science and Machine Learning in the financial enterprise
Adopting Data Science and Machine Learning in the financial enterprise
 
Predictive Analytics for Non-programmers
Predictive Analytics for Non-programmersPredictive Analytics for Non-programmers
Predictive Analytics for Non-programmers
 
How to be a Digital Products Ninja by ServiceNow Sr. PM
How to be a Digital Products Ninja by ServiceNow Sr. PMHow to be a Digital Products Ninja by ServiceNow Sr. PM
How to be a Digital Products Ninja by ServiceNow Sr. PM
 
Your Roadmap, Your Product Story & Datadriven Product Management
Your Roadmap, Your Product Story & Datadriven Product ManagementYour Roadmap, Your Product Story & Datadriven Product Management
Your Roadmap, Your Product Story & Datadriven Product Management
 
5 Steps to Create a High Impact Learning
5 Steps to Create a High Impact Learning5 Steps to Create a High Impact Learning
5 Steps to Create a High Impact Learning
 
How to Use Artificial Intelligence by Microsoft Product Manager
 How to Use Artificial Intelligence by Microsoft Product Manager How to Use Artificial Intelligence by Microsoft Product Manager
How to Use Artificial Intelligence by Microsoft Product Manager
 
How to Navigate Being a Product Manager by Spotify PM
How to Navigate Being a Product Manager by Spotify PMHow to Navigate Being a Product Manager by Spotify PM
How to Navigate Being a Product Manager by Spotify PM
 
How to Master Product Management Case Studies by fmr Groupon PM
How to Master Product Management Case Studies by fmr Groupon PMHow to Master Product Management Case Studies by fmr Groupon PM
How to Master Product Management Case Studies by fmr Groupon PM
 
Intro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product ManagerIntro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product Manager
 
What does-x api-mean-for-your-learning-data and analytics-strategy-slideshare
What does-x api-mean-for-your-learning-data and analytics-strategy-slideshareWhat does-x api-mean-for-your-learning-data and analytics-strategy-slideshare
What does-x api-mean-for-your-learning-data and analytics-strategy-slideshare
 
Practical model management in the age of Data science and ML
Practical model management in the age of Data science and MLPractical model management in the age of Data science and ML
Practical model management in the age of Data science and ML
 

Más de Lucidworks

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategyLucidworks
 
Drive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceDrive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceLucidworks
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsLucidworks
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesLucidworks
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Lucidworks
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...Lucidworks
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Lucidworks
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Lucidworks
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentLucidworks
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeLucidworks
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Lucidworks
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchLucidworks
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyLucidworks
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Lucidworks
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceLucidworks
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchLucidworks
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondLucidworks
 
Webinar: Lucidworks Managed Search
Webinar: Lucidworks Managed SearchWebinar: Lucidworks Managed Search
Webinar: Lucidworks Managed SearchLucidworks
 
KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...
KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...
KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...Lucidworks
 

Más de Lucidworks (20)

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce Strategy
 
Drive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in SalesforceDrive Agent Effectiveness in Salesforce
Drive Agent Effectiveness in Salesforce
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant Products
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized Experiences
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - Europe
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 Research
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise Search
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and Beyond
 
Webinar: Lucidworks Managed Search
Webinar: Lucidworks Managed SearchWebinar: Lucidworks Managed Search
Webinar: Lucidworks Managed Search
 
KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...
KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...
KMWorld 2019 - Diane Burley - Bringing a Consumer-Like Experience to the Digi...
 

Último

Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 

Último (20)

Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 

Multimodal Learning to Rank in Production Scale E-commerce Search

  • 1. Where is my Sofa?! The Power of Multimodal Learning to Rank in E- commerce Search Kamelia Aryafar, Ph.D. Chief Algorithms Officer and EVP, Overstock.com @KAryafar 1
  • 2. What is Artificial Intelligence?
  • 3. What is AI? Artificial Intelligence: Human intelligence exhibited by machines Machine Learning: An approach to achieve artificial intelligence Deep Learning: A technique for implementing machine learning 3
  • 4. Data Science Extract insights from structured and unstructured data! Use any tool and technique that makes sense! A data-driven problem solver! Diverse backgrounds make a better data science team! What is Data Science? Who is a Data Scientist? 4
  • 6. a dream home is really what’s on the inside 6
  • 7. How Can AI Help your Business? ● Who are our customers? ● How can we help them best to: ○ Find what they need Discovery and Engagement ○ Find what they are inspired by Retention ● How can we engage new customers? Growth AI can help by creating new experiences and optimizing existing experiences internally and externally! 7
  • 8. Customer Journey Machine learning techniques are often used to know who the customers are! Core Optimization
  • 9. Offsite Marketing Machine learning techniques are often used to connect with customers offsite. Marketing Tech 9
  • 10. Personalization and Recommendations PHOTO CAN GO HERE Various recommender systems are used to personalize product pages based on user history, actions and product content. Personalization 10
  • 11. Deep Learning PHOTO CAN GO HERE Enable users to explore different styles through your platform. Style Transfer Discovering Style Trends through Deep Visually Aware Latent Item Embeddings Murium Iqbal, Adair Kovac, Kamelia Aryafar, CVPR 2018 11
  • 12. Sponsored Search and Ranking PHOTO CAN GO HERE 12
  • 13. Organic Search and Ranking PHOTO CAN GO HERE Machine learning techniques are often used to personalize and rerank search results. Search as an AI Application 13
  • 14. Other Applications ● Fraud detection and prevention ● Customer relationship management ● Inventory management ● Supply chain optimization ● ... AI can be used in different places for different products and different teams! 14
  • 17. 17
  • 18. 18
  • 20. Learning to Rank as a Machine Learning Problem 20
  • 21. Approaches to Learning to Rank Pointwise • For an item find a score or implicit ordering • Possible class imbalance Pairwise • Ranking transformed into classification or regression • Balanced classes Listwise • Create a ranked list for each query
  • 22. Pairwise Preference Learning  Each training instance represents a pair of items from same set of search results in your logs for each query.  Learner must learn to order item1 and item2 correctly, with respect to user preference decisions found in your logs.
  • 23. Label Creation for Pairwise Features 23 Transform any implicit relevance judgement pair into either a well- ordered or non-well ordered instance.
  • 24. Model: RankingSVM Transformation to Pairwise Classification
  • 33. 33
  • 34. 34
  • 35. 35
  • 36. Model: RankingSVM Transformation to Pairwise Classification
  • 37. 37
  • 41. 41
  • 43. Experiment-driven Culture! Baseline Establish a simple learning to rank baseline that works for your environment and is scalable! Offline Experiments Enable offline experimentation early on by correlating online performance metrics with model KPIs. Iterate! Iterate on better models, more features and continue enhancing your ranking model!
  • 44. Kamelia Aryafar, Ph.D. | Chief Algorithms Officer and EVP Overstock.com @KAryafar Thank you!
  • 46. Get Your Hands Dirty Early On ● Start coding early on! ● Start small and expand upon your expertise, you don’t need to know all the machine learning foundations and tools to get started! ● Use data often and frequently to come up with hypothesis ○ Don’t fit the tool or model to the question, ask the question first! ● Data science in production ○ Learn about deployment techniques and tools and how to scale your models to large scale data sets ● Data Science is experiment-driven, be a perfectionist and keep iterating! ● Give back to the community, share your code on github, write papers and posts, …! ● Continue learning and coding! 46
  • 47. Interested in Leadership? ● Start working on more collaborative projects ● Define a roadmap and goal for your own projects and share it widely ● Keep track of your progress and iterate ● Leadership is not only about technical skills! ● Build relationships with your team and external teams ● Collaborate externally! ● Share your work with the data science community! ● Ask! 47
  • 49. Pain Points ● Integrate data scientists with business to deliver value: ○ Data science team structures ● Data science in production: ○ How does data science add value to your organization? ● Data science tool set 49
  • 50. Data Science Team Structures Centralized Embedded No right answer! + Synergy, ownership, mentorship,... _ Disconnect with business, Silo effect, slower hiring,... + Direct business impact, fast hiring ,... _ Lack of mentorship, duplicated effort, technical debt ,... 50
  • 51. Data Science as a Platform Team 51
  • 52. Data Science in Production Data Data Clean- up ● Data Visualization, Dashboards ● Production model and inference ○ Live ○ Offline ● Data evaluation and insight ● Automate expensive manual processes Clear requirements defined by business! What are you expecting from your data scientists? 80 % 52
  • 53. Data Science Tool Set ● Cloud Offerings ● Different languages ● Deployment Pipelines ● Model Versioning ● Metrics Evaluation Tools ● Open source or not? Use the best tool for the job! 53
  • 54. Culture Shift: Experiment-driven Culture ● Enable teams to run valuable A/B tests ○ opportunity sizing ○ Aligning data science teams with strategic initiatives ● Establish best practices for experimentation ○ Offline validation metrics ○ Establish online/offline performance correlation ● Data science is an iterative process! Don’t expect quick wins! 54
  • 56.  Machine learning is an iterative process, always be A/B testing!  Establish a production baseline with an MVP  Establish offline metrics and online model performance correlation early on to enable offline testing  Keep iterating on everything: featurization, data source and modeling!  Gain confidence using historical data  Know what you are testing! Takeaways 56

Notas del editor

  1. Overstock is an online retailer that’s all about helping people create their dream home for less.