Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Extracting information from images using
deep learning and transfer learning.
Application to personalized recommendations.
Introduction and Context
Iterative building of a
recommender system
Labeling Images
Pragmatic deep learning for
dummies
Po...
Dataiku
•  Founded in 2013
•  90 + employees, 100 + clients
•  Paris, New-York, London, San Francisco,
Singapore
Data Scie...
Client Key Figures
E-business vacation retailer
Negotiate the best prize for their clients
Discount luxury
Sale Image is p...
Specificities
Highly temporary sales
-> Classical recommender system
fail
-> Time event linked (Christmas, ski,
summer)
Ex...
Iterative Building of a Recommender System
Basic Recommendation Engines
Other Factors
One Meta Model to Rule Them All
Recommenders	
  as	
  
features	
  
Machine	
  learning	
  to	
  
op5mize	
  purchasing	
 ...
Recommender system for Home Page Ordering
Cleaning, combining
and enrichment of
data
Recommendation
Engines
Optimization o...
Why use Image ?
We want do distinguish
« Sun
and
Beach »
« Ski »
A picture is worth a thousand words
Integrating Image Information
Sales Images Labelling
Model
Pool + Palm Trees
Hotel
+ Mountains
Pool + Forest + Hotel
+ Sea...
Image Labelling For Recommendation Engine
Pragma&c	
  Deep	
  learning	
  for	
  “Dummies”	
  
Using Deep Learning models
Common Issues
“I don’t have GPUs server” “I don’t have a deep leaning expert”
“I don’t have lab...
“I don’t have (or few) labelled data”
-> Is there similar data ?
Solution 1 : Pre trained models
PLACES	
  DATABASE	
  US	...
tower: 0.53
skyscraper: 0.26
swimming_pool/outdoor: 0.65
inn/outdoor: 0.06
Solution 1 : Pre trained models
If there is ope...
Solution 2 : Transfer Learning
Credit	
  :	
  	
  Fei-­‐Fei	
  Li	
  &	
  Andrej	
  Karpathy	
  &	
  Jus5n	
  Johnson	
  h...
Solution 2 : Transfer Learning
Use the network as a feature extractor
Fine tune the model Keras makes it super easy !
(Pos...
PLACES	
  DATABASE	
   US	
  SUN	
  DATABASE	
  
Training	
  
(op5onal)	
  
Pre-­‐trained	
  model	
  
VGG16	
  
tower: 0....
Post Treatment & Results
(Or how we transfer the labelling information)
Using	
  Images	
  informa&on	
  for	
  BI	
  on	
...
Labels post-processing
Complementary information Redondant information
Issue with our approach:
Solution : Matrix Factoriz...
Topic extraction with Non-Negative Matrix Factorization
•  Non Negative Matrix factorization (NMF)
X = WH
•  X : image x t...
Image content detection
Topic scores determine the importance of topics in an image
TOPIC	
   TOPIC	
  SCORE	
  
(%)	
  
G...
Note on model performance
•  Images labels are used for similarity
Calling herb field “putting green”:
•  Is not important...
Results
Results ?
1) Visits :
•  France and Morocco
•  Pool displayed
2) First Recommendation
•  Mostly France &
Mediterranean
•  ...
Results ?
1) Visits :
•  Spain
•  Sun & Beach
•  Pool displayed
2) First Recommendation
•  Displays nature…
3) Only Images...
Learned along the way
What’s next ?
AYrac5veness	
  =	
  %	
  visits	
  with	
  tag	
  /	
  %	
  sales	
  with	
  tag	
  -...
What’s Next ?
Kenya
Prague
Berlin
Cambodia
Conclusion
Do iterative data science !
Start simple and grow
Evaluate at each steps
Image labelling = BI on steroids
Trans...
Thank you for your attention !
Basic Recommendation Engines
•  Implementation
•  Everything in SQL
-  Vertica
-  Then Impala
-  Then Spark
•  Collaborati...
One Meta Model to Rule Them All
•  Negative sampling
•  Take all purchases tuples : (user, product, timestamp)-> 1
•  Sele...
Classification and not Detection problem…
•  Only have probabilities of each class
•  Selecting based on probability thres...
Solution 3 : What about APIs ?
Solution 3 : What about APIs ?
•  Price
•  Their cost
often rather cheap. Ex: 100 K request for less than 300$
•  VS the o...
What about APIs ? Use for generating labels !
How to steal model:
•  1) Score part of the database for training
•  2) Trai...
What about APIs ? Use for generating labels !
Experiment:
•  5000 requests on API. (4500/500 split)
•  Transfer learning w...
What about APIs ? Results
Precision	
   75	
  
Label	
   Probability	
   Label	
   Probability	
  
landscape 1,0000 sunset...
Próxima SlideShare
Cargando en…5
×

Extracting information from images using deep learning and transfer learning — Pierre Gutierrez (dataiku labs) @PAPIs Connect — São Paulo 2017

345 visualizaciones

Publicado el

For online businesses, recommender systems are paramount. There is an increasing need to take into account all the user information to tailor the best product offer, tailored to each new user.

Part of that information is the content that the user actually sees: the visuals of the products. When it comes to products like luxury hotels, pictures of the room, the building or even the nearby beach can significantly impact users’ decision.

In this talk, we will describe how we improved an online vacation retailer recommender system by using the information in images. We’ll explain how to leverage open data and pre-trained deep learning models to derive information on user taste. We will use a transfer learning approach that enables companies to use state of the art machine learning methods without needing deep learning expertise.

Publicado en: Tecnología
  • ➤➤ How Long Does She Want You to Last? Here's the link to the FREE report ■■■ https://tinyurl.com/rockhardxx
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

Extracting information from images using deep learning and transfer learning — Pierre Gutierrez (dataiku labs) @PAPIs Connect — São Paulo 2017

  1. 1. Extracting information from images using deep learning and transfer learning. Application to personalized recommendations.
  2. 2. Introduction and Context Iterative building of a recommender system Labeling Images Pragmatic deep learning for dummies Post Processing AKA: Image for BI on steroids Outline Results More images !
  3. 3. Dataiku •  Founded in 2013 •  90 + employees, 100 + clients •  Paris, New-York, London, San Francisco, Singapore Data Science Software Editor of Dataiku DSS DESIGN Load and prepare your data PREPARE Build your models MODEL Visualize and share your work ANALYSE Re-execute your workflow at ease AUTOMATE Follow your production environment MONITOR Get predictions in real time SCORE PRODUCTIO N
  4. 4. Client Key Figures E-business vacation retailer Negotiate the best prize for their clients Discount luxury Sale Image is paramount Purchase is impulsive 18 Millions of clients. Hundreds of sales opened everyday
  5. 5. Specificities Highly temporary sales -> Classical recommender system fail -> Time event linked (Christmas, ski, summer) Expensive Product -> Few recurrent buyers -> Appearance counts a lot
  6. 6. Iterative Building of a Recommender System
  7. 7. Basic Recommendation Engines
  8. 8. Other Factors
  9. 9. One Meta Model to Rule Them All Recommenders  as   features   Machine  learning  to   op5mize  purchasing   probability   Combine   Recommend   Describe  
  10. 10. Recommender system for Home Page Ordering Cleaning, combining and enrichment of data Recommendation Engines Optimization of home display the application automatically runs and compiles heterogeneous data Generation of recommendations based on user behaviour Every customer is shown the 10 sales he is the most likely to buy Customer visits Purchases Sales Images Metal model combine recommendations to directly optimize purchasing probability Meta Model +7% revenue Sales information (A/B testing) Batch Scoring every night
  11. 11. Why use Image ? We want do distinguish « Sun and Beach » « Ski » A picture is worth a thousand words
  12. 12. Integrating Image Information Sales Images Labelling Model Pool + Palm Trees Hotel + Mountains Pool + Forest + Hotel + Sea Sea + Beach +Forest + Hotel Sales descriptions vector CONTENT  BASED   Recommender System
  13. 13. Image Labelling For Recommendation Engine Pragma&c  Deep  learning  for  “Dummies”  
  14. 14. Using Deep Learning models Common Issues “I don’t have GPUs server” “I don’t have a deep leaning expert” “I don’t have labelled data” (or too few) “I don’t have the time to wait for model training ” I don’t want to pay to pay for private apis” / “I’m afraid their labelling will change over time”
  15. 15. “I don’t have (or few) labelled data” -> Is there similar data ? Solution 1 : Pre trained models PLACES  DATABASE  US   SUN  DATABASE   205  categories   2.5  M  images   307  categories   110  K  images  
  16. 16. tower: 0.53 skyscraper: 0.26 swimming_pool/outdoor: 0.65 inn/outdoor: 0.06 Solution 1 : Pre trained models If there is open data, there is an open pre trained model ! •  Kudos to the community •  Check the licensing Example  with  Places  (Caffe  Model  Zoo)  :    
  17. 17. Solution 2 : Transfer Learning Credit  :    Fei-­‐Fei  Li  &  Andrej  Karpathy  &  Jus5n  Johnson  hYp://cs231n.stanford.edu/slides/winter1516_lecture11.pdf  
  18. 18. Solution 2 : Transfer Learning Use the network as a feature extractor Fine tune the model Keras makes it super easy ! (Possible to freeze layer to train just the top) hYps://github.com/PGu5/Deep_Learning_tuto  
  19. 19. PLACES  DATABASE   US  SUN  DATABASE   Training   (op5onal)   Pre-­‐trained  model   VGG16   tower: 0.53 skyscraper: 0.26 Re-­‐Training   Transferred  Data  :   Last  convolu5onal   layer  features   Re-­‐trained  model   TensorFlow   2  fully  connected  layers   Caffe   Model  Zoo     GPU   CPU   GPU   Leverage existing knowledge ! Solution 2 : Transfer Learning Accuracy:  72%,  Top-­‐5  Acc:  90  %  >  state  of  the  art  on  dataset  alone   Transfer  
  20. 20. Post Treatment & Results (Or how we transfer the labelling information) Using  Images  informa&on  for  BI  on  steroids    
  21. 21. Labels post-processing Complementary information Redondant information Issue with our approach: Solution : Matrix Factorization
  22. 22. Topic extraction with Non-Negative Matrix Factorization •  Non Negative Matrix factorization (NMF) X = WH •  X : image x tags, non negative •  W : image x theme •  H : theme x tag (scikit learn implementation) •  Most represented Themes •  Swimming-pool_Apartment_Putting-green •  Ocean_Coast_SandBar •  Coast_SeaCliff_RockArch •  Beach_Coast_BoardWalk •  Bridge_Viaduc_River •  Palace_BuildingFacade-Mansion •  Castle_Mansion_Monastery •  HotelRoom_Bedroom_DormRoom •  Dimension Reduction •  200x200 pixels -> 600 tags => 30 themes •  Faster content based filtering •  Image often sparse combination of themes Faster content based filtering •  Each theme has the same explication power Balanced vector for content based •  Explicability Each theme corresponds to a few labels
  23. 23. Image content detection Topic scores determine the importance of topics in an image TOPIC   TOPIC  SCORE   (%)   Golf  course  –  Fairway  –  PuHng  green   31   Hotel  –  Inn  –  Apartment  building   outdoor   30   Swimming  pool  –  Lido  Deck  –  Hot  tub   outdoor   22   Beach  –  Coast  -­‐  Harbor   17   TOPIC   TOPIC  SCORE  (%)   Tower  –  Skyscraper  –  Office  building   62   Bridge  –  River  –  Viaduct   38  
  24. 24. Note on model performance •  Images labels are used for similarity Calling herb field “putting green”: •  Is not important if all herbs field are called this way. •  Would be if we had lot’s of golf trips sales. •  Improving the NN performance ? •  Labels are used in NMF and reduced to themes •  Themes are used to calculate similarities for CB recommenders •  CB Recommenders are used as a feature in meta model •  Meta model give probabilities of purchase = order •  Users only check 10 sales… -> what is the change of online performance for 1% accuracy ?
  25. 25. Results
  26. 26. Results ? 1) Visits : •  France and Morocco •  Pool displayed 2) First Recommendation •  Mostly France & Mediterranean •  Fails to display pools 3) Only Images recommendation •  Pool all around the world •  Does not respect budget 4) Third column = Right Mix
  27. 27. Results ? 1) Visits : •  Spain •  Sun & Beach •  Pool displayed 2) First Recommendation •  Displays nature… 3) Only Images ? •  Pool all around the world 4) Third = Right Mix •  Get the bungalow feature !
  28. 28. Learned along the way What’s next ? AYrac5veness  =  %  visits  with  tag  /  %  sales  with  tag  -­‐1     For  ski  sales,  indoor  pictures  performs  beYer!    
  29. 29. What’s Next ? Kenya Prague Berlin Cambodia
  30. 30. Conclusion Do iterative data science ! Start simple and grow Evaluate at each steps Image labelling = BI on steroids Transfer Learning Kick-start your project Gain time and money Any Data Scientist can do it Deep Learning Don’t start from scratch ! Is there existing data ? Is there a pre-trained model ?
  31. 31. Thank you for your attention !
  32. 32. Basic Recommendation Engines •  Implementation •  Everything in SQL -  Vertica -  Then Impala -  Then Spark •  Collaborative filtering •  Score(user, future sale j) = Sum_{i:visited sale} sim(i,j) •  Content based filtering •  Sale profile: (sale_id, feature, value) •  User profile: (user_id, feature, value) •  Sparse coding + join does the trick Join  on   user_id   Join  on  user_id  +  sales  id  
  33. 33. One Meta Model to Rule Them All •  Negative sampling •  Take all purchases tuples : (user, product, timestamp)-> 1 •  Select 5 sales open at the same date the user did not buy -> 0 •  The model directly optimize purchasing probability •  Machine learning model •  Features : recommender systems. •  Logistic Regression Regularizing effect : we don’t want to overfit leaks. •  Reranking approach. Similar to Google or Yandex (Kaggle challenge)
  34. 34. Classification and not Detection problem… •  Only have probabilities of each class •  Selecting based on probability threshold fails •  Keeping all information is not sparse Labels post-processing Deep/Transfer Learning 5-10 tags per images2s/image with CPU x20 speed up with GPU Our images Cafe -> we keep 5 labels With probabilities per image
  35. 35. Solution 3 : What about APIs ?
  36. 36. Solution 3 : What about APIs ? •  Price •  Their cost often rather cheap. Ex: 100 K request for less than 300$ •  VS the one of redeveloping (probably not as well) •  Full Database scoring •  APIs are often limited query per month. •  Make sure to be able to avoid cold start problem •  Stability •  Use model versioning •  Avoid covariate shift, distribution drift
  37. 37. What about APIs ? Use for generating labels ! How to steal model: •  1) Score part of the database for training •  2) Train a model •  3) Score your entire database ! (Or don’t, it’s illegal) But I have only 5000 requests ? -> Use Transfer Learning ! Tramèr, Florian, et al. "Stealing Machine Learning Models via Prediction APIs." arXiv preprint arXiv:1609.02943 (2016).
  38. 38. What about APIs ? Use for generating labels ! Experiment: •  5000 requests on API. (4500/500 split) •  Transfer learning with MIT Places Pre-trained Model •  Scikit learn Multilabel model (Or don’t, it’s illegal) (demo, not used in any real project)
  39. 39. What about APIs ? Results Precision   75   Label   Probability   Label   Probability   landscape 1,0000 sunset 0,9998 sky 1,0000 no person 0,9996 outdoors 1,0000 water 0,9990 nature 1,0000 park 0,9849 rock 1,0000 river 0,9678 travel 1,0000 scenic 0,8031 Label   Probability   Label   Probability   beach 1,0000 ocean 1,0000 summer 1,0000 relaxation 1,0000 sand 1,0000 island 1,0000 tropical 1,0000 idyllic 1,0000 travel 1,0000 seashore 0,9998 seascape 1,0000 water 0,9997 (demo, not used in any real project) Recall   80  Accuracy   95  

×