While not new at all, Machine Learning has been on the rise of the past years, both because of the ubiquity of data and because of the increase in adoption of Cloud Computing. In recent years, however, ML has become more prevalent than ever - mainly due to its ease of use and its accessibility to non-mathematicians. In some cases, ML can do things that would’ve been extremely difficult, if not impossible, for us to achieve in the past. In other cases, however, ML is here to assist us, rather than replace us, by relieving us of our most boring and repetitive tasks, and this often has to do with the low accuracy in which ML models operate. In this talk we are going to build business workflows using the joint effort of humans and software to automate those boring tasks, while compensating for the inaccuracy of ML with human intervention.

1. How I built a ML-human hybrid
workflow using Computer Vision
Amir Shitrit
Software Architect
amirs@codevalue.net
@amir_shitrit
http://codevalue.net

2. 2

3. What to (not) expect
3

4. About Me
Amir Shitrit
 Software Architect
 Love the cloud and also distributed systems
 And animals!
4

5. 5

6. 6

7. A short story
7

8. The “catalog”
8

9. Also
9

10. Selling on Simania

11. 11
Photo by Ed Robertson on Unsplash

12. What do I have to do with it?
12
Me, when I was younger

13. First things first
13
 my own digital catalog

14. Then, search the book in the catalog
 But how?
14

15. 15

16. Option 1
 BARCODE
 DANACODE
 WHATEVERCODE

17. Option 2
 OCR
17

18. Some history about OCR
18

19. The challenge
19

20. Using Google’s
20

21. OCR Services
 TEXT_DETECTION
21
 DOCUMENT_TEXT_DETECTION

22. Problems with OCR
 Not exactly accurate
 Photographing each book individually
22

23. 23

24. 24

25. The best one!
25

26. About to give up
26
Photo by Steve Johnson on Unsplash

27. Aha!
27

28. When you come to think of it …
 Who needs accuracy anyway?
28
Photo by Katerina Holmes from Pexels

29. Some classification-related terms
29
accuracy =
TP + TN
total
precision =
TP
actual results
recall =
TP
predicted results

30. Some classification
-related terms
 Accuracy
 Precision
 Recall
30

31. Precision over accuracy
31

32. Search for in “broken” text
32

33. Still some differences
 ‫ספורי‬ ‫אוצר‬
6
‫לפני‬ Y
 ‫הענה‬ ‫לפני‬ ‫ספורים‬ ‫אוצר‬
33

34. Fuzzy search in Elasticsearch
 https://en.wikipedia.org/wiki/Levenshtein_distance
34

35. Fuzzy search in Elasticsearch
 https://en.wikipedia.org/wiki/Levenshtein_distance
35

36. And if that doesn’t work?
 HaaS = Human as a Service
 HITL
36

37. What about multiple books?
37

38. Again, Vision API to the rescue
38

39. Back to the metrics
 Object Detection
 OCR
 Fuzzy Search
39

40. Which metrics should I use for: Object Detection
40

41.  Precision
41
Which metrics should I use for: OCR

42. Which metrics should I use? (Fuzzy) Search in catalog
42

43. Demo
 Demo
43

44. What we just saw
44

45. Code
45

46. Costs
46
 Google Cloud Vision API
 1K OBJECT_DETECTIONS = free
 1K OCR = also free
 Every next 1K is 1.5$
 Azure
 Storage – $0.0196 per GB
 (hot LRS standard storage)
 Egress - $0.05 per GB
 (for serving the beautiful web UI)
 Compute – currently on-prem

47. So…
47

48. Key takeaways
 Ain’t no need a math wiz
 Cloud services are easy to use, but
 Choose the right metrics for the right steps
 AI + NI = Better together
 If you can’t join them, beat (the crap out of) them
48

49. What’s next?
 Taking on Amazon
49

50. Resources
 Google Cloud Vision API
 https://cloud.google.com/vision
 Elasticsearch – Fuzzy query
 https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-
query.html
 HITL on Wikipedia
 https://en.wikipedia.org/wiki/Human-in-the-loop
 Binary classification metrics
 https://en.wikipedia.org/wiki/Binary_classification
 https://medium.com/@shrutisaxena0617/precision-vs-recall-386cf9f89488
50

51. Q
A
51

52. Amir Shitrit
Software Architect
amirs@codevalue.net
@amir_Shitrit
http://codevalue.net