This was my talk at the Dutch #MediaInnovators event at the SXSW2017, in a session together with #SoundandVision #VPRO and #VARA

1. Proprietary and Confidential
Exploration is the New Search
/ Lora Aroyo
/ Vrije Universiteit Amsterdam
/ IBM Netherlands (CAS)
/ Tagasauris Inc
/ http://lora-aroyo.org
/ @laroyo

2. 2
massive amount of online digital video content

3. 3
online video content is 64% of Internet traffic

4. 4
300h of video uploaded each min on YouTube

5. 5
in 2020 it would take a person more than 5 million
years to watch the videos shared online in a month

6. 6
there are videos out there
relevant to you …

7. 7
but …

8. 8
most online videos
NOT DESCRIBED
CAN’T BE FOUND
WON’T BE SEEN

9. 9
what can we do
to describe videos better?

10. 1
machine computation can do a lot

11. 11
recognise objects & concepts
Proprietary and Confidential

12. unlock rich semantics
Proprietary and Confidential
12

13. Proprietary and Confidential
[setting] WESTERN
[character] DOLORES
[story arc] FLASHBACK 7
[object] TRAIN
[action] SHOOTOUT
[mood] SURREAL
[audience] SHOCKED
[extras] 22
[role] BYSTANDER
13
discover creative contexts

14. 14
link across online videos
Proprietary and Confidential

15. 15
put all that descriptive information together
to know what’s in a video
on a second-by-second basis
6
so, we can …
Proprietary and Confidential

16. 16
add also behavioral information
to know how people watch & talk about videos
6
and, we can …
Proprietary and Confidential
Viewership
Channels profiles
Social Media Buzz
Audience Demographics
Audience Contexts
Platform profiles

17. 17
all this will make
video search better

18. 18
but we need more …

19. 19
SEARCH vs. EXPLORATION
/ helps only when you know what to search for
/ assumes you understand a topic
/ accuracy driven
/ click-through driven
/ helps when you don’t know what to search for /
/ helps you understand & deepen in a topic /
/ serendipity driven /
/ focused on engagement /

20. 20
what can we do
to support video exploration?

21. 2
human computation can do a lot

22. 2
CrowdTruth.org
bring more semantics of the real world

23. 2
diversity of opinions, sen)ments, contexts
collected in a decentralized way
aggregated views over collec1ons
introduces perspectives
http://crowdtruth.org

24. Proprietary and Confidential
24
consolidate experts & end users perspec)ves
for con1nuous improving video descrip1ons

25. 25
humans & machines can go a long way

26. Proprietary and Confidential
HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION
/ Machines help to break-down video into granular moments, i.e. shots & scenes
/ Machines generate multitude of paths within and across videos
/ Humans perform simple actions, e.g. watching, following and rating a path
/ Machines generalise from these actions using explicit semantics
/ Machines learn to evolve & improve exploration path
/ Orchestrate a continuous human and machine symbiosis
/ The ultimate aim is to reach a tipping point for video exploration,
e.g. web search, speech recognition

27. 2
to transform the online video experience

28. 2
Multimedia Explorations for
Smart Culture

29. 2
Interac1ve Explora)on & Discovery in Context
linking objects to events and en&&es
building automa1c storylines (narra1ves) on topics or objects
Access to integrated online mul)media collec1ons
using Linked Open Data to integrate metadata of various
heritage collec1ons
DIVE+
Aggregated views over the collec1on
collec1ng perspec&ves from crowds & niches
http://diveproject.beeldengeluid.nl/

30. 3
Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der;O ssenbruggen, J.R. van; Schreiber, G.
AutomaDc Heritage Metadata Enrichment with Historic Events Museums and the Web 2011 hKp://www.museumsandtheweb.com/mw2011/
papers/automaDc_heritage_metadata_enrichment_with_hi

31. 31
types

32. 32
filters

33. 3
explore entities
related
entities

34. 3
entity
exploration

35. 3
object
exploration

36. 3
people filter

37. 3
event filter

38. 3
narrative

39. 3

40. 4
Video Explorations with
Micro-moments

41. MOMENTS REDEFINE CURRENT VIDEO SEARCH
MOBILE/SOCIAL
OPTIMIZED
RELEVANT SEARCH RESULTS
PREVIEW SPECIFIC MOMENTS
OR WATCH FULL VIDEOS
DISCOVER ACTIONABLE
MOMENTS WITHIN VIDEO

42. MOMENTS: PERSONAL VIDEO CHANNEL
SEARCH FOR
ANYTHING, WITH
ANYTHING
HYPERMEDIA PLAYER
W/ LINKED MOMENTS &
FULL VIDEOS
AI ACTIVELY
LEARNS USER
PREFERENCES
DISCOVERS
MORE & MORE
CONTENT

43. AI, DEEP LEARNING & NETWORK EFFECTS
Ensemble algorithmic processing
Video, audio & text
Extract & understand entities
Does what computers are good at
AI & DEEP LEARNING:
//
//
// Human assisted computing
Collective intelligence (incl. fans)
Waze effect
Does what humans are good at
NETWORK EFFECTS:
//
//
//
OBSERVE & LEARN
VERIFY & EXTEND
*U.S. Patent Application #13/863,751
Web-scale layer of structured, linked data.
////
DOMAIN-SPECIFIC DATA
MOMENTS

44. 4

45. 4

47. 4

48. Proprietary and Confidential
HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION
/ Machines help to break-down video into granular moments, i.e. shots & scenes
/ Machines generate multitude of paths within and across videos
/ Humans perform simple actions, e.g. watching, following and rating a path
/ Machines generalise from these actions using explicit semantics
/ Machines learn to evolve & improve exploration path
/ Orchestrate a continuous human and machine symbiosis
/ The ultimate aim is to reach a tipping point for video exploration,
e.g. web search, speech recognition

49. 49
to explore strange new worlds ….
to boldly WATCH what no-one has WATCHED before

50. Proprietary and Confidential
Exploration is the New Search
/ Lora Aroyo
/ Vrije Universiteit Amsterdam
/ IBM Netherlands
/ Tagasauris Inc
/ http://lora-aroyo.org
/ @laroyo