As the fastest-growing type of content on the Internet, consumer produced videos are a wealth of information about the world that's essentially untapped. We present ICSI's research on the large-scale video search methods using an application that reveals the geo-location of a consumer- produced video based on its content. Gerald Friedland, University of California, Berkeley.

1. Finding the Needle
in the Video Haystack
Dr. Gerald Friedland
Director Audio and Multimedia Research
International Computer Science Institute Berkeley, CA
friedland@icsi.berkeley.edu

2. The Internet is Multimedia
2

3. Multimedia in the Internet
is Growing
3

4. Multimedia in the Internet
is Growing
3
• YouTube alone claims 48 72 100 hours
video uploads every minute.

5. Multimedia in the Internet
is Growing
3
• YouTube alone claims 48 72 100 hours
video uploads every minute.
• Youku (Chinese YouTube) claims 80k
video uploads per day

6. Multimedia in the Internet
is Growing
3
• YouTube alone claims 48 72 100 hours
video uploads every minute.
• Youku (Chinese YouTube) claims 80k
video uploads per day
• Flickr, Instagram, Liveleak, Vimeo...

7. 4

8. The Opportunity
5

9. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
5

10. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
– sociology,
5

11. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
– sociology,
– medicine,
5

12. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
– sociology,
– medicine,
– economics,
5

13. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
– sociology,
– medicine,
– economics,
– …
5

14. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
– sociology,
– medicine,
– economics,
– …
• Problem: Videos need to be searchable
beyond keywords.
5

15. The Opportunity
• Consumer-Produced Multimedia allows
empirical studies at never-before-seen
scale:
– sociology,
– medicine,
– economics,
– …
• Problem: Videos need to be searchable
beyond keywords.•
5

16. Our Approach
6
Ball sound
Male voice (near)
Child’s voice (distant)
Child’s whoop (distant)
Room tone
Cameron learns to catch (http://www.youtube.com/watch?v=o6QXcP3Xvus)

17. Our Approach
Multimodal exploitation of video content,
including audio and temporal information.6
Ball sound
Male voice (near)
Child’s voice (distant)
Child’s whoop (distant)
Room tone
Cameron learns to catch (http://www.youtube.com/watch?v=o6QXcP3Xvus)

18. Location Estimation
7
J. Choi, G. Friedland, V. Ekambaram, K. Ramchandran: "Multimodal Location
Estimation of Consumer Media: Dealing with Sparse Training Data," in
Proceedings of IEEE ICME 2012, Melbourne, Australia, July 2012.

19. Bayesian graphical
framework
8
{berkeley,
sathergate,
campanile}
{berkeley,
haas}
{campanile} {campanile,
haas}
Node:
Geoloca7on
of
the
image
Edge:
Correlated
loca7ons
(e.g.
common
tag)
Edge
Poten,al:
Strength
of
an
edge,
(e.g.
posterior
distribu7on
of
loca7ons
given
common
tags)
p(xi, xj|{tk
i } {tk
j })
p(xj|{tk
j })p(xi|{tk
i })