The document discusses finding and organizing media like photos and videos that illustrate real-world events. It describes scraping various event directories and linking the data. Media can be found by searching based on geo-tags, titles, or uploader. Visual analysis is used to prune media results. The goal is to help users explore, annotate and share media related to events. Challenges include interlinking diverse data sources and detecting unscheduled events.
DevEX - reference for building teams, processes, and platforms
Finding media illustrating events
1. Finding Media
Illustrating Events
Xueliang Liu <xueliang.liu@eurecom.fr>
Raphaël Troncy <raphael.troncy@eurecom.fr>
Benoit Huet <benoit.huet@eurecom.fr>
2. What are Events?
Events are observable occurrences grouping
People Places Time
Experiences documented by Media
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -2
8. Organize the mess
Event
Media
Scrape event directories
Link the information
Find media illustrating events
Design the application Interface
Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -8
9. Róisín Murphy at Nouveau Casino
E0-001-005971169-9
350591
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy -9
10. Representing Events with
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 10
11. Interlinking
Linking Agents with
Freebase, Dbpedia, MusicBrainz
Linking Venues with
Geonames, Dbpedia, Foursquare (via Uberblic)
Linking Events with
Last.fm, Upcoming, Eventful
Linking Categories with
Facebook, Eventful, Upcoming, Zevents, LinkedIn,Eventbrite,
TicketMaster
Linking Users with
Social Graph API
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 11
12. Róisín Murphy at Nouveau Casino
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 12
13. Media explicitly associated with the event
APIs
Machine tags
“lastfm:events”
4790 photos, 263 1.7 million images over
videos over 110 events 108.000 events
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 13
14. Representing Media with Media Ontology
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 14
15. How much data is there?
Event Agent Location Photos User
Last.fm 57,258 50,150 16,471 1,425,318 18,542
Upcoming 13,114 0 7,330 347,959 4,518
Eventful 37,647 6,543 14,576 0 0
Total 108,019 56,693 38,377 1,773,277 23,060
1,248,021 geo-tagged photos
by propagating information from events!
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 15
16. How fast media are uploaded?
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 16
17. Finding more media that illustrate an event
A. Compute the bounding box area of a venue
B. Retrieve all media geo-tagged in this area
C. Retrieve all media with a similar title
D. Prune the results with visual analysis
E. Extend the result set with all media from the
same uploader
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 17
18. A. Bounding box of Nouveau Casino?
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 18
19. B. 74 photos taken in this area this day
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 19
20. C. 85 additional photos with a similar title
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 20
22. How is the visual pruning performed?
Model dataset: photo id + photo geo
Testing dataset: similar title
Low-level features used:
Color moments, Gabor texture, Edge histogram
L1 distance on the K-nearest neighbors
Threshold
Min L1 distance between two model image pairs
Conservative approach
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 22
23. E. 66 photos after uploader heuristics
hellerpop
DustGraph / Stefan
cartoixa
13 photos 46 photos
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 23
24. Same process for videos
1 video (id)
3 videos (geo)
26 videos (title)
Visual pruning
performed on
key frames
Nb positive > 50%
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 24
25. How illustrated are the events?
Query By ID Query By Geo Query By Title Visual Pruning Heuristic
Photos 5 74 (74) 85 (85) 6 (6) 66 (66)
Videos 1 3 (0) 23 (0) 13 (0) -
(title)
Videos 10 (10)
(title+venue)
20 events
Model dataset: 785 photos
Testing dataset: 1766 photos (1573 positive, 193 negative)
Results: 439 photos (99% precision, 28% recall)
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 25
26. User Interface
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 26
27. Conclusions
Method for finding media illustrating scheduled
events
Search media with machine and geo tags
Search media with title and normal tags
Prune visually and retrieve all media from confirmed users
Challenge: do not necessarily trust the geo-coordinates
Event-based approach for users to explore,
annotate and share media
UX can help semantics, semantics can help UX
Outstanding challenges in interlinking and curating the data
Detecting events from social media activity
19/04/2011 - Finding Media Illustrating Events - ICMR 2011, Trento, Italy - 27