Semantic structuring and linking of event-centric data in the social web
1. WP4 - semantic structuring and
linking of event-centric data
in the social web
Raphaël Troncy
raphael.troncy@eurecom.fr / @rtroncy
2. WP4: Original Workplan
Task Title Duration
T4.1 Specification of an event model for representing
personal events
M1 – M32
T4.2 Linking personal events to multimedia content from
the wider world
M1 – M32
T4.3 Linking personal events to knowledge and promoting
diverse and opinionated interaction
M1 – M32
T4.4 Linking with people and discovering new contacts M1 – M32
T4.5 Promoting sustainable interaction between users M1 – M32
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 2
EURECOM: 76 PM
TUM-MMK: 4 PM
3. What are Events?
Events are observable occurrences grouping
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Experiences documented by Media
People Places Time
- 3
5. User Tasks and Challenges
ALIAS Final Meeting - Munich, Germany
Goals:
Discover PAST, PRESENT and FUTURE events
Live, relive and predict experiences through shared media
Enhance search and recommendations mechanisms
Support the decision making process
Challenges
Semantization of Data (events, media, people)
Reconciliation of sparse and highly heterogeneous data
User-centric design: discovering, deciding and recommending
Scalability, Caching and Optimization
04/06/2013 - - 5
6. Scenario on e-tickets
Supporting the elderly in finding events
and purchasing tickets online
Enabling the elderly to re-live past events they have
attended with their family and relatives
Need data about:
Events: http://linkedevents.org/ontology/ (LODE)
Media: http://www.w3.org/TR/mediaont-10/
Tickets: http://www.heppnetz.de/ontologies/tio/ns.html (TIO)
Meteo: http://purl.org/ns/meteo#
Personalization
Local event guides are more appropriate!
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 6
7. Events through the web …
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Event Directories
Media Platforms
Social Networks
Allows opportunistic/serendipitous discovery
Limited social features and Information
incompleteness (low coverage, weak decision)
Aids remembering and sharing past experiences
Lack of consistent events descriptions
Attendance: popularity and friends' whereabouts,
attendees’ opinions.
Limited discovery mechanism: need to filter out
relevant information.
Explore semantic connections to create better overviews of events
- 7
8. There’s a lot of information out there…
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 8
9. A lot of information…
http://www.flickr.com/photos/mwparenteau/432039783
04/06/2013 - ALIAS Final Meeting - Munich, Germany
10. Approach
ALIAS Final Meeting - Munich, Germany04/06/2013 -
Aggregate heterogeneous data sources
Unify into one data model using Semantic
Web technologies
Enrich with media, social data and
background knowledge
Identify meaningful and/or interesting
relationships between events, media, and
people
- 10
11. Architecture (generic dashboard)
ALIAS Final Meeting - Munich, Germany
Real-time Reconciliation
- Tag-based mapping
- Instance matching
Heterogeneous data sources
Data Crawling
RDF Conversion
RSS Update
Web Applications
SPARQL
REST API (Elda)
04/06/2013 - - 11
12. WP4.1: Data (Event) Model
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Ontology for Events:
LODE: http://linkedevents.org/ontology/
Extension for personal events:
Based on WP1 outputs (YOU, TUM-GSING, PME)
Taxonomy of event categories
Triple store gathering:
Events and media descriptions: user’s long-term
memories
System interaction: user’s short-term memory
- 12
13. ALIAS Final Meeting - Munich, Germany
Data Modeling – LODE ontology
04/06/2013 -
LODE is a minimal model that encapsulates the factual properties of
events: What, Where, When and Who.
- 13
14. Róisín Murphy at Nouveau Casino
04/06/2013 - ALIAS Final Meeting - Munich, Germany
350591
E0-001-005971169-9
- 14
16. Róisín Murphy at Nouveau Casino
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 16
17. Media explicitly associated with the event
04/06/2013 - ALIAS Final Meeting - Munich, Germany
2.3+ million images
over 120.000+ events
Machine tags
“lastfm:events”
APIs
11.000+ videos
over 3000+ events
- 17
18. Representing Media with Media Ontology
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 18
19. ALIAS Final Meeting - Munich, Germany
Data Crawling
Various API specifications
Policy management
Requests chaining
Different input and output schemas
Time-consuming
A Restful Service leveraging the commonalities of API specifications
A Web dashboard to easily handle data collection and interlinking
http://eventmedia.eurecom.fr/dashboard/
04/06/2013 - - 19
20. Real-Time Reconciliation
ALIAS Final Meeting - Munich, Germany
Participants
Media
Cost & description
Precise time
agents
04/06/2013 -
Need for semantic connections between events, media, agents and locations
- 20
21. ALIAS Final Meeting - Munich, Germany
Live Update
Tag = *:event
Per week
• 1500 Photos
• 130 events
Number of Items posted per days of the weekNumber of Items posted during the last days
04/06/2013 - - 21
22. Interlinking
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Event
Media
MusicBrainz
DBpedia
Geonames
Foursquare
Uberblic
Freebase
Agent
Last.fm
Eventful
MusicBrainz
DBpedia
Freebase
Uberblic
New York Times
Lieu
Last.fm
Eventful
Upcoming
DBpedia
Freebase
Foursquare
Geonames
Évènement
Last.fm
Eventful
Upcoming
DBpedia
Freebase
Uberblic
- 22
23. Instance Matching
Events/Agents/Locations semantic connections
SPARQL-based Instance matching
Filtering by the issuance date
Challenges
Structural heterogeneity
Different representations of the same value
User-generated knowledge featuring human mistakes
Different granularities between data sources
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 23
24. The events similarity is a mutual agreement of their factual properties
Based on top-k dependencies between properties
Evaluation: Precision: 96% and Recall: 94%
Ground truth of 300 matched instances
ALIAS Final Meeting - Munich, Germany
Events connections
p1 p2
dependency
title1 title2 0.30
place1 place2 0.28
title1 agent2 0.26
agent1 agent2 0.21
description1 title2 0.16
Minimal conditions to fetch
similar events using SPARQL
1st level
Refine the results
2nd level
04/06/2013 - - 24
25. Agents connections (precision-oriented)
Internal reconciliation + DBpedia, Musicbrainz and BBC
Label matching + contextual filtering (Cosine)
Granularity? (1 to N mappings) Cosine inequality
Locations connections (precision-oriented)
Internal reconciliation + DBpedia and Foursquare
It is straightforward thanks to the consistency of locations descriptions
Top-k Google search
Geographical distance
ALIAS Final Meeting - Munich, Germany
Agents & Locations connections
Cosine (d, a+b) > Cosine (d ,a)
04/06/2013 - - 25
26. About the dataset
SPARQL endpoint: http://eventmedia.eurecom.fr/sparql
Restful API : http://eventmedia.eurecom.fr/rest/event/
ALIAS Final Meeting - Munich, Germany
Event Agent Location Photos User
Last.fm 61,173 62,577 15,917 1;464,885 130,249
Upcoming 19, 276 15 10,269 308,148 3,932
Eventful 45,204 8,725 18,763 3,028 321
Total 125,650 71,313 44,948 1,760,787 159,263
DBpedia MusicBrainz Eventful
Last.Fm 13064 34247 2865
Eventful 2515 3616 -
04/06/2013 - - 26
27. WP4.2: Linking Events with Media
Start with media assets explicitly associated
with some events
Enrich the dataset using large scale multimedia
analysis combined with event properties
(time, location, topic)
Learning, training, re-training: image and speech!
Apply to personal media collections and enrich gradually
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 27
28. User Uploading Behavior Analysis
Most of event-related photos are uploaded in
five days.
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 28
29. Achievements
Devise a generic approach for collecting media
originating from specific events based on their
visual characteristics, capture date, location
and owner.
Propose two alternative approaches for
assessing visual similarity between canditate
media and known event media:
Visual matching against known event media (k-NN)
Visual event model trained from examples (SVN)
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 29
30. The Framework (Visual Matching)
The medias are collected via the Flickr API,
thanks to the events’ multiple facets
Visual Pruning and Owner Refinement are
employed to improve the performance
04/06/2013 - ALIAS Final Meeting - Munich, Germany
location
time
title
id
Event
query by
machine tag
query
by geo
query
by text
Media
Owner
Refine
Results
Pruning
Media
Media
- 30
31. Media Collection
• Multimodal features are exploited for
collecting data:
• Location
• Time
• Text
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Flickr
API
- 31
32. Visual Pruning and Owner Refinement
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 32
35. Automated Event Modeling Framework
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Event
tags
Rank tags
by frequency
tag1
tag2
tag3
……….tagN
Rank Photos
by distance to tags
Top N
tags
Pic1
Pic2
Pic3
PicM ……
Positive
Samples
Negative
Samples
Top M
Photos
Event
Model
- 35
37. WP4.3: Linking Events with Knowledge
Add semantic categories to media assets
Re-rank and re-organize media content for
personalized user groups
Language detection
Enrich multimedia presentation with rich
knowledge and content from the Semantic Web
Opinions: articles from newspapers (e.g. New York Times)
Topical diversity
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 37
38. Sharing photos vs event categories
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 38
39. Sharing photos vs number of participants
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 39
40. Sharing photos vs country of participants
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 40
41. WP4.4: People, Social Network and Events
User attendance, user profiles
Popularity and friends’ whereabouts
Similar interests, network, recommendations (behaviors)
Social networks
EventMedia Facebook application
https://www.facebook.com/TheEventMedia
(like us!)
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 41
43. Towards user profiles (LDA, T = 10)
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 43
44. Facebook native event application
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 44
45. Facebook native event application
Event model is weak:
missing rich categories, the performers and artists, etc.
the user cannot reuse existing events
Task: design a Facebook application that
allows for creating and sharing events, and
closely interacting with EventMedia
Task: crawl public events and related data
hosted by Facebook using the Graph API in
order to continuously feed EventMedia
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 45
46. EventMedia Facebook application
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 46
https://www.facebook.com/TheEventMedia
EventMedia meets here.com
Results (worldwide)
4763 venues have been interlinked
30011 events organized in those venues
465175 photos shared on social networks
during events at those venues
Examples:
http://here.com/services/place/528u173z
-672bb28e202e4032848cd6a1fe4b3851
(560 events, 9462 photos) on
eventmedia
http://here.com/services/place/040u29fp-
70d8361e99d64a4d978f107eaa69b15a
(2 events, 542 photos) on eventmedia
48. WP4.5: Promoting Sustainable Interaction
Developing end-user interfaces for searching
and browsing media content
Allow opportunistic/serendipitous discovery
Allow social feature, notification
Support decision making upon attending
events
Module for cross lingual chat or other IM
Google translation service
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 48
50. Web Application
Challenges:
Modern single-page app on top of semantic technologies
Fast, user-friendly and responsive UI
Dealing with high variability of data
Technologies:
Backbone.js: MVC framework enables complex and
responsive UI
Elda Linked Data API maps SPARQL queries
to RESTful URLs
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 50
51. What's on this evening?
ALIAS Final Meeting - Munich, Germany04/06/2013 - - 51
52. What's on this evening?
ALIAS Final Meeting - Munich, Germany04/06/2013 - - 52
53. Demo: Live your event
ALIAS Final Meeting - Munich, Germany04/06/2013 -
http://eventmedia.eurecom.fr
- 53
54. Demo: Parallel Facetted Browsing
ALIAS Final Meeting - Munich, Germany04/06/2013 -
http://eventmap-ui.appspot.com/
- 54
55. EventMedia has been awarded
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 55
http://www.youtube.com/watch?v=8dVD0SqCB6s
56. ALIAS Chat Module (with translation)
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 56
57. WP4: Deliverables (all delivered)
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Deliverable Title Due date
D4.1 Module for representing events related to user’s long term and
short term memory
M7 (Jan 2011)
D4.2 Module for cross-media linking of personal events to web content,
v1
M14 (Sep 2011)
D4.3 Module for knowledge enrichment of event descriptions M14 (Sep 2011)
D4.4 Module for personalized discovery of new contacts on line M30 (Dec 2012)
D4.5 Module for retrieval of opinionated content M33 (Mar 2013)
D4.6 Module for cross-media linking of personal events to web content,
v2
M32 (Feb 2013)
D4.7 Module for recommendation of topics and activities suitable for
maintaining social contacts
M32 (Feb 2013)
D4.8 Module for cross-lingual chat support M35
(May 2013)
- 57
58. WP4 Publications and Impact
26 high quality publications + 2 awards
+ 3 more under review: RecSys 2013 + Journal of Web Semantics +
Semantic Web Journal
Semantic Web Challenge 2012 at ISWC 2012: 1st price
IESD Challenge at EKAW 2012: 1st price
MediaEval Benchmark
2011: 7 participants (18 interested)
2012: 5 participants (25 interested)
2013: ongoing
2 PhD Thesis:
Xueliang Liu (03/12/2012): Event-based Social Media Data Mining
Houda Khrouf (scheduled for the end of 2013)
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 58
59. Publications 1/4
1. Raphaël Troncy, Ryan Shaw and Lynda Hardman. LODE: une ontologie pour représenter des
événements dans le web de données (on HAL). In 21th Journées Francophones d'Ingénierie des
Connaissances (IC'10), pages 69-80, Nîmes, France, June 8-11, 2010.
2. Raphaël Troncy, Ryan Shaw and Lynda Hardman. LODE: une ontologie pour représenter des
événements dans le web de données (on HAL). In 21th Journées Francophones d'Ingénierie des
Connaissances (IC'10), pages 69-80, Nîmes, France, June 8-11, 2010.
3. Raphaël Troncy, Bartosz Malocha and André Fialho. Linking Events with Media. In the Open
Track of the Linked Data Triplification Challenge, colocated with the 6th International
Conference on Semantic Systems (I-SEMANTICS'10), Graz, Austria, September 1-3, 2010.
4. Raphaël Troncy, André Fialho, Lynda Hardman and Carsten Saathoff. Experiencing Events
through User-Generated Media. In (ISWC'10) International Workshop on Consuming Linked Data
(COLD'10), CEUR Proceedings Vol. 665, Shanghai, China, November 7-11, 2010.
5. Xueliang Liu, Raphaël Troncy and Benoît Huet. Finding Media Illustrating Events. In 1st ACM
International Conference on Multimedia Retrieval (ICMR'11), Trento, Italy, April 17-20, 2011
6. Houda Khrouf and Raphaël Troncy. Réconcilier les événements dans le web de données. In
22nd Journées Francophones d'Ingénierie des Connaissances (IC'11), Chambéry, France, May
16-20, 2011.
7. Martha Larson, Adam Rae, Claire-Helene Demarty, Christoph Kofler, lorian Metze, Raphaël
Troncy, Vasileios Mezaris, Gareth J.F. Jones. Working Notes Proceedings of the Multimedia
Benchmark Workshop (MediaEval 2011), CEUR Proceedings Vol. 807, editors, Santa Croce in
Fossabanda, Pisa, Italy, September 1-2, 2011.
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 59
60. Publications 2/4
8. Symeon Papadopoulos, Raphaël Troncy, Vasileios Mezaris, Benoît Huet and Ioannis
Kompatsiaris. Social Event Detection at MediaEval 2011: Challenges, Dataset and Evaluation. In
MediaEval Benchmarking Initiative for Multimedia Evaluation, CEUR Proceedings Vol. 807, Pisa,
Italy, September 1-2, 2011.
9. Xueliang Liu, Benoît Huet and Raphaël Troncy. EURECOM @ MediaEval 2011 Social Event
Detection Task. In MediaEval Benchmarking Initiative for Multimedia Evaluation, CEUR
Proceedings Vol. 807, Pisa, Italy, September 1-2, 2011.
10. Marieke van Erp, Willem Robert van Hage, Laura Hollink, Anthony Jameson and Raphaël
Troncy. Proceedings of the International Workshop on Detection, Representation, and
Exploitation of Events in the Semantic Web (DeRiVE'11), CEUR Proceedings Vol. 779, editors,
Bonn, Germany, October 23, 2011.
11. Houda Khrouf and Raphaël Troncy. EventMedia Live: Reconciliating Events Descriptions in the
Web of Data. In (ISWC'11) 6th International Workshop on Ontology Matching (OM'11), Bonn,
Germany, October 23-27, 2011.
12. Houda Khrouf and Raphaël Troncy. EventMedia: Visualizing Events and Associated Media.
Demo Session at the 10th International Semantic Web Conference (ISWC'2011), Bonn, Germany,
October 23-27, 2011.
13. Xueliang Liu, Raphaël Troncy and Benoît Huet. Using Social Media to Identify Events. In (ACM
Multimedia) 3rd Workshop on Social Media (WSM'11), Scottsdale, Arizona, USA, November 28-
December 1st, 2011.
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 60
61. Publications 3/4
14. Liu, Xueliang; Huet, Benoit. Social event discovery by topic inference. In 13th International
Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2012), 23-25 May
2012, Dublin City University, Ireland
15. Houda Khrouf, Ghislain Atemezing, Thomas Steiner, Giuseppe Rizzo and Raphaël Troncy.
Confomaton: A conference enhancer with social media from the cloud. In 9th Extended
Semantic Web Conference (ESWC 2012), May 27-31, 2012, Heraklion, Crete
16. Houda Khrouf, Ghislain Atemezing, Giuseppe Rizzo, Raphaël Troncy and Thomas Steiner.
Aggregating social media for enhancing conference experience. In 1st International Workshop
on Real-Time Analysis and Mining of Social Streams (RAMSS 2012), June 4, 2012, Dublin,
Ireland
17. Symeon Papadopoulos, Emmanouil Schinas, Vasileios Mezaris, Raphaël Troncy and Ioannis
Kompatsiaris. Social Event Detection at MediaEval 2012: Challenges, Dataset and Evaluation. In
MediaEval Benchmarking Initiative for Multimedia Evaluation, CEUR Proceedings Vol. 927, Pisa,
Italy, October 4-5, 2012.
18. Sven Buschbeck, Anthony Jameson, Raphaël Troncy, Houda Khrouf, Osma Suominen and
Adrian Spirescu. A Demonstrator for Parallel Faceted Browsing. In (EKAW) Intelligent
Exploration of Semantic Data Workshop (IESD 2012), Winner of the IESD Challenge, Galway,
Ireland, October 8-12, 2012.
19. Marieke van Erp, Laura Hollink, Willem Robert van Hage, Raphaël Troncy and David A. Shamma.
Proceedings of the Second International Workshop on Detection, Representation, and
Exploitation of Events in the Semantic Web (DeRiVE 2012), CEUR Proceedings Vol. 902, editors,
Boston, USA, November 12, 2012.
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 61
62. Publications 4/4
20. Houda Khrouf, Vuk Milicic and Raphaël Troncy. EventMedia Live: Exploring Events Connections
in Real-Time to Enhance Content. In 11th International Semantic Web Conference (ISWC 2012),
First Prize Winner of the Semantic Web Challenge, Boston, USA, November 11-15, 2012.
21. Xueliang Liu and Benoit Huet. Gathering training sample automatically for social event visual
modeling. ACM International Workshop on Socially Aware Multimedia (SAM 2012), in
conjunction with ACM Multimedia 2012, 29 October 2012, Nara, Japan
22. Symeon Papadopoulos, Emmanouil Schinas, Vasileios Mezaris, Raphaël Troncy and Ioannis
Kompatsiaris. The 2012 Social Event Detection Dataset. In ACM Multimedia Systems (MMSys
2013), Dataset Session, Oslo, Norway, February 27-March 1, 2013
23. Sven Buschbeck, Raphaël Troncy, Anthony Jameson, Houda Khrouf, Adrian Spirescu, Osma
Suominen, Tanja Schneeberger and Eero Hyvönen. Parallel Faceted Browsing. In ACM
Conference on Human Factors in Computing Systems (CHI 2013), Interactivity Track, Paris,
France, April 27-May 2, 2013
24. Xueliang Liu and Benoit Huet. EventEnricher: a novel way to collect media illustrating events.
ICMR 2013, ACM International Conference on Multimedia Retrieval, April 16-20, 2013, Dallas,
Texas, USA
25. Xueliang Liu and Benoit Huet. Heterogeneous features and model selection for event-based
media classification. ICMR 2013, ACM International Conference on Multimedia Retrieval, April
16-20, 2013, Dallas, Texas, USA
26. Xueliang and Benoit Huet. On the automatic online collection of training data for visual event
modeling. Multimedia Tools and Applications, February 2013, ISSN: 1380-7501
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 62