Event-based Summarization for Media Hyperlinking @ EBMIP 2013 Workshop on Event-based Media Integration and Processing co-located with ACM Multimedia 2013 – October 21-22, Barcelona, Spain
The exponential growth of media sharing and demand on the Web comes with a need for effective methods to explore them. Hence, media hyperlinking, which consists in linking together videos based on their content, uncovering the relation between them, is becoming an important functionality for providing users with a way to navigate between video entities and satisfy their information needs. Thanks to such technology, multimedia search can often be replaced by recommendation. A particular usage of hyperlinking is to provide, through a second screen application, extra information or content about the video watched on a main screen (TV). In this talk, we will focus on media hyperlinking from the news: the task at hand consists in locating and identifying relevant media items, and display them on the second screen. The related material is selected based on underlying events that will be detected in the news: events are seen as structuring elements, defined in terms of date, location, intent and attendance. Two approaches for event-based mining of such additional and related information will be presented. Each of them satisfying a different user information need.
Event-based Summarization for Media Hyperlinking @ EBMIP 2013 Workshop on Event-based Media Integration and Processing co-located with ACM Multimedia 2013 – October 21-22, Barcelona, Spain
2. Hyperlinking and Second Screen
Nielsen 12/2012: 85 Percent Of Tablet And
Smartphone Owners Use Devices As “Second
Screen” Monthly, 40 Percent Do So Daily
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking
3. Hyperlinking and Second Screen
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
4. Audiovisual Explosion
EU alone hosts 500+ online video platforms
42.7m hrs of footage in online archives of
broadcasters and producers (61% of archive
footage is online)
UGC on the advance:
YouTube receives 60 hrs of video/minute
Internet video is now 40 percent of consumer
Internet traffic, and will reach 62 percent by the
end of 2015
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
5. Hyperlinking at the fragment level
MediaMixer:
“set up and sustain a community of video producers,
hosters and redistributors…”
“who will be supported in the adoption of semantic
multimedia technology…”
“to build an European market for media fragment repurposing and re-selling.”
http://community.mediamixer.eu
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
6. Hyperlinking and Second Screen
LinkedTV: linking television to
Web content
Second screen scenario for
enriching television content
and achieving interaction
between user and content
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
7. Enrichment framework
Input video
Media Annotation
Segment annotation
Annotated
set of media
items
Media
items
Web Media Search
Multimedia Mining
Set of relevant
media with cues
Personalization,
Display, etc
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
8. Topic Enrichment
Need to enrich given topics:
based on terms/topics appearing in video (passive or
active)
based on user query (active)
Select and organize media items
Different approaches for searching for information
Diverse information rendering
What are people interested to see ?
In this work, we propose and investigate,
Event-based socially aware enrichment
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
9. Event-based socially aware enrichment
Publish content
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
10. Event-based socially aware enrichment
Search term
Google
Trends
09/10/2013
Query content
Benoit HUET - Event-based Summarization for
Media Hyperlinking
11. Outline
Related work
Framework
Enrichments for Passive Second Screen
Query Formulation
Static Event visualisation
Enrichments for Active Second Screen
Trends mining
Focused search
Video selection
Evaluation and Results
Conclusion
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
12. Related work
Chieu and al. (ACM CIGIR 2004)
Query based event extraction along a timeline
Yan et al. (CIKM2012)
Visualizing timelines: evolutionary summarization via iterative reinforcement
between text and image streams.
Christiansen et al. (WIDM '12)
Modeling topic trends on the social web using temporal signatures
Chen et al (Knowledge and Data Engineering 2007)
Hot Topic Extraction Based on Timeline Analysis and Multidimensional
Sentence Modeling
Tan et al. (ACM MM’10)
Topical summarization of web videos by visual-text time-dependent
alignment.
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
13. Our Contributions
Event-based methodology
4W paradigm (Who, What, Where, When)
Mining events from user contributed data and
search trends
People-based point of view (not editorial choice)
Event discovery
Event illustration
Mining from a pool of potential media items
Choice of illustrative media item(s)
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
14. Passive Second Screen Framework
Aim:
Enrich & Illustrate the Event on the TV Screen
Step1: parse the semantic from query string
Step2: Extract events from Social News site
Step3: Illustrating with tag cloud and image collage
Step3
Step1
query
Step2
time
location
events
topic
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
15. Event Querying
Events are queried from Digg.com
a news aggregator with an editorially driven front page
Two advantages:
1. Save storage and
computation
2. Leverage from
collective intelligence
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
16. Textual Summarization
Source: Full textual search on Twitter;
query words = event title
filter by time = event occurrence date
Representation: Tag Cloud
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
17. Visual Summarization
Source: Full textual search on Google Images
Query words = event title
Sorted by the similarity of image and event title
Representation: Visual Collage
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
20. Active Second Screen Framework
Aim:
Propose Enrichments which Illustrate the Event
on the TV Screen
Interaction with the viewer
Multiple choices
Dynamic rendering
TV / Main screen might need to be paused
Leverage on user generated content and collective
intelligence
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
21. Active Second Screen Framework
Goal = automatically create and illustrate a
timeline of events on a given topic
Event: occurrence of abnormal activity on a
limited time segment, that captured a lot of
interest and triggered massive web search
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
22. Trends mining
Bursts = peaks: sudden increase in the search
volume => unexpected event, massive search
behavior
Long-term events: abnormal high interest
spread over several time units
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
23. Trends mining
Granularity:
Trend value: Week (imposed by Google trends)
Trend keywords: Monthly
Feature:
Burst value= left derivative of the trends values
Threshold to select bursts:
Adaptive threshold
mean + standard deviation
Output:
Set of week dates
Objective:
link trendy time-segment to some event in the real world
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
24. Focused search on time segments
Goal: search for items relevant to the event on
the extracted dates
Extracting keywords for a focused search:
Input: initial search term + dates (time segment)
Query to Google trends
Output: list of associated rising search terms
Focused media search:
Private / Public Media repositories
Social Media platform
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
25. Video Selection
Goal: represent each event with a media item
Features:
Using text features (from media title + description)
Discard non-English content
TF-IDF with cosine distance
Media Item Selection
Item with average similarity to all other media items
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
26. Example query: Snowden
1. Get trends
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 26
27. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
2. Mine time segments
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 27
28. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
3. Extract associated keywords
(month granularity)
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 28
29. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
3. Focused media search:
Tag cloud of associated text
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 29
30. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
4. Illustrative video selection
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 30
31. Example query: Snowden
Snowden identity was
revealed on the 9th
Snowden flied to Russia on
the 23rd
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
5. Ground truth
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 31
32. Experiments
Focus on people: automatic event-based
timeline generation of 5 celebrities
“Ground truth” extracted from expert
biographies and Wikipedia (from 2004)
Beyoncé
Knowles
28 events
10/21/2013
Mark
Zuckerberg
27 events
Oscar
Pistorius
9 events
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
Batman
9 events
- p 32
Edward
Snowden
43 events
33. Event extraction Evaluation
Performance of our approach to detect
important events
TP: true positive, TP: false positive, FN: false negative, DE: discovered event
query
# bursts
TP
FP
FN
DE
Beyoncé K.
9
7
1
21
1
M.Zuckerberg
6
2
4
25
0
O.Pistorius
1
1
0
8
0
Batman
6
2
2
7
2
E. Snowden
2
2
0
41
0
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 33
34. Event discovery: Beyoncé
Burst week: 22nd to 28th July, 2013
Nothing in the ground truth
Associated keywords:
Beyonce fall video
Beyonce falling
Beyonce falls
Beyonce orlando
Beyonce stairs
Caida beyonce
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 34
35. Video representative evaluation
Challenge: choice of an illustrative video
Event Relatedness ER (mean over all events):
evaluation of the content of the video (not the quality)
Query
# events
Event
Relatedness
Beyoncé K.
8
0,81
M. Zuckerberg
2
1
O. Pistorius
1
0
Batman
4
1
E. Snowden
2
1
Based on user generated textual features
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 35
36. Example query: Pistorius
Week: 10th to 16th February, 2013
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
37. Example query: Pistorius
Week: 10th to 16th February, 2013
Associated keywords:
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
38. Example query: Pistorius
Week: 10th to 16th February, 2013
Associated keywords:
DataSet tag cloud:
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
39. Example query: Pistorius
Week: 10th to 16th February, 2013
Associated keywords:
Dataset tag cloud:
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
Illustrative video:
Nothing about Pistorius himself or the murder case
One of his opponents
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking
40. Example query: Pistorius
Week: 10th to 16th February, 2013
Associated keywords:
Dataset tag cloud:
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
Illustrative video:
Why: associated text
BBC News -Paralympic Star Oscar Pistorius 'shoots girlfriend' 2013 Inscreva-se em
nosso canal. Subscribe yourself to our channel. Suscríbase a nuestro canal, News for
Oscar Pistorius girlfriend shot dead Oscar Pistorius charged with murder after
girlfriend shot dead The Guardian ?- 17 hours ago South African athlete will appear in
court on Friday after Reeva Steenkamp was found dead at his home in Pretoria.
Pistorius' girlfriend killed on Valentine's Day she was looking forward to CNN? - 15
hours ago South African sprinter Oscar Pistorius charged with murder after girlfriend,
Reeva Steenkamp, shot dead in his home New York Daily News? - 6 hours ago
Pistorius 'shot, killed' girlfriend - The Courier-Mail www.couriermail.com.au/.../
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking
41. Conclusion
Socially Aware Hyperlinking Approaches
Automated Event-based Illustration of topics
Creation of an Event-based media-illustrated
timeline related to a topic
“new” Events Discovery = non-official but
triggered massive attention/searches
Future work:
Work with visual features, ASR.
Adding facial information
Work on long-term events
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking
42. References
http://www.linkedtv.eu
http://www.mediamixer.eu
M. Sahuguet and B. Huet, “Mining the Web for Multimedia-based
Enriching”, Multimedia Modeling 2014, Dublin, Ireland
X. Liu and B. Huet “Event Representation and Visualization from
Social Media”, Pacific Rim Conference on Multimedia 2013,
Dec 13-16, Nanjing, China
M. Sahuguet and B. Huet, “Socially motivated multimedia topic
timeline summarization”, 2nd ACM International Workshop on
Socially-Aware Multimedia, In conjunction with ACM Multimedia
2013, 21 October 2013, Barcelona, Spain
X. Liu and B. Huet “On the automatic online collection of training
data for visual event modeling” Multimedia Tools and
Applications, February 2013, ISSN: 1380-7501
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking
44. Thank you for your attention
Questions?
Benoit Huet
EURECOM
tel: +33 (0)493008179
Campus SophiaTech, 450 route des Chappes
fax: +33 (0)493008200
06410 Biot Sophia Antipolis
email: Benoit.Huet@eurecom.fr
France
http://www.eurecom.fr/en/people/huet-benoit
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
Notas del editor
Two first reference tackle the problem of timeline summarization from a topic-oriented collection of documents => summarizationThe 2 next references deal with hot topics discovery and the evolution of a topic over time => more trends analysis orientedLast work is the most similar to ours. 2 main differences: our trend analysis is adaptive to the dataset (they use a fixed threshold for any query) ; timeline illustration is made from news (“official”) while we are people-oriented.