New Approaches to Interactive Multimedia Content Retrieval from different Sources

New Approaches to Interactive
Multimedia Content Retrieval from
different Sources
Julián Moreno Schneider
LaBDA Group, Computer Science Department
Universidad Carlos III de Madrid, Spain
jmschnei@inf.uc3m.es

Content
New Approaches to Interactive Multimedia
Content Retrieval from different Sources
2
 Motivation
 Background
 Objectives
 Proposal
 Sports-domain Scenario and Validation
 Adaptation techniques and Validation
 Health-domain Scenario and Evaluation
 Future directions
 Publications

Motivation (I)
3
 Multimedia content is increasing at staggering rates
 Devices and formats are very diverse and move
away from traditional modes.

Motivation (II)
4

Motivation (III)
5
 Problem description
 Current Limitation: multimedia elements retrieved by
textual metadata
 Users need access in a transparent, faster and easier
way to many independent sources containing information
in different formats (such as video, text, audio, images,
graphics, etc.).

Motivation (IV)
6
 Clarifying the problem
 Seeking the album of a song having the audio file and the
artist’s name
+ ‘I want you back’  The Jackson 5

Content
7
 Motivation
 Background
 Objectives
 Proposal
 Formal model
 Adaptation techniques and validation
 Publications

 Organization by the components
 Multimodal Information (Collections)
 Query
 Information Retrieval Approaches
 Retrieval Selection
 Fusion
 Interactions
Background (I)
8

 Multimodal Information
Background (II)
9
Image and Text
Image and Audio
Image and Video
Text
and
Video
Multimodal
Federated Web Search Track
Jou et al. [2013]

Background (III)
10
 Query Modalities
Text (Monomodal)
Image (Monomodal)
Text and Image
Video and Image
Text and Audio
Multimodal
Yang et al. [2002]
de Vries [1998]
Marchand-Maillet et al. [2011]

Background (IV)
11
 Retrieval Approaches
 Text Retrieval
 Low-level features
 Combined Indexes
Low-level features
Text-based
(metadata)
retrieval
Full text
retrieval
Salton et al. [1975]
Romberg et al. [2012]Lana-Serrano et al. [2011]

Background (V)
12
 Retrieval Engine Selection Strategy
Unknown
Strategy
By Elements
By Query Terms
Probabilistic
Renaud and Azzopardi [2012]
Demner-Fushman et al. [2012]
Romberg et al. [2012].
Chernov et al. [2006]
Balog et al. [2012]

Background (VI)
13
 Result Fusion or Aggregation
 Pre-RE fusion:
 Joint indexes (prior fusion)
 Post-RE fusion
 Randomness
 Source or type
 Scores (unification)
 Aggregated search
Arampatzis et al. [2011]
Balog et al. [2012]
Romberg et al. [2012]

Background (VII)
14
 Semantic Knowledge
 Annotation-based Retrieval
 Multimedia Ontology Retrieval
 Combination of
multimedia ontologies
Worring et al. [2007]
Medina-Ramírez [2007]
Castells et al. [2007]

Background (VIII)
15
 User Interactions
• Relevance Judgments
o Direct
o Indirect
Document browsing
Clicks logging and analysis
Query history
• Log Analysis
• Surveys
• Dwell time
• Eye tracking
• Gestures, lip motion, speech
and facial expression

Discussion
16
 Limitations
 Handler strategy (specially adapted to the user
experience)
 Multimodality in query and results
 Multimodal semantically related collection
 Spanish
 Out of the scope of this thesis
 Retrieval approaches
 Fusion algorithms
 Innovation in Interaction Logging

Content
17
 Motivation
 Background
 Objectives
 Proposal
 Formal model
 Publications

Objectives (I)
18
1
• Propose a formal model to define
multimodal information retrieval (IMR)
systems.
2
• Develop two multimodal prototypes based
on the proposed model and evaluate them
3
• Design and define techniques to adapt
MIR System based on user experience.

Objectives (II)
19
 Methodology
Formal Model
Interactions
Sports Domain
Scenario
Adaptation techniques
Evaluation
Evaluation
1
2 3
4 5
Health Domain
Scenario
6

Content
20
 Motivation
 Background
 Objectives
 Proposal
 Formal Model
 Publications

Formal Model (I)
21
 Architecture is composed by the most common
components used in IR models.

Formal Model (II)
22
 Multimodal Information
 𝒞 = 𝐶1, 𝐶2, … 𝐶 𝑁
 𝐶𝑖 = 𝐷𝑖1, 𝐷𝑖2, … 𝐷𝑖𝑀
 𝐷𝑖𝑗 = 𝑑𝑖𝑗1, 𝑑𝑖𝑗2, … 𝑑𝑖𝑗𝑃
 𝑑𝑖𝑗𝑝 {text, audio, video, image}
 SemanticRelations
 Multimedia: 𝛿 𝑚
= 𝐷𝑖𝑗, 𝐷 𝑥𝑦
 isFrameOf(image17, video004)
 Semantic: 𝛿 𝑠
= 𝐷𝑖𝑗, 𝐶𝑜 𝑝 ∧ 𝐶𝑜 𝑝, 𝐷 𝑥𝑦
 shows(image23,FC_Barcelona) ∧
mentions(video12,FC_Barcelona)

Formal Model (III)
23
 Multimodal Query
 𝑄 = 𝑞1, 𝑞2, … 𝑞 𝐾
 ℳ(𝑄) =
𝑚𝑜𝑛𝑜 ∨ 𝑖, 𝑗 ℳ 𝑞𝑖 = ℳ(𝑞 𝑗)
𝑚𝑢𝑙𝑡𝑖 ∋ 𝑖, 𝑗 ℳ 𝑞𝑖 ≠ ℳ(𝑞 𝑗)
 RetrievalEngines (RE)
 𝑅𝐸 = 𝒞, 𝑄, 𝒫
 𝒮 = 𝒫 𝑄
 𝒮 = ℛ1, ℛ2, … ℛ 𝐿
 Example of RE: 𝒫1 =
TF − IDF
𝒫2 = Color Histograms

Formal Model (IV)
24
 Handler
 ℋ = ℰ, 𝑄, Ξ
 Ξ: set of rules
 𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛𝑠 → ℰ = 𝑅𝐸1, 𝑅𝐸2, … 𝑅𝐸 𝑍
 Example:
 query = question → ℰ = 𝑄𝐴, 𝐹𝑇

Formal Model (V)
25
 Results Fusion
 𝒮𝑓𝑖𝑛𝑎𝑙 = 𝒜 · 𝒱
 𝒮𝑓𝑖𝑛𝑎𝑙 = ℛ1 ⋯ ℛ 𝑀
 𝒜 = 𝛼1 ⋯ 𝛼 𝑁
 𝒱 = 𝒮1 ⋯ 𝒮 𝑁 =
𝛽11 ⋯ 𝛽1𝑁
⋮ ⋱ ⋮
𝛽 𝑀1 ⋯ 𝛽 𝑀𝑁
 Interactions
 ℐ = 𝑈, 𝑆𝑒, 𝒯, Φ
 𝑈useridentifier, 𝑆𝑒sessionidentifier, 𝒯 timestamp and
Φadditionalinformation
 Visualizationusingexistingtechniques (clouds, lists,
grouping, …)

Content
26
 Motivation
 Background
 Objectives
 Proposal
 Formal Model
 Publications

Proposal: Sports-Domain Prototype (XII)
27
 Architecture

Proposal: Sports-Domain Prototype (VI)
28
 Buscamedia Collection
 Developed in the framework of
the Buscamedia Project
 Sports Domain
 Multimodal documents
 10000 Texts
 350 Images
 15 Videos
 Recruited in October 2010
 Semantically Related

Proposal: Sports-Domain Prototype (VII)
29
 Text, Audio and Text + Image
 Información sobre el accidente de la foto +

Proposal: Sports-Domain Prototype (VIII)
30
 Retrieval Engines
 Question Answering (QA),
Full Text Search (FT), Ontology-based
Search (ONT), Object Detection in Image (ODI), OCR
in Image (OCRI), Audio Transcription (AT)
 RE selection (Handler)
 Simple Approach
 Expert-defined rule-based approach
 Question  {QA,FT}
 Txt(short)+img  {ONT,FT,{ODI,OCRI}}

Proposal: Sports-Domain Prototype (X)
31
 Fusion Strategy: Round-Robin Approach

Proposal: Sports-Domain Prototype (XI)
32
 User Interactions
 Searches
 Documents Browsing
 Relevance Judgments
 Visualizations

Validation and Results (I)
33
 Objective
 User Preferences
 Requested sources?
 Preferred modes?
 Preferred visualizations?
 More used query modes?
 Expert-defined Rules Validation
 Comparison with Baseline (Full Text Search Engine)
 Web Interface to test with users
 2 months
 235 users

Validation and Results (II)
34
 What query types are used?
 981 queries: 239 predefined and 742 user-generated.
 Short, long and question queries more often than
concepts.
 Sources ‘usage’ by query type.
 Visualizations
 Answer List, Answer / Concept Cloud, Concept
Groups, Individual Document

Validation and Results (III)
35
 Baseline: logs from users
 IR performance
 Mean Average
Precision
(MAP)
 Mean
Reciprocal
Rank (MRR)
 R-Precision

Adapting IR Functionality (I)
36
 Motivation
 Background
 Objectives
 Proposal
 Formal Model
 Publications

Adapting IR Functionality (II)
37
 Rule-Based MIR
(qmode=t, qtype=long)  ont , qa , f t
(qmode=t, qtype=question, qlength=14)  qa , f t , ont
(qmode=t, qtype=short, qlength=2, qentities=alonso)  ont, qa, ft

Adapting IR Functionality (III)
38
 Adaptation architecture

Adapting IR Functionality (IV)
39
 Classification Algorithms
 Decision trees, multilayer perpectron and simple K-means
 Query features
 Mode, type, length, number of entities, entities, number of
verbs, topic
 Ranking Scores
 Interaction-based
 Lowest-position
 Average-position
 Iteration
 Mathematical

Validating IR Functionality Adaptation (I)
40
 Definition of SilverStandard
Example with 4 entity features:
qmode=‘t’; qtype=‘short’; qlength=‘1’; qentities=‘Barcelona’  ft, ont, qa
Query: Barcelona

Validating IR Functionality Adaptation (II)
41
 The best combination is:
 Query features: mtle
 Classification algorithms: J4.8
 Ranking scores: Average Position Score

Monitoring health social media (I)
42
 Motivation
 Background
 Objectives
 Proposal
 Formal Model
 Publications

Monitoring health social media (I)
43
 Online: http://trendminer.daedalus.es/views/dashboard.php

Monitoring health social media (III)
44
 Annotation Pipeline
Documents
Index
Twitter Saluspot
Relations Manager
Disambiguation
Medical Events Filter
Topics Analyzer
Morpho-syntactic Parser
Language Identification
Resources
• DrugsGaz
• DrugsATC
• AdrsMedDRA
• DiseasesUMLS
• SpanishDrugEffectDB
AnotationPipeline

Monitoring health social media (IV)
45
 IMIR System

Monitoring health social media (V)
46
 Results’ Combination

Health-domain Prototype Evaluation
47
 No User Evaluation
 NER & Relation Extraction Performance
 NER
 Relations extraction
Drugs R P F-m
Strict 0,68 0,75 0,76
Lenient 0,68 0,75 0,76
Effects R P F-m
Strict 0,43 0,75 0,54
Lenient 0,47 0,83 0,6
SpanishDrugEffectDB Coocurrences
Wind. R P F-m R P F-m
30 Strict 0,08 0,57 0,14 0,63 0,44 0,52
30 Lenient 0,13 0,96 0,24 0,88 0,61 0,72

Conclusions (I)
48
 Formal model for IMIR systems
 Two prototypes based on the formal model in two
different scenarios:
 Sports domain
 Health social media
 Scenario 1: Adaptation of multimodal IR
 Best result: NDCG=81,54% (2,81% gain)
 Good RE performance  Small improvements

Future Lines (I)
49

Future Lines (II)
50
 Second Screen

Publications: Journals
51
 Bedmar, I. S., Martínez, P., Arenaz, R. R., and
Schneider, J. M. (2015). Exploring spanish health
social media for detecting drug effects. BMC Medical
Informatics and Decision Making, 15. 183, 216
 Martínez, P., Fernández, J. L. M., Bedmar, I. S.,
Schneider, J. M., Luna, A., and Arenaz, R. R. (2015).
Turning user generated health-related content into
actionable knowledge through text analytics
services. Computers in Industry.

Publications: Conferences
52
 SEPLN
 Julián Moreno-Schneider, José Luis Martínez Fernández,
Paloma Martínez, and Thierry Declerck. Prueba de Concepto de
Expansión de Consultas basada en Ontologías de Dominio
Financiero.
 AMR
 Julián Moreno-Schneider, José Luis Martínez Fernández, and
Paloma Martínez. A Proof-of-Concept for Orthographic Named
Entity Correction in Spanish Voice Queries.
 González, M., Moreno Schneider, J., Martínez, J. L., and
Martínez, P. (2013). An illustrated methodology for evaluating
asr systems.
 Schneider, J. M., Salazar, M. G., Martínez, P., and Fernández, J.
L. M. (2011). Some experiments in evaluating asr systems
applied to multimedia retrieval.

Publications: Conferences
53
 CLEF Conference
 Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez, P.
(2010a). Temporal information needs in respubliqa: an attempt
to improve accuracy. the uc3m participation at clef 2010.
 Vicente-Díez, M. T., De Pablo-Sanchez, C., Martínez, P.,
Moreno-Schneider, J., and Salazar, M. G. (2009). Are passages
enough? the miracle team participation in qaclef2009.
 SemEval
 Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez, P.
(2010b). Uc3m system: Determining the extent, type and value
of time expressions in tempeval-2.

Research and Development (R&D) projects
54
 Trendminer (FP7-ICT 287863)
 Buscamedia (CEN-20091026)
 Bravo (Búsqueda de Respuestas Avanzada
Multimodal y Multilingüe) (TIN2007-67407-C03-01)
 MAVIR (S-0505/TIC-0267) and MAVIR2 (S-
2009/TIC-1542)

55
 ‘‘New Approaches to Interactive Multimedia Content
Retrieval from different Sources’’
Julián Moreno Schneider
jmschnei@inf.uc3m.es
Thank you for your attention

New Approaches to Interactive Multimedia Content Retrieval from different Sources

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to New Approaches to Interactive Multimedia Content Retrieval from different Sources

Similar to New Approaches to Interactive Multimedia Content Retrieval from different Sources (20)

More from Grupo HULAT

More from Grupo HULAT (20)

Recently uploaded

Recently uploaded (20)

New Approaches to Interactive Multimedia Content Retrieval from different Sources

Editor's Notes