Provenance for Multimedia: COMM, D&S, M3O. Talk given for the W3C Provenance Incubator Group, May 28th 2010.

1. Provenance for Multimedia:
COMM, D&S, M3O
Raphaël Troncy <raphael.troncy@eurecom.fr>
Multimedia Semantics, EURECOM (FR)

2. Common Scenario: Tagging Approach
Reg1
The "Big Three" at the Yalta
Conference (Wikipedia)
 Localize a region
 Draw a bounding box, a circle around a shape
 Annotate the content
 Interpret the content
 Tag: Winston Churchill, UK Prime Minister, Allied Forces, WWII
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -2

3. Common Scenario: SW Approach
Reg1
The "Big Three" at the Yalta
Conference (Wikipedia)
 Localize a region
 Draw a bounding box, a circle around a shape
 Annotate the content
 Interpret the content
 Link to knowledge on the Web
:Reg1 foaf:depicts dbpedia:Winston_Churchill
dbpedia:Winston_Churchill skos:altLabel
"Sir Winston Leonard Spencer-Churchill"
dbpedia:Winston_Churchill rdf:type foaf:Person
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -3

4. Research Problem
Seq4
Reg1
Seq1
The "Big Three" at the Yalta
Conference (Wikipedia) A history of G8 violence (video)
(© Reuters)
 Multimedia objects are complex
 MPEG-7
 Compound information objects, fragment identification
 Semantic annotation
 Subjective interpretation, context dependent  D&S | OIO
 Linked data principle
 Open to reuse existing knowledge  RDF
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -4

5. 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -5

6. COMM: Design Rationale
 Approach:
 NO 1-to-1 translation from MPEG-7 to OWL/RDF
 Need for patterns: use DOLCE, a well designed foundational
ontology as a modeling basis
 Design patterns:
 Ontology of Information Objects (OIO)
Formalization of information exchange
Multimedia = complex compound information objects
 Descriptions and Situations (D&S)
Formalization of context
Multimedia = contextual interpretation (situation)
 Define multimedia patterns that translate MPEG-7 in the
DOLCE vocabulary
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -6

7. COMM: Core Functionalities
 Most important MPEG-7 functionalities:
 Decomposition of multimedia content into segments
 Annotation of segments with metadata
Administrative metadata: creation & production
Content-based metadata: audio/visual descriptors
Semantic metadata: interface with domain specific ontologies
 Note that all are subjective and context
dependent situations
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -7

8. COMM: D&S / OIO Patterns
3
2
1
Definition of design patterns for decomposition and
annotation based on D&S and OIO
1. MPEG-7 describes digital data (multimedia information objects)
with digital data (annotation)
2. Digital data entities are information objects
3. Decompositions and annotations are situations that satisfy the
rules of a method or algorithm
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -8

9. COMM: Decomposition Pattern
MPEG-
MPEG-7
7
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -9

10. COMM: Semantic Pattern
Domain
Ontologies
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 10

11. COMM Example: Region Annotation
http://en.wikipedia.org/wiki/
Image:Yalta_Conference.jpg
dns:realized-by
dns:setting
core:semantic-
core:image-data
annotation
dns:plays dns:defines foaf:Person
loc:region- loc:spatial-mask- core:semantic-label-
locator-descriptor role role
dns:played-by
rdf:type
dns:defines dns:played-by
http://en.wikipedia.org/wiki/
loc:bounding-box 5 25 10 20 15 15 10 10 5 15"^^xsd:string
Churchill
data:has-rectangle
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 11

12. Provenance of Annotation: Naïve Approach
dbpedia:Winston_Churchill
dc:creator
foaf:depicts
:me
:Reg1
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 12

13. Provenance of Annotation: COMM / M3O
ex:annotation-pattern
defines
ex:annotated-concept ex:annotation-concept
classifies classifies
satisfies
ex:big-3-yalta dbpedia:Winston_Churchillin
hasSetting
ex:annotation-situation
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 13

14. Provenance of Annotation: COMM / M3O
ex:annotation-pattern
defines
ex:annotated-concept ex:annotation-concept ex:applied-method-role dc:creator
classifies classifies
satisfies
classifies classifies
ex:big-3-yalta dbpedia:Winston_Churchillin ex:manual-annotation
:me
hasSetting
ex:annotation-situation
http://m3o.semantic-multimedia.org/ontology/2010/02/28/provenance.owl
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 14

15.  Low-level features
extraction
 Classifier: SVM
 Input parameters
 Threshold
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 15

16. Summary: Provenance in multimedia semantics
 Direct approach:
 Reification: what is the semantics of what is being reified?
 Named Graph: how to represent the parameters used for detecting a
face in an image?
 Provenance vocabularies:
 OPM in RDF [Moreau et al.], http://openprovenance.org/model/opm.owl
 Provenir Vocab [Sahoo et al.],
http://knoesis.wright.edu/provenir/provenir.owl
 Provenance Vocab [Hartig and Zhao], http://purl.org/net/provenance/
 prv:createdBy, prv:performedAt, prv:performedBy, etc.
 D&S pattern:
 Formalized description of a particular context (situation)
 Specialized for multimedia in COMM and M3O
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 16