Más contenido relacionado Similar a Content without context is meaningless (12) 1. © Ramesh Jain
Ramesh Jain
and
Pinaki Sinha
Department of Computer Science
University of California, Irvine
jain@ics.uci.edu
Content Without Context
is Meaningless
2. © Ramesh Jain
Brave New Ideas
On a Nagging Old Problem – that won’t go
away.
About the dead elephant in the room.
Or about the Emperor’s new clothes.
“If you keep doing what you’ve always done,
you’ll keep getting what you’ve always got.”
This Presentation
3. © Ramesh Jain
Semantic Gap
Discussions
Semantic Gap
remains
strong.
Lets stop ignoring the Elephant in the room.
Our approaches have
completely failed for
the last two decades.
4. © Ramesh Jain
What is our Goal?
1. Using Intensity values to bridge the Semantic
Gap.
Or
2. Bridging the Semantic Gap in (multimedia)
data.
Or
3. Publishing another paper in ACM
Multimedia (or CVPR).
5. © Ramesh Jain
You will be disappointed in this
presentation if you are interested in
1. Using Intensity values to bridge the Semantic
Gap.
Or
3. Publishing another paper in ACM
Multimedia (or CVPR).
6. © Ramesh Jain
Reduction: Popular Research Approach 1
X is an important problem, let me work on it.
X is too complex. Simplify it to X’.
X’ is too complex. Simplify it to X2
’.
…
X99
’ can be formally defined, rigorously explored,
and clearly solved. Solve it.
Publish it in a conference with less than 16%
acceptance rate.
Announce that X is a solved problem.
7. © Ramesh Jain
Data Discovery: Popular Research Approach 2
Your algorithm fails on a real data set D.
Select a more relevant data set D1. Fails again.
Carefully select a subset of Flickr data. Still Fails.
…
Select a better dataset - Coral Photos, or create a
set and somehow make it Work.
Run several experiments – have several graphs so
reviewers don’t complain.
Publish it in a conf. with < 16% acceptance rate.
8. © Ramesh Jain
Back to Content and Context
Content
Meaning or message: The meaning or message
contained in data, as distinct from its appearance,
form, or style.
Context
Surrounding conditions: The circumstances or
events that form the environment within which
something exists or takes place.
9. © Ramesh Jain
Falling Tree and George Berkeley
"If a tree falls in a forest and no one is
around to hear it, does it make a sound”
"No. Sound is the sensation excited in the
ear when the air or other medium is set in
motion.“
Observation, Reality, and Perception.
11. © Ramesh Jain
Data Streams are Omnipresent
Visual DATA
Audio DATA
Text DATA
Location DATA
EXIF DATA
Time Line
DataType
15. © Ramesh Jain
Bits and Bytes
Alphanumeric Characters
Lists, Arrays, Documents, Images …
Transformations
16. © Ramesh Jain
Semantic Gap
The semantic gap is the lack of coincidence
between the information that one can extract
from the (visual) data and the interpretation
that the same data have for a user in a given
situation. A linguistic description is almost
always contextual, whereas an (image) may
live by itself.
Content-Based Image Retrieval at the End of the Early Years
Found in: IEEE Transactions on Pattern Analysis and Machine Intelligence
Arnold Smeulders , et. al., December 2000
17. © Ramesh Jain
M ultim edia
R esearcher
Hammer
"To a man with a
hammer, everything
looks like a nail." —
Mark Twain
Where is the
Semantic Gap?
I got my Machine
Learning Hammer.
18. © Ramesh Jain
Learning from Birds
The birds flying at the tips and at the front
are rotated in a timely cyclical fashion to
spread flight fatigue equally among the flock
members.
The formation makes communication easier
and allows the birds to maintain visual
contact with each other.
19. © Ramesh Jain
Data is just that --- DATA, but
Some of us have introduced a caste system in
multimedia!!!
Video
Meta Data
Audio
Text
Multimedia Caste SystemTraditional Caste System
Priest
Warrior
Traders
Who Cares
21. © Ramesh Jain
Events and Objects
Exist in the real world.
Captured using different sensory
mechanism.
Each sensor captures only a limited aspect.
Are used to understand a Situation.
23. © Ramesh Jain
Events are ‘Connectors’
Events create ‘Context’
SpatialCausal
Experiential Informational
TemporalStructural
People
Things
Places
Time
Experiences
Events
Events Connect:
Events are represented
using 6 facets.
25. © Ramesh Jain
Dealing with Photo Explosion
3 Billion Photos uploaded on Facebook –
every month.
We know that most people in the world are
not yet addicted to the Web, but do take
photos.
26. © Ramesh Jain
Context Starts Before the Photo is
Taken
Where
When
Why
Who (Photographer)
Which device
Parameters of the device
27. © Ramesh Jain
Types of Context
Context in Content
Device Parameters
Data Acquisition Context
Perceiver
Interpretation Context
28. © Ramesh Jain
Context in Content
Relationship among different objects and
even in their subparts in real world can be
utilized in analysis of data.
This has been studied since early days in
computer vision and is being rediscovered
once again.
29. © Ramesh Jain
Device Parameters
Environmental parameters of the digital
devices at the time of photo taking may play
key role.
What is a camera?
30. © Ramesh Jain
Data Acquisition Context
Knowledge about the person taking photos,
location, and environmental conditions at the
time of photo acquisition (e.g., sun angle,
cloudy, rainy, night, indoor, etc.) affect the
content of the image.
32. © Ramesh Jain
Perceiver
The knowledge and personality of the
perceiver play a key role in interpretation of
data.
Knowledge
Personality:
Rohrsach
Tests
33. © Ramesh Jain
Interpretation Context
Real world situation in which the data is
interpreted results in focus on different
aspects of the data.
34. © Ramesh Jain
What is a Camera?
Kodak ‘Moment’.
EXIF data is all metadata related to the Event.
Exposure Time
Aperture Diameter
Flash
Metering Mode
ISO Ratings
Focal Length
Time
Location
Face
35. © Ramesh Jain
Sony CyberShot DSC-T2 Touchscreen 8MP Digital
Camera with Smile Detection
Camera = Event Capture Device
37. © Ramesh Jain
A new problem statement for media analysis
Given apriori knowledge about the world (concepts and
relations between concepts)
Given a priori knowledge about events (lattice of events)
Given as contextual information
A set of data/ media
A possibly empty sets of tags (natural language
descriptions)
The current event type to be instantiated (possibly to be first
discovered)
Identify the entities occurring in the event, as described by
media, data and text
with >98% precision and recall
with minimal/ acceptable user involvement
Fausto Giunchiglia, DISI, University of Trento
38. © Ramesh Jain
Annotation Sources
Calendars
Annotations
Environment
Conditions
Application
Knowledge
Geographic
Landmarks
Camera
Parameters
39. © Ramesh Jain
Annotation Process
Calendars
Annotations
Environment
Conditions
Application
Knowledge
Geographic
Landmarks
Camera
Parameters
Keynote Talk at ACM MM 2010,
Speaker: Duncan WattsLocation: Palazzo dei Congressi
EXIF: Indoor Scene; Face
Keynote Talk
at ACMMM10
by Duncan Watts
on
Using the Web to
do Social Science
40. © Ramesh Jain
Semantic Gap is a Tough Problem –
Lets Fight with it using all tools.
Fighting
with
a
Monster.
Notas del editor DATA MODEL:
To capture events we need a common event model in order to avoid obfuscating event exploration and event-driven access to media.We used E-Model that introduces 6 facets for events (the circle diagram in the next slide) as the initial basis for our data model. We are currently building the formalizations of our data model which is based on this circle diagram. We are formally defining each facet in our data model. Each facet may involve structures and/or induced sub-graphs in addition to the RDF structure. However this is our future work and we will not talk about that here.
