SlideShare una empresa de Scribd logo
1 de 73
Descargar para leer sin conexión
Image Search: Then and Now
Integrated Knowledge Solutions
iksinc@yahoo.com
sikrishan@gmail.com
iksinc.wordpress.com
Outline
• Introduction
• Image = Content + Context
• Content Based Image Retrieval (CBIR)
• Bridging the Semantic Gap
• Using Social Interactions for Retrieval
• Where do we go from here
What is Image Search?
• Image search means retrieving images from an
image database that satisfy the user’s need.
• The user need may be expressed in the following
ways:
– Keywords or text describing the image content
– An exemplar image
• Other names for image search
– Image retrieval
– Image similarity search
– Content based image retrieval (CBIR)
Document Search Not a New Problem
Nalanda University was one of the first universities
in the world, founded in the 5th Century BC, and
reported to have been visited by the Buddha during
his lifetime. At its peak, in the 7th century AD,
Nalanda held some 10,000 students when it was
visited by the Chinese scholar Xuanzang.
The Royal Library of Alexandria, in Egypt, seems to have been
the largest and most significant great library of the ancient
world. It functioned as a major center of scholarship from its
construction in the third century B.C. until the Roman
conquest of Egypt in 48 B.C.
However, Earlier
Few Document Producers
Many Document Consumers
But Now a Days
No Distinction Between Document Producers
and Consumers
Image Search: Then and Now
Some Relevant Numbers
Flickr has over 6 billion pictures as of August 2011,
and 3.5 million images are uploaded daily.
Photobucket has more than 10 Billion images, and
over 4 million images are uploaded everyday.
Facebook has over 60 Billion photos and more than
350 million photos are uploaded everyday.
Instagram has over 20 billion photos. About 60
million photos are uploaded everyday.
An image now a days is not just a
picture but it is a picture with
thousand words
Image = Content + Context
Tags
Cherry
blossom
Japantown
San
Francisco
Peace
Pagoda
Content Context
So, image retrieval should benefit
from the contextual component, if
present.
How?
But, first let us look at image
retrieval from the content
perspective only
QBIC/signal
similarity
Concept
/semantic
similarity
Concept
plus
context
History of Image Retrieval
1993
2002
1999
A Typical QBIC Type Image Retrieval
System
Feature
Extraction
FeaturesMedia
Collection
Indexing &
Matching
Query Feature
Extraction
Retrieved
Results
Relevance
Feedback
Such systems/approaches are often referred to as Content
Based Image Retrieval (CBIR)
Image Search: Then and Now
Image Search: Then and Now
Semantic Gap
Early systems produced results
wherein the retrieved
documents were visually similar
(signal level similar) but not
necessarily similar in showing
the same semantic concept.
Content-Based Image Retrieval at the End of the Early Years,
IEEE Transactions on Pattern Analysis and Machine Intelligence , Arnold
Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta ,
Ramesh Jain , December 2000
http://www.searchenginejournal.com/7-similarity-based-image-search-engines/8265/
Semantic Gap
Users also like to query using descriptive
words rather than query images or other
multimedia objects. This requires retrieval
systems to correlate low-level features
with high level concepts.
Visually dissimilar
images representing
the same concept.
Semantic Gap Challenge
How to Bridge the Semantic Gap?
Manual annotation
Use machine learning to:
• Build image category classifiers to
perform semantic filtering of the
results
• Build specific detectors for objects to
associate concepts with images
•Build object models using low level
features
Exploit context:
• Text surrounding images
• Associated sound track and
closed captions in videos
• Query history
Crowdsourcing for Manual Annotation
Image Search: Then and Now
Example of Image Search using Keywords
Search result in 2010
Example of Image Search using Keywords
Search result in 2014
The results are better organized in sub-categories
Example of Image Search using Keywords
Example of Image Search using Keywords
Search result in 2014
Again, the results are better organized in sub-
categories
Exploiting Context: An Example
Kulesh, Petrushin and Sethi, “The PERSEUS Project: Creating Personalized Multimedia News Portal,”
Proceedings Second Int’l Workshop on Multimedia Data Mining, 2001
Machine Learning of Image Concepts
• Challenging problem
• Presence of multiple concepts/multiple instances
• Disproportionate number of negative examples
• Manpower need for labeling training examples
Feature Extraction Issues
Whole image based features.
Easy to use but not very
effective
Region based features. Both
regular region structure and
segmented regions are popular
Salient objects based features.
Connected regions
corresponding to dominant
visual properties of objects in an
image
Scale Invariant Feature Transform
(SIFT) Descriptors
SIFT descriptors or its variants are
currently the most popular features
in use. Each image generates
thousands of features (key point
descriptors) with each feature
typically consisting of 128 values
http://www.vlfeat.org/
D. G. Lowe, “Distinctive image
features from scale-invariant
keypoints,” IJCV, 2004.
Learning Image Concepts
• Both supervised and unsupervised
learning methods (SVM, DT, AdaBoost,
VQ etc.) have been used
• Early work limited to few tens of
categories; however some of the current
systems can work with thousands of
categories/concepts
VQ Based Learning Classifier
Test
Image
Best
Codebook
Label
Water Codebook
Sky Codebook
Fire Codebook
Mustafa & Sethi (2004)
http://vision.stanford.edu/teaching/cs223b/lecture/lecture14_intro_objr
ecog_bow_cs223b.pdf
Bag of Words Approach
Bag of Words Representation of
Images
Co-occurrence of Bag of Words
Image
Collection
Edge
Analysis
Images
Collection of
Binary Image
Blocks
Clustering
Local
Feature
Descriptors
(Codewords)
Codeword
Representation
Of Images
Co-occurrence
Matrices of
Local Features
Compute
Distances
Image
Distance
Matrix
Pathfinder
Network
Mukhopadhyay, Ma, and Sethi, “Pathfinder Networks for Content Based Image
Retrieval Based on Automated Shape Feature Discovery,” ISMSE 2004
Co-occurrence of BoW
Original image
Representation by
feature indices
(cluster membership)
Co-occurrence matrix
)},(),,(max{),( ABhBAhBAH 
))max(min(),( AaBbbaBAh 
Hausdorff metric
Manhattan distance
Notice how similar
images are placed
together in the graph
Image Search: Then and Now
Image Search: Then and Now
Object Detectors for Image Concepts
PASCAL Visual Object Classes Challenge
Project
http://labelme.csail.mit.edu/
Web-based annotation tool to segment and label image
regions. Labeled objects in images are used as training images
to build object detectors.
IMARS provides a large number of built-in classifiers for visual categories that cover places, people, objects, settings,
activities and events. It is easy to add new ones. IMARS can work on PC or laptop (trial version is available at IBM
alphaWorks). IMARS can also work at large-scale for high-volume batch processing of millions and images and videos
per day. Several demos of IMARS are available (see IMARS demos)
Image Category Classifiers Examples
Semantic labeling. (a) An MPE semantic retrieval system groups images by semantic
concept and learns a probabilistic model for each concept. (b) The system represents
each image by a vector of posterior concept probabilities.
From Pixels to Semantic Spaces: Advances in Content-Based Image
Retrieval (Nuno Vasconcelos, IEEE Computer, July 2007)
Image Classification via Probabilistic
Modeling
Image = Content + Context
Tags
Cherry
blossom
Japantown
San
Francisco
Peace
Pagoda
Content Context
Tagging
All time most popular tags at Flickr
About Tags
• User centered
• Imprecise and often overly personalized
• Tag distribution follows power law
• Most users use very few distinct tags while a small group of users works
with extremely large set of tags
• Also known as Folksonomy, social tagging, and social classification
Why Not Use Social Tags for Retrieval?
Problem: The relevant tag is
often not at the top of the list.
Only less than 10% of the
images have their most relevant
tag at the top of the list.
Solution: Improve tagging by
suggesting potential tags to a
user / tag ranking /tag
completion etc.
Dong Liu, Xian-Sheng Hua, Linjun Yang, Meng Wang, Hong-
Jiang Zhang. Tag Ranking. WWW 2009. Madrid, Spain
Tag Recommendation using Tags
Co-occurrences
Given a target image and initial tags, use co-occurrence of tags to
recommend tags for the target image. This approach doesn’t take into
account the visual features co-occurrences.
Tag Recommendation using Tags
Co-occurrences and Visual Similarity
Kucuktunc, Sevil, Tosun, Zitouni, Duygulu, and Can (SAMT 08)
Given a target image and initial tags, use the existing tagged images to
suggest tags for the target image.
Tag Ranking
Tag Ranking: Another Approach
Dong Liu, Xian-Sheng Hua, Linjun Yang, Meng Wang, Hong-
Jiang Zhang. Tag Ranking. WWW 2009. Madrid, Spain
How to Compute Tag Similarity
Image Search: Then and Now
Tag Recommendation After Tag
Ranking
• Given an untagged image, find its visually similar “k” images
• Pool the top two ranked tags from k images and select the unique tags as
recommended tags
Tag Completion
The complete tag matrix is
generated by imposing
constraints based on visual
similarity, tag to tag similarity,
and similarity with the initial
tag matrix. The matrix
completion is done by an
optimization procedure.
Wu and Jain, IEEE-PAMI, JANUARY
2011
What about Taggers & Commenters?
Question: How can we incorporate taggers/commenters
characteristics for improved tag recommendations?
Answer: Use three sets of features: derived from image to
be tagged, user’s tag history, and user’s social interactions
Tag History & Social Interaction
Features
Tag history features are based
on the tags the user has used
in the past
Social interaction features are
derived from tags/comments
posted by the user’s
friends/favorite posters
X. Chen & H. Shin, ICDM 2010
Current Status of Image Search
• Extensive interest as evident from conferences, journals, and
special issues
• Overall, solid progress is being made
• Efforts towards performance evaluation with benchmarked
collections are gaining more traction
• Integration of content and context through tags and
comments is receiving increasing attention to help improve
retrieval
• Killer applications are beginning to emerge as visual search
gains prominence
• Need for more applications outside entertainment
Performance Evaluation Efforts
ImageCLEF2013
- Annotation Task:
- 250000 Training Images
- 95 (develop), 116 (test) concepts to be identified
- A lot of label Noise inside the training set, due to the automatic label
extraction from websites
Performance Evaluation Efforts
TRECVID workshops, an offshoot of TREC, are yearly evaluation meetings since
2003. The goal of the workshops is to encourage research in content-based
video retrieval and analysis by providing large test collections, realistic system
tasks, uniform scoring procedures, and a forum for organizations interested in
comparing their results.
Application Examples
Tattoo-ID: Automatic Tattoo Image Retrieval for Suspect & Victim Identification (Anil K. Jain,
Jung-Eun Lee, and Rong Jin)
CBIR for Whole Slide Imageries
• The availability of digital whole slide data sets
represent an enormous opportunity to carry out
new forms of numerical and data- driven query,
in modes not based on textual, ontological or
lexical matching.
– Search image repositories with whole images or
image regions of interest
– Carry our search in real-time via use of scalable
computational architectures
Extraction from Image
repositories based upon
spatial information
Analysis of data
in the digital domain
…001011010111010111..
Resultant Surface Map or
gallery of matching images
or
Slide courtesy of Ulysses J. Balis, M.D.
Director, Division of Pathology Informatics
Department of Pathology
University of Michigan Health System
Medical Image Retrieval
Text
“Find all the cases in which a tumor decrease in size
for less than three month post treatment, then
resumed a growth pattern after that period”
QUERY ?
Text + medical image
“Find images with large-sized frontal lobes brain tumors for
patients approximately 35 years old”
+Medical image
QUERY IMAGE-BASED CONCEPTS
Medical image ij - Specific Signature
ImageiQuery
VB-Spec CUIp
VB-Gen CUI1
VB-Spec CUIkIMAGE-BASED
ONTOLOGY
GENRAL AND SPECIALIZED
QUERY MEDICAL IMAGE
VISUAL ANALYSIS
Text query
CUIn
CUI1
CUI2
QUERY TEXT-BASED CONCEPTS
Textual query i - Indexes
MEDICAL
ONTOLOGY
TEXT QUERY
CONCEPTS
EXTRACTION
Image Search Products
http://www.picalike.com/products/similarity-search.php
Image Search Products
http://www.pcsso.com/
Image Search Products
Image Search Products
http://viral.image.ntua.gr/
Image Search: Then and Now
Take Home Message
• Image/video retrieval is moving in the
commercial domain. Lot more activity is expected
in near future
• Multimodal/cross-modal retrieval is gaining
importance
• Approaches combining social search and visual
search techniques are expected to gain
prominence
• Crowdsourcing is a cheap and effective way of
tagging media
Acknowledgement
• This presentation is based on the work of
numerous researchers from the MIR/ML/CVPR
community. I have tried to give
credit/references wherever possible. Any
omission is unintentional and I apologize for
that.
• Also want to thank my present and past
students and collaborators.
Questions?
Email: iksinc@yahoo.com
Email: sikrishan@gmail.com

Más contenido relacionado

La actualidad más candente

ImageProcessing10-Segmentation(Thresholding) (1).ppt
ImageProcessing10-Segmentation(Thresholding) (1).pptImageProcessing10-Segmentation(Thresholding) (1).ppt
ImageProcessing10-Segmentation(Thresholding) (1).pptVikramBarapatre2
 
3D Graphics & Rendering in Computer Graphics
3D Graphics & Rendering in Computer Graphics3D Graphics & Rendering in Computer Graphics
3D Graphics & Rendering in Computer GraphicsFaraz Akhtar
 
Multimedia color in image and video
Multimedia color in image and videoMultimedia color in image and video
Multimedia color in image and videoMazin Alwaaly
 
Movie Recommender system
Movie Recommender systemMovie Recommender system
Movie Recommender systemPalakNath
 
Content based image retrieval(cbir)
Content based image retrieval(cbir)Content based image retrieval(cbir)
Content based image retrieval(cbir)paddu123
 
Texture mapping in_opengl
Texture mapping in_openglTexture mapping in_opengl
Texture mapping in_openglManas Nayak
 
Object detection presentation
Object detection presentationObject detection presentation
Object detection presentationAshwinBicholiya
 
Computational photography by Sanket Mane
Computational photography by Sanket ManeComputational photography by Sanket Mane
Computational photography by Sanket Manesanketmane29
 
Noise filtering
Noise filteringNoise filtering
Noise filteringAlaa Ahmed
 
Object Detection & Tracking
Object Detection & TrackingObject Detection & Tracking
Object Detection & TrackingAkshay Gujarathi
 
Object detection with deep learning
Object detection with deep learningObject detection with deep learning
Object detection with deep learningSushant Shrivastava
 
Introduction to Image Compression
Introduction to Image CompressionIntroduction to Image Compression
Introduction to Image CompressionKalyan Acharjya
 
A Study of Image Tampering Detection
A Study of Image Tampering DetectionA Study of Image Tampering Detection
A Study of Image Tampering DetectionIRJET Journal
 
Advance image processing
Advance image processingAdvance image processing
Advance image processingAAKANKSHA JAIN
 
Image analysis using python
Image analysis using pythonImage analysis using python
Image analysis using pythonJerlyn Manohar
 

La actualidad más candente (20)

Noise Models
Noise ModelsNoise Models
Noise Models
 
Image Sensing & Acquisition
Image Sensing & AcquisitionImage Sensing & Acquisition
Image Sensing & Acquisition
 
ImageProcessing10-Segmentation(Thresholding) (1).ppt
ImageProcessing10-Segmentation(Thresholding) (1).pptImageProcessing10-Segmentation(Thresholding) (1).ppt
ImageProcessing10-Segmentation(Thresholding) (1).ppt
 
Image Stitching for Panorama View
Image Stitching for Panorama ViewImage Stitching for Panorama View
Image Stitching for Panorama View
 
3D Graphics & Rendering in Computer Graphics
3D Graphics & Rendering in Computer Graphics3D Graphics & Rendering in Computer Graphics
3D Graphics & Rendering in Computer Graphics
 
Dip chapter 2
Dip chapter 2Dip chapter 2
Dip chapter 2
 
Multimedia color in image and video
Multimedia color in image and videoMultimedia color in image and video
Multimedia color in image and video
 
Movie Recommender system
Movie Recommender systemMovie Recommender system
Movie Recommender system
 
Content based image retrieval(cbir)
Content based image retrieval(cbir)Content based image retrieval(cbir)
Content based image retrieval(cbir)
 
Image formation
Image formationImage formation
Image formation
 
Texture mapping in_opengl
Texture mapping in_openglTexture mapping in_opengl
Texture mapping in_opengl
 
Object detection presentation
Object detection presentationObject detection presentation
Object detection presentation
 
Computational photography by Sanket Mane
Computational photography by Sanket ManeComputational photography by Sanket Mane
Computational photography by Sanket Mane
 
Noise filtering
Noise filteringNoise filtering
Noise filtering
 
Object Detection & Tracking
Object Detection & TrackingObject Detection & Tracking
Object Detection & Tracking
 
Object detection with deep learning
Object detection with deep learningObject detection with deep learning
Object detection with deep learning
 
Introduction to Image Compression
Introduction to Image CompressionIntroduction to Image Compression
Introduction to Image Compression
 
A Study of Image Tampering Detection
A Study of Image Tampering DetectionA Study of Image Tampering Detection
A Study of Image Tampering Detection
 
Advance image processing
Advance image processingAdvance image processing
Advance image processing
 
Image analysis using python
Image analysis using pythonImage analysis using python
Image analysis using python
 

Destacado

How to Do Research: Seven Steps to Successful Research
How to Do Research: Seven Steps to Successful ResearchHow to Do Research: Seven Steps to Successful Research
How to Do Research: Seven Steps to Successful ResearchSi Krishan
 
Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...XavierArrufat
 
Intro to Excel Basics: Part II
Intro to Excel Basics: Part IIIntro to Excel Basics: Part II
Intro to Excel Basics: Part IISi Krishan
 
Introduction to the Continua at GWA
Introduction to the Continua at GWA Introduction to the Continua at GWA
Introduction to the Continua at GWA David Gerber
 
Firefox OS - HTML5 for a truly world-wide-web
Firefox OS - HTML5 for a truly world-wide-webFirefox OS - HTML5 for a truly world-wide-web
Firefox OS - HTML5 for a truly world-wide-webChristian Heilmann
 
RTC Ski Equipment Branding Images
RTC Ski Equipment Branding ImagesRTC Ski Equipment Branding Images
RTC Ski Equipment Branding Imageslaura_powell
 
Tesla ERP - Introduction
Tesla ERP - IntroductionTesla ERP - Introduction
Tesla ERP - IntroductionTesla ERP
 
Miii.1. aprend. herramientas para mi trabajo ok
Miii.1. aprend. herramientas para mi trabajo okMiii.1. aprend. herramientas para mi trabajo ok
Miii.1. aprend. herramientas para mi trabajo okpromocomunitaria
 
Externalización Servicios Contact Center, Responsabilidad Compartida
Externalización Servicios Contact Center, Responsabilidad CompartidaExternalización Servicios Contact Center, Responsabilidad Compartida
Externalización Servicios Contact Center, Responsabilidad CompartidaRodrigo Navarro
 
Ante natal clinic - protocol
Ante natal  clinic - protocolAnte natal  clinic - protocol
Ante natal clinic - protocoldrmcbansal
 
Social media in banking
Social media in bankingSocial media in banking
Social media in bankingSi Krishan
 

Destacado (20)

How to Do Research: Seven Steps to Successful Research
How to Do Research: Seven Steps to Successful ResearchHow to Do Research: Seven Steps to Successful Research
How to Do Research: Seven Steps to Successful Research
 
Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...
 
Intro to Excel Basics: Part II
Intro to Excel Basics: Part IIIntro to Excel Basics: Part II
Intro to Excel Basics: Part II
 
Introduction to the Continua at GWA
Introduction to the Continua at GWA Introduction to the Continua at GWA
Introduction to the Continua at GWA
 
Firefox OS - HTML5 for a truly world-wide-web
Firefox OS - HTML5 for a truly world-wide-webFirefox OS - HTML5 for a truly world-wide-web
Firefox OS - HTML5 for a truly world-wide-web
 
Cierto indice magico 15
Cierto indice magico 15Cierto indice magico 15
Cierto indice magico 15
 
Un concepto mal empleado. Dpto. de RR. HH
Un concepto mal empleado. Dpto. de RR. HHUn concepto mal empleado. Dpto. de RR. HH
Un concepto mal empleado. Dpto. de RR. HH
 
Inserciones 2010
Inserciones 2010Inserciones 2010
Inserciones 2010
 
Shipbuilding-web
Shipbuilding-webShipbuilding-web
Shipbuilding-web
 
RTC Ski Equipment Branding Images
RTC Ski Equipment Branding ImagesRTC Ski Equipment Branding Images
RTC Ski Equipment Branding Images
 
Vegetable 03
Vegetable 03Vegetable 03
Vegetable 03
 
Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale
 
Tesla ERP - Introduction
Tesla ERP - IntroductionTesla ERP - Introduction
Tesla ERP - Introduction
 
Coutadas Trail Magazine
Coutadas Trail MagazineCoutadas Trail Magazine
Coutadas Trail Magazine
 
Miii.1. aprend. herramientas para mi trabajo ok
Miii.1. aprend. herramientas para mi trabajo okMiii.1. aprend. herramientas para mi trabajo ok
Miii.1. aprend. herramientas para mi trabajo ok
 
Externalización Servicios Contact Center, Responsabilidad Compartida
Externalización Servicios Contact Center, Responsabilidad CompartidaExternalización Servicios Contact Center, Responsabilidad Compartida
Externalización Servicios Contact Center, Responsabilidad Compartida
 
Informe mensual
Informe mensualInforme mensual
Informe mensual
 
Ante natal clinic - protocol
Ante natal  clinic - protocolAnte natal  clinic - protocol
Ante natal clinic - protocol
 
Ultrasonido
UltrasonidoUltrasonido
Ultrasonido
 
Social media in banking
Social media in bankingSocial media in banking
Social media in banking
 

Similar a Image Search: Then and Now

CONTENT BASED IMAGE RETRIEVAL SYSTEM
CONTENT BASED IMAGE RETRIEVAL SYSTEMCONTENT BASED IMAGE RETRIEVAL SYSTEM
CONTENT BASED IMAGE RETRIEVAL SYSTEMVamsi IV
 
Spot the Dog: An overview of semantic retrieval of unannotated images in the ...
Spot the Dog: An overview of semantic retrieval of unannotated images in the ...Spot the Dog: An overview of semantic retrieval of unannotated images in the ...
Spot the Dog: An overview of semantic retrieval of unannotated images in the ...Jonathon Hare
 
A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...
A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...
A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...Journal For Research
 
Novel Hybrid Approach to Visual Concept Detection Using Image Annotation
Novel Hybrid Approach to Visual Concept Detection Using Image AnnotationNovel Hybrid Approach to Visual Concept Detection Using Image Annotation
Novel Hybrid Approach to Visual Concept Detection Using Image AnnotationCSCJournals
 
Research Inventy: International Journal of Engineering and Science
Research Inventy: International Journal of Engineering and ScienceResearch Inventy: International Journal of Engineering and Science
Research Inventy: International Journal of Engineering and Scienceresearchinventy
 
Research Inventy : International Journal of Engineering and Science is publis...
Research Inventy : International Journal of Engineering and Science is publis...Research Inventy : International Journal of Engineering and Science is publis...
Research Inventy : International Journal of Engineering and Science is publis...researchinventy
 
A Comparative Study of Content Based Image Retrieval Trends and Approaches
A Comparative Study of Content Based Image Retrieval Trends and ApproachesA Comparative Study of Content Based Image Retrieval Trends and Approaches
A Comparative Study of Content Based Image Retrieval Trends and ApproachesCSCJournals
 
A Survey on Content Based Image Retrieval System
A Survey on Content Based Image Retrieval SystemA Survey on Content Based Image Retrieval System
A Survey on Content Based Image Retrieval SystemYogeshIJTSRD
 
Comparison of Various Web Image Re - Ranking Techniques
Comparison of Various Web Image Re - Ranking TechniquesComparison of Various Web Image Re - Ranking Techniques
Comparison of Various Web Image Re - Ranking TechniquesIJSRD
 
Paper id 25201471
Paper id 25201471Paper id 25201471
Paper id 25201471IJRAT
 
Image web crawler
Image web crawlerImage web crawler
Image web crawlerdixitas
 
Evolving a Medical Image Similarity Search
Evolving a Medical Image Similarity SearchEvolving a Medical Image Similarity Search
Evolving a Medical Image Similarity SearchSujit Pal
 
An Enhance Image Retrieval of User Interest Using Query Specific Approach and...
An Enhance Image Retrieval of User Interest Using Query Specific Approach and...An Enhance Image Retrieval of User Interest Using Query Specific Approach and...
An Enhance Image Retrieval of User Interest Using Query Specific Approach and...IJSRD
 
A novel Image Retrieval System using an effective region based shape represen...
A novel Image Retrieval System using an effective region based shape represen...A novel Image Retrieval System using an effective region based shape represen...
A novel Image Retrieval System using an effective region based shape represen...CSCJournals
 
Searching Images: Recent research at Southampton
Searching Images: Recent research at SouthamptonSearching Images: Recent research at Southampton
Searching Images: Recent research at SouthamptonJonathon Hare
 
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...Gene Moo Lee
 

Similar a Image Search: Then and Now (20)

CONTENT BASED IMAGE RETRIEVAL SYSTEM
CONTENT BASED IMAGE RETRIEVAL SYSTEMCONTENT BASED IMAGE RETRIEVAL SYSTEM
CONTENT BASED IMAGE RETRIEVAL SYSTEM
 
Ts2 c topic (1)
Ts2 c topic (1)Ts2 c topic (1)
Ts2 c topic (1)
 
Ts2 c topic
Ts2 c topicTs2 c topic
Ts2 c topic
 
Spot the Dog: An overview of semantic retrieval of unannotated images in the ...
Spot the Dog: An overview of semantic retrieval of unannotated images in the ...Spot the Dog: An overview of semantic retrieval of unannotated images in the ...
Spot the Dog: An overview of semantic retrieval of unannotated images in the ...
 
A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...
A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...
A NOVEL WEB IMAGE RE-RANKING APPROACH BASED ON QUERY SPECIFIC SEMANTIC SIGNAT...
 
Yang.ppt
Yang.pptYang.ppt
Yang.ppt
 
Novel Hybrid Approach to Visual Concept Detection Using Image Annotation
Novel Hybrid Approach to Visual Concept Detection Using Image AnnotationNovel Hybrid Approach to Visual Concept Detection Using Image Annotation
Novel Hybrid Approach to Visual Concept Detection Using Image Annotation
 
Research Inventy: International Journal of Engineering and Science
Research Inventy: International Journal of Engineering and ScienceResearch Inventy: International Journal of Engineering and Science
Research Inventy: International Journal of Engineering and Science
 
Research Inventy : International Journal of Engineering and Science is publis...
Research Inventy : International Journal of Engineering and Science is publis...Research Inventy : International Journal of Engineering and Science is publis...
Research Inventy : International Journal of Engineering and Science is publis...
 
A Comparative Study of Content Based Image Retrieval Trends and Approaches
A Comparative Study of Content Based Image Retrieval Trends and ApproachesA Comparative Study of Content Based Image Retrieval Trends and Approaches
A Comparative Study of Content Based Image Retrieval Trends and Approaches
 
A Survey on Content Based Image Retrieval System
A Survey on Content Based Image Retrieval SystemA Survey on Content Based Image Retrieval System
A Survey on Content Based Image Retrieval System
 
Comparison of Various Web Image Re - Ranking Techniques
Comparison of Various Web Image Re - Ranking TechniquesComparison of Various Web Image Re - Ranking Techniques
Comparison of Various Web Image Re - Ranking Techniques
 
Paper id 25201471
Paper id 25201471Paper id 25201471
Paper id 25201471
 
Image web crawler
Image web crawlerImage web crawler
Image web crawler
 
research paper
research paperresearch paper
research paper
 
Evolving a Medical Image Similarity Search
Evolving a Medical Image Similarity SearchEvolving a Medical Image Similarity Search
Evolving a Medical Image Similarity Search
 
An Enhance Image Retrieval of User Interest Using Query Specific Approach and...
An Enhance Image Retrieval of User Interest Using Query Specific Approach and...An Enhance Image Retrieval of User Interest Using Query Specific Approach and...
An Enhance Image Retrieval of User Interest Using Query Specific Approach and...
 
A novel Image Retrieval System using an effective region based shape represen...
A novel Image Retrieval System using an effective region based shape represen...A novel Image Retrieval System using an effective region based shape represen...
A novel Image Retrieval System using an effective region based shape represen...
 
Searching Images: Recent research at Southampton
Searching Images: Recent research at SouthamptonSearching Images: Recent research at Southampton
Searching Images: Recent research at Southampton
 
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...
Content Complexity, Similarity, and Consistency in Social Media: A Deep Learn...
 

Más de Si Krishan

Machine Learning 2 deep Learning: An Intro
Machine Learning 2 deep Learning: An IntroMachine Learning 2 deep Learning: An Intro
Machine Learning 2 deep Learning: An IntroSi Krishan
 
Bringing AI to Business Intelligence
Bringing AI to Business IntelligenceBringing AI to Business Intelligence
Bringing AI to Business IntelligenceSi Krishan
 
Intro to Excel Basics: Part I
Intro to Excel Basics: Part IIntro to Excel Basics: Part I
Intro to Excel Basics: Part ISi Krishan
 
Introduction to Data Mining
Introduction to Data MiningIntroduction to Data Mining
Introduction to Data MiningSi Krishan
 
Machine learning and multimedia information retrieval
Machine learning and multimedia information retrievalMachine learning and multimedia information retrieval
Machine learning and multimedia information retrievalSi Krishan
 

Más de Si Krishan (6)

Machine Learning 2 deep Learning: An Intro
Machine Learning 2 deep Learning: An IntroMachine Learning 2 deep Learning: An Intro
Machine Learning 2 deep Learning: An Intro
 
Bringing AI to Business Intelligence
Bringing AI to Business IntelligenceBringing AI to Business Intelligence
Bringing AI to Business Intelligence
 
Ml intro
Ml introMl intro
Ml intro
 
Intro to Excel Basics: Part I
Intro to Excel Basics: Part IIntro to Excel Basics: Part I
Intro to Excel Basics: Part I
 
Introduction to Data Mining
Introduction to Data MiningIntroduction to Data Mining
Introduction to Data Mining
 
Machine learning and multimedia information retrieval
Machine learning and multimedia information retrievalMachine learning and multimedia information retrieval
Machine learning and multimedia information retrieval
 

Último

Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media PlatformsMahmoud Yasser
 
PPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggPPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggbhadratanusenapati1
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-ProfitsTimothy Spann
 
Stochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxStochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxjkmrshll88
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Neo4j
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfJasonBoboKyaw
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe321k
 
How to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentHow to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentAggregage
 
Unleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IMUnleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IMMarco Wobben
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingMarketingTrips
 
Using DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data WarehouseUsing DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data WarehouseThinkInnovation
 
Brain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptxBrain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptxShammiRai3
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTimothy Spann
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfmxlos0
 
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsGain Insights
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxEmmanuel Dauda
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptxFurkanTasci3
 

Último (20)

Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media Platforms
 
PPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfggggPPT for Presiding Officer.pptxvvdffdfgggg
PPT for Presiding Officer.pptxvvdffdfgggg
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits
 
Stochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxStochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptx
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
 
Target_Company_Data_breach_2013_110million
Target_Company_Data_breach_2013_110millionTarget_Company_Data_breach_2013_110million
Target_Company_Data_breach_2013_110million
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdf
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe
 
How to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product DevelopmentHow to Build an Experimentation Culture for Data-Driven Product Development
How to Build an Experimentation Culture for Data-Driven Product Development
 
Unleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IMUnleashing Datas Potential - Mastering Precision with FCO-IM
Unleashing Datas Potential - Mastering Precision with FCO-IM
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân Marketing
 
Using DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data WarehouseUsing DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data Warehouse
 
Brain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptxBrain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptx
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdf
 
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded Analytics
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potx
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
 

Image Search: Then and Now

  • 1. Image Search: Then and Now Integrated Knowledge Solutions iksinc@yahoo.com sikrishan@gmail.com iksinc.wordpress.com
  • 2. Outline • Introduction • Image = Content + Context • Content Based Image Retrieval (CBIR) • Bridging the Semantic Gap • Using Social Interactions for Retrieval • Where do we go from here
  • 3. What is Image Search? • Image search means retrieving images from an image database that satisfy the user’s need. • The user need may be expressed in the following ways: – Keywords or text describing the image content – An exemplar image • Other names for image search – Image retrieval – Image similarity search – Content based image retrieval (CBIR)
  • 4. Document Search Not a New Problem
  • 5. Nalanda University was one of the first universities in the world, founded in the 5th Century BC, and reported to have been visited by the Buddha during his lifetime. At its peak, in the 7th century AD, Nalanda held some 10,000 students when it was visited by the Chinese scholar Xuanzang.
  • 6. The Royal Library of Alexandria, in Egypt, seems to have been the largest and most significant great library of the ancient world. It functioned as a major center of scholarship from its construction in the third century B.C. until the Roman conquest of Egypt in 48 B.C.
  • 7. However, Earlier Few Document Producers Many Document Consumers
  • 8. But Now a Days No Distinction Between Document Producers and Consumers
  • 10. Some Relevant Numbers Flickr has over 6 billion pictures as of August 2011, and 3.5 million images are uploaded daily. Photobucket has more than 10 Billion images, and over 4 million images are uploaded everyday. Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday. Instagram has over 20 billion photos. About 60 million photos are uploaded everyday.
  • 11. An image now a days is not just a picture but it is a picture with thousand words
  • 12. Image = Content + Context Tags Cherry blossom Japantown San Francisco Peace Pagoda Content Context
  • 13. So, image retrieval should benefit from the contextual component, if present. How? But, first let us look at image retrieval from the content perspective only
  • 15. A Typical QBIC Type Image Retrieval System Feature Extraction FeaturesMedia Collection Indexing & Matching Query Feature Extraction Retrieved Results Relevance Feedback Such systems/approaches are often referred to as Content Based Image Retrieval (CBIR)
  • 18. Semantic Gap Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept. Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence , Arnold Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain , December 2000 http://www.searchenginejournal.com/7-similarity-based-image-search-engines/8265/
  • 19. Semantic Gap Users also like to query using descriptive words rather than query images or other multimedia objects. This requires retrieval systems to correlate low-level features with high level concepts. Visually dissimilar images representing the same concept.
  • 21. How to Bridge the Semantic Gap? Manual annotation Use machine learning to: • Build image category classifiers to perform semantic filtering of the results • Build specific detectors for objects to associate concepts with images •Build object models using low level features Exploit context: • Text surrounding images • Associated sound track and closed captions in videos • Query history
  • 24. Example of Image Search using Keywords Search result in 2010
  • 25. Example of Image Search using Keywords Search result in 2014 The results are better organized in sub-categories
  • 26. Example of Image Search using Keywords
  • 27. Example of Image Search using Keywords Search result in 2014 Again, the results are better organized in sub- categories
  • 28. Exploiting Context: An Example Kulesh, Petrushin and Sethi, “The PERSEUS Project: Creating Personalized Multimedia News Portal,” Proceedings Second Int’l Workshop on Multimedia Data Mining, 2001
  • 29. Machine Learning of Image Concepts • Challenging problem • Presence of multiple concepts/multiple instances • Disproportionate number of negative examples • Manpower need for labeling training examples
  • 30. Feature Extraction Issues Whole image based features. Easy to use but not very effective Region based features. Both regular region structure and segmented regions are popular Salient objects based features. Connected regions corresponding to dominant visual properties of objects in an image
  • 31. Scale Invariant Feature Transform (SIFT) Descriptors SIFT descriptors or its variants are currently the most popular features in use. Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values http://www.vlfeat.org/ D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” IJCV, 2004.
  • 32. Learning Image Concepts • Both supervised and unsupervised learning methods (SVM, DT, AdaBoost, VQ etc.) have been used • Early work limited to few tens of categories; however some of the current systems can work with thousands of categories/concepts
  • 33. VQ Based Learning Classifier Test Image Best Codebook Label Water Codebook Sky Codebook Fire Codebook Mustafa & Sethi (2004)
  • 35. Bag of Words Representation of Images
  • 36. Co-occurrence of Bag of Words Image Collection Edge Analysis Images Collection of Binary Image Blocks Clustering Local Feature Descriptors (Codewords) Codeword Representation Of Images Co-occurrence Matrices of Local Features Compute Distances Image Distance Matrix Pathfinder Network Mukhopadhyay, Ma, and Sethi, “Pathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discovery,” ISMSE 2004
  • 37. Co-occurrence of BoW Original image Representation by feature indices (cluster membership) Co-occurrence matrix )},(),,(max{),( ABhBAhBAH  ))max(min(),( AaBbbaBAh  Hausdorff metric Manhattan distance
  • 38. Notice how similar images are placed together in the graph
  • 41. Object Detectors for Image Concepts PASCAL Visual Object Classes Challenge
  • 42. Project http://labelme.csail.mit.edu/ Web-based annotation tool to segment and label image regions. Labeled objects in images are used as training images to build object detectors.
  • 43. IMARS provides a large number of built-in classifiers for visual categories that cover places, people, objects, settings, activities and events. It is easy to add new ones. IMARS can work on PC or laptop (trial version is available at IBM alphaWorks). IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day. Several demos of IMARS are available (see IMARS demos) Image Category Classifiers Examples
  • 44. Semantic labeling. (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept. (b) The system represents each image by a vector of posterior concept probabilities. From Pixels to Semantic Spaces: Advances in Content-Based Image Retrieval (Nuno Vasconcelos, IEEE Computer, July 2007) Image Classification via Probabilistic Modeling
  • 45. Image = Content + Context Tags Cherry blossom Japantown San Francisco Peace Pagoda Content Context
  • 46. Tagging All time most popular tags at Flickr
  • 47. About Tags • User centered • Imprecise and often overly personalized • Tag distribution follows power law • Most users use very few distinct tags while a small group of users works with extremely large set of tags • Also known as Folksonomy, social tagging, and social classification
  • 48. Why Not Use Social Tags for Retrieval? Problem: The relevant tag is often not at the top of the list. Only less than 10% of the images have their most relevant tag at the top of the list. Solution: Improve tagging by suggesting potential tags to a user / tag ranking /tag completion etc. Dong Liu, Xian-Sheng Hua, Linjun Yang, Meng Wang, Hong- Jiang Zhang. Tag Ranking. WWW 2009. Madrid, Spain
  • 49. Tag Recommendation using Tags Co-occurrences Given a target image and initial tags, use co-occurrence of tags to recommend tags for the target image. This approach doesn’t take into account the visual features co-occurrences.
  • 50. Tag Recommendation using Tags Co-occurrences and Visual Similarity Kucuktunc, Sevil, Tosun, Zitouni, Duygulu, and Can (SAMT 08) Given a target image and initial tags, use the existing tagged images to suggest tags for the target image.
  • 52. Tag Ranking: Another Approach Dong Liu, Xian-Sheng Hua, Linjun Yang, Meng Wang, Hong- Jiang Zhang. Tag Ranking. WWW 2009. Madrid, Spain
  • 53. How to Compute Tag Similarity
  • 55. Tag Recommendation After Tag Ranking • Given an untagged image, find its visually similar “k” images • Pool the top two ranked tags from k images and select the unique tags as recommended tags
  • 56. Tag Completion The complete tag matrix is generated by imposing constraints based on visual similarity, tag to tag similarity, and similarity with the initial tag matrix. The matrix completion is done by an optimization procedure. Wu and Jain, IEEE-PAMI, JANUARY 2011
  • 57. What about Taggers & Commenters? Question: How can we incorporate taggers/commenters characteristics for improved tag recommendations? Answer: Use three sets of features: derived from image to be tagged, user’s tag history, and user’s social interactions
  • 58. Tag History & Social Interaction Features Tag history features are based on the tags the user has used in the past Social interaction features are derived from tags/comments posted by the user’s friends/favorite posters X. Chen & H. Shin, ICDM 2010
  • 59. Current Status of Image Search • Extensive interest as evident from conferences, journals, and special issues • Overall, solid progress is being made • Efforts towards performance evaluation with benchmarked collections are gaining more traction • Integration of content and context through tags and comments is receiving increasing attention to help improve retrieval • Killer applications are beginning to emerge as visual search gains prominence • Need for more applications outside entertainment
  • 60. Performance Evaluation Efforts ImageCLEF2013 - Annotation Task: - 250000 Training Images - 95 (develop), 116 (test) concepts to be identified - A lot of label Noise inside the training set, due to the automatic label extraction from websites
  • 61. Performance Evaluation Efforts TRECVID workshops, an offshoot of TREC, are yearly evaluation meetings since 2003. The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections, realistic system tasks, uniform scoring procedures, and a forum for organizations interested in comparing their results.
  • 62. Application Examples Tattoo-ID: Automatic Tattoo Image Retrieval for Suspect & Victim Identification (Anil K. Jain, Jung-Eun Lee, and Rong Jin)
  • 63. CBIR for Whole Slide Imageries • The availability of digital whole slide data sets represent an enormous opportunity to carry out new forms of numerical and data- driven query, in modes not based on textual, ontological or lexical matching. – Search image repositories with whole images or image regions of interest – Carry our search in real-time via use of scalable computational architectures Extraction from Image repositories based upon spatial information Analysis of data in the digital domain …001011010111010111.. Resultant Surface Map or gallery of matching images or Slide courtesy of Ulysses J. Balis, M.D. Director, Division of Pathology Informatics Department of Pathology University of Michigan Health System
  • 64. Medical Image Retrieval Text “Find all the cases in which a tumor decrease in size for less than three month post treatment, then resumed a growth pattern after that period” QUERY ? Text + medical image “Find images with large-sized frontal lobes brain tumors for patients approximately 35 years old” +Medical image QUERY IMAGE-BASED CONCEPTS Medical image ij - Specific Signature ImageiQuery VB-Spec CUIp VB-Gen CUI1 VB-Spec CUIkIMAGE-BASED ONTOLOGY GENRAL AND SPECIALIZED QUERY MEDICAL IMAGE VISUAL ANALYSIS Text query CUIn CUI1 CUI2 QUERY TEXT-BASED CONCEPTS Textual query i - Indexes MEDICAL ONTOLOGY TEXT QUERY CONCEPTS EXTRACTION
  • 71. Take Home Message • Image/video retrieval is moving in the commercial domain. Lot more activity is expected in near future • Multimodal/cross-modal retrieval is gaining importance • Approaches combining social search and visual search techniques are expected to gain prominence • Crowdsourcing is a cheap and effective way of tagging media
  • 72. Acknowledgement • This presentation is based on the work of numerous researchers from the MIR/ML/CVPR community. I have tried to give credit/references wherever possible. Any omission is unintentional and I apologize for that. • Also want to thank my present and past students and collaborators.

Notas del editor

  1. MPE(Minimum probability of error