1. A Large Scale Concept Ontology
for Multimedia Understanding
Milind Naphade, John R. Smith, Alexander Hauptmann, Shih-Fu Chang &
Edward Chang
IBM Research, Carnegie Mellon University, Columbia University & University of California at Santa Barbara
naphade@us.ibm.com jsmith@us.ibm.com alex@cs.cmu.edu sfchang@ee.columbia.edu
echang@xanadu.ece.ucsb.edu
April 2005
NRRC NWRRC
MITRE

2. Central Idea
• Collaborative activity of three
critical communities – Users,
Library Scientists and Knowledge
Experts, and Technical
Researchers, Algorithm, System
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Page
PNN MITRE

3. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Page
PNN MITRE

4. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Vision, Machine
Learning, Detection
Analytics
Technical Researchers, Algorithm
Designers & System Developers
Page
PNN MITRE

5. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Knowledge
Vision, Machine Representation,
Learning, Detection Library Scientists
Analytics Standardization
Technical Researchers, Algorithm Ontology Experts
Designers & System Developers
Page
PNN MITRE

6. Central Idea
• Collaborative activity of three Users (Analysts,
critical communities – Users, Broadcasters)
Intelligence
Library Scientists and Knowledge
Community,
Experts, and Technical
Broadcasting
Researchers, Algorithm, System
Corporations
and Solution Designers – to create
a user-driven concept ontology for
analysis of video broadcast news
Lexicon and
Ontology
1000 or more
Knowledge
concepts
Vision, Machine Representation,
Learning, Detection Library Scientists
Analytics Standardization
Technical Researchers, Algorithm Ontology Experts
Designers & System Developers
Page
PNN MITRE

7. Problem
• Users and analysts require richly annotated video content for
accomplishing required access and analysis functions over
massive amount of video content.
• Big Barriers:
- Research community needs to advance technology for
bridging gap from low-level features to semantics
- Lack of large scale useful well-defined semantic lexicon
- Lack of user-centric ontology
- Lack of corpora annotated with rich lexicon
- Lack of feasibility studies for any ontology if defined
• Examples:
- The TRECVID lexicon defined from a frequentist
perspective. Its not user-centric.
• No effort to date to design lexicon by joint partnership between
different communities (users, knowledge experts, technical)
Page
PNN MITRE

8. Workshop Goals
Page
PNN MITRE

9. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
Page
PNN MITRE

10. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
Page
PNN MITRE

11. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
Page
PNN MITRE

12. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
Page
PNN MITRE

13. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
Page
PNN MITRE

14. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
Page
PNN MITRE

15. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
Page
PNN MITRE

16. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
Page
PNN MITRE

17. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
Page
PNN MITRE

18. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
Page
PNN MITRE

19. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
- Perform benchmark concept modeling, detection and evaluation
Page
PNN MITRE

20. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
- Perform benchmark concept modeling, detection and evaluation
- Analyze concept detection performance and revise concept ontology
Page
PNN MITRE

21. Workshop Goals
• Organize series of workshops that bring together three critical
communities – Users, Library Scientists and Knowledge Experts, and
Technical Researchers – to create a ontology on order of 1000
concepts for analysis of video broadcast news
• Ensure impact through focused collaboration of these different
communities to achieve balance of usefulness, feasibility and size
• Specific Tasks:
- Solicit input on user needs and existing practices
- Analyze applications, prior work, concept modeling requirements
- Develop draft concept ontology for video broadcast news domain
- Solicit input on technical capabilities
- Analyze technical capabilities for concept modeling and detection
- Form benchmark and define annotation tasks
- Annotate benchmark dataset
- Perform benchmark concept modeling, detection and evaluation
- Analyze concept detection performance and revise concept ontology
- Conduct gap analysis and identify outstanding research challenges
Page
PNN MITRE

22. Workshop Format and Duration
Page
PNN MITRE

23. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
Page
PNN MITRE

24. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
Page
PNN MITRE

25. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
Page
PNN MITRE

26. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
Page
PNN MITRE

27. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
Page
PNN MITRE

28. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
Page
PNN MITRE

29. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
Page
PNN MITRE

30. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
Page
PNN MITRE

31. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
Page
PNN MITRE

32. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
Page
PNN MITRE

33. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
Page
PNN MITRE

34. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
Page
PNN MITRE

35. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
• Part 1: Validation and Refinement
Page
PNN MITRE

36. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
• Part 1: Validation and Refinement
• Part 2: Outstanding Challenges and Recommendations
Page
PNN MITRE

37. Workshop Format and Duration
• Propose to hold two multi-week workshops accompanied by
annotation, experimentation, and prototyping tasks
• Focus on video broadcast news domain
• Workshop Organization:
- Pre-workshop 1: Call for Input on User Needs and Existing Practices
- Ontology Definition Workshop (two-weeks):
• Part 1: User Needs
• Part 2: Technical Analysis
- Ad hoc Tasks
• Task 1: Annotation
• Task 2: Experimentation
• Task 3: Evaluation
- Ontology Evaluation Workshop (two-weeks):
• Part 1: Validation and Refinement
• Part 2: Outstanding Challenges and Recommendations
• Substantial off-line tasks for annotation and experimentation require
Page
PNN MITRE

38. Broadcast News Video Content Description Ontology
• Why the Focus on Broadcast News Domain?
Broadcast News Ontology
- Critical mass of users, content providers, applications
- Good content availability (TRECVID, LDC, FBIS)
- Shares large set of core concepts with other domains
•
News
Ontology Formalism:
Production Broadcast
Grammars
- Entity-Relationship (E-R) Graphs
News
Content
- RDF, DAML / DAML+OIL, W3C OWL
News
- MPEG-7, MediaNet, VEML
Domain
• Seed Representations:
- TRECVID-2003 News Lexicon (Annotation Forum)
Core
- Library of Congress TGM-I
Video
- CNN, BBC Classification Systems
MPEG-7 Video Annotation Tool
Page
PNN MITRE

39. Broadcast News Video Content Description Ontology
• Why the Focus on Broadcast News Domain?
Broadcast News Ontology
- Critical mass of users, content providers, applications
- Good content availability (TRECVID, LDC, FBIS)
- Shares large set of core concepts with other domains
•
News
Ontology Formalism:
Production Broadcast
Grammars
- Entity-Relationship (E-R) Graphs
News
Content
- RDF, DAML / DAML+OIL, W3C OWL
News
- MPEG-7, MediaNet, VEML
Domain
• Seed Representations:
- TRECVID-2003 News Lexicon (Annotation Forum)
Core
- Library of Congress TGM-I
Video
- CNN, BBC Classification Systems
Concepts
Objects Sites
Actions
Person
Outdoors Indoors
People
Face
News
Monolog
News News
Anchor Studio
Subject Dialog
Crowd
MPEG-7 Video Annotation Tool
Page
PNN MITRE

40. Broadcast News Video Content Description Ontology
• Why the Focus on Broadcast News Domain?
Broadcast News Ontology
- Critical mass of users, content providers, applications
- Good content availability (TRECVID, LDC, FBIS)
- Shares large set of core concepts with other domains
•
News
Ontology Formalism:
Production Broadcast
Grammars
- Entity-Relationship (E-R) Graphs
News
Content
- RDF, DAML / DAML+OIL, W3C OWL
News
- MPEG-7, MediaNet, VEML
Domain
• Seed Representations:
- TRECVID-2003 News Lexicon (Annotation Forum)
Core
- Library of Congress TGM-I
Video
- CNN, BBC Classification Systems
Concepts
Objects Sites
Actions
Person
Outdoors Indoors
People
Face
News
Monolog
News News
Anchor Studio
Subject Dialog
Crowd
MPEG-7 Video Annotation Tool
Page
PNN MITRE

41. Approach (Pre-workshop and 1st workshop)
Page
PNN MITRE

42. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
Page
PNN MITRE

43. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
Page
PNN MITRE

44. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Page
PNN MITRE

45. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
Page
PNN MITRE

46. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
Page
PNN MITRE

47. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Page
PNN MITRE

48. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
Page
PNN MITRE

49. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
Page
PNN MITRE

50. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
Page
PNN MITRE

51. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Page
PNN MITRE

52. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
Page
PNN MITRE

53. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
Page
PNN MITRE

54. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
• Form benchmark and define annotation tasks
Page
PNN MITRE

55. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
• Form benchmark and define annotation tasks
Output: Version 1
Page
PNN MITRE

56. Approach (Pre-workshop and 1st workshop)
• Pre-workshop: Call for Input
- Solicit input on user needs and existing practices
• Ontology Definition Workshop
Part 1: User Needs
• Analyze use cases, concept modeling requirements, prior
lexicon and ontology work
• Develop draft concept ontology for video broadcast news
domain
Output: Version 1
• Requirements and Existing Practices
• Domain Concepts and Ontology System
• Video Concept Ontology
Part 2: Technical Analysis
• Analyze technical capabilities for concept modeling and
detection
• Form benchmark and define annotation tasks
Output: Version 1
• Benchmark (Use cases, Annotation)
Page
PNN MITRE

57. Approach (Ad-hoc Tasks and 2nd workshop)
Page
PNN MITRE

58. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Page
PNN MITRE

59. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
Page
PNN MITRE

60. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Page
PNN MITRE

61. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
Page
PNN MITRE

62. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Page
PNN MITRE

63. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
Page
PNN MITRE

64. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Page
PNN MITRE

65. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
Page
PNN MITRE

66. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
Page
PNN MITRE

67. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
Page
PNN MITRE

68. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Page
PNN MITRE

69. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
Page
PNN MITRE

70. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Page
PNN MITRE

71. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
Page
PNN MITRE

72. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Page
PNN MITRE

73. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
Page
PNN MITRE

74. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
Page
PNN MITRE

75. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Page
PNN MITRE

76. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
Page
PNN MITRE

77. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
- Conduct gap analysis and identify outstanding research challenges
Page
PNN MITRE

78. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
- Conduct gap analysis and identify outstanding research challenges
Output:
Page
PNN MITRE

79. Approach (Ad-hoc Tasks and 2nd workshop)
• Ad hoc Group
Task 1: Annotation
- Annotate benchmark dataset
Task 2: Experimentation
- Perform benchmark concept modeling and detection
Task 3: Evaluation
- Evaluation of concept detection, ontology and use of automatic detection
for use cases and evaluation
Output:
- Benchmark v.2
- Concept Detection Evaluation v.1
- Ontology Evaluation v.1
Query Answering Effectiveness with Automated Detection Evaluation v.1
-
• Ontology Evaluation Workshop
Part 1: Validation
- Analyze evaluation of ontology, concept detection and its application to use case
answering.
Output
- Domain Concepts v.2 and Ontology System v.2
- Video Concept Ontology v.2
Part 2: Outstanding Challenges
- Conduct gap analysis and identify outstanding research challenges
Output:
- Research Challenges v.1
Page
PNN MITRE

80. Input
Task
s
Output
Documents
Page
PNN MITRE

81. Users: Ex. Ontology & catalog systems:
Catalog
 IC analysts  RDF
Use cases Existing practices
terms
 Broadcasters  Topic Maps
User needs
Ex. Usage:  DAML / DAML+OIL
 Searching  W3C OWL
Ontology
 Browsing  Dublin Core
Queries Queries systems
 Navigation  OCLC
 Threading  Open Directory Project
 Summarization  MARC
 MODS
 MediaNet
Pre-Workshop:  VEML
Call for Input Ex. Catalog terms
Prior work
Appl.
 LOC TGM-I
Analysis
Analysis  MPEG-7 classif. Schemes
Ex. Queries
 TRECVID
 BBC
Input
Task
s
Output
Documents
Page
PNN MITRE

82. Users: Ex. Ontology & catalog systems:
Catalog
 IC analysts  RDF
Use cases Existing practices
terms
 Broadcasters  Topic Maps
User needs
Ex. Usage:  DAML / DAML+OIL
 Searching  W3C OWL
Ontology
 Browsing  Dublin Core
Queries Queries systems
 Navigation  OCLC
 Threading  Open Directory Project
 Summarization  MARC
 MODS
 MediaNet
Pre-Workshop:  VEML
Call for Input Ex. Catalog terms
Prior work
Appl.
 LOC TGM-I
Analysis
Analysis  MPEG-7 classif. Schemes
Ex. Queries
 TRECVID
 BBC
• Identifies
• Documents
indexing
Existing
Requirements
requirements for existing
Practices
Study v.1
broadcast news practices for
Study v.1
video including indexing
example queries broadcast news
Workshop 1:
User Needs
Modeling
Concept
Analysis
Analysis
• Specifies
• Identifies and ontology system
Ontology
Domain
defines domain for broadcast
System
Concepts
concepts, terms, news video
Study v.1
Study v.1
classification domain
systems for
broadcast news
Input
video (ex. Ontology
objects, actions,
Design
sites, events)
Task
s
• Specifies draft
lexicon and Output
Video Concept ontology for Documents
Ontology broadcast news
Page v.1 video domain
PNN MITRE

83. Input
Task
s
Output
Documents
Page
PNN MITRE

84. Systems:
Technical capabilities  Video logging
Video Concept Systems &  Video retrieval
Ontology Prototypes
Ex. State-of-art techniques:
Workshop 1: Draft v.1  Content-based search
Benchmark
Technical Analysis  Segmentation
State-of-art Results  Tracking
Techniques  High-level feature detection
 Story segmentation
Ex. Benchmarks:
 TRECVID
Benchmark
• Defines Formation
benchmark for Technical
concept Analysis
detection for
video broadcast
news (dataset,
detection and
search tasks,
Benchmark
metrics)
Draft v.1
Input
Task
s
Output
Documents
Page
PNN MITRE

85. Systems:
Technical capabilities  Video logging
Video Concept Systems &  Video retrieval
Ontology Prototypes
Ex. State-of-art techniques:
Workshop 1: Draft v.1  Content-based search
Benchmark
Technical Analysis  Segmentation
State-of-art Results  Tracking
Techniques  High-level feature detection
 Story segmentation
Ex. Benchmarks:
 TRECVID
Benchmark
• Defines Formation
benchmark for Technical
concept Analysis
detection for
video broadcast
news (dataset,
detection and
search tasks,
Benchmark
metrics)
Draft v.1
• Identifies and
maps
Ad Hoc Task 1: Feature Concept techniques for
Extraction Modeling modeling and
Annotation detecting each
• Identifies v. 1 v.1
Annotation of draft
features
Task 1 concepts
required for
modeling each
of draft
• Refines concepts
Ad Hoc Task 2,3:
benchmark to Input
include Benchmark Experimentation Experimentation
annotated Draft v.2
ground-truth for
Task
experimentation
s
and evaluation
Query Evaluation
Concept with Automatic Output
Detection Detection v.1 Documents
Evaluation v.1
Ontology
Page Evaluation v.1
PNN MITRE

86. Workshop 2: Evaluation
Input
Task
s
Output
Documents
Page
PNN MITRE

87. Ontology
Query Evaluation
Evaluation v.1
Concept with Automatic
Detection Workshop 2: Evaluation
Detection v.1
Evaluation v.1
Analysis
• Revises
• Revises lexicon ontology system
Ontology
Domain
based on based on
System
Concepts
performance performance
Study v.2
Study v.2
analysis analysis
Ontology
Re-design
Requirements
Study v.1
• Refines lexicon
Video Concept
and ontology for
Ontology
broadcast news
v.2
video domain
Input
Task
s
Output
Documents
Page
PNN MITRE

88. Ontology
Query Evaluation
Evaluation v.1
Concept with Automatic
Detection Workshop 2: Evaluation
Detection v.1
Evaluation v.1
Analysis
• Revises
• Revises lexicon ontology system
Ontology
Domain
based on based on
System
Concepts
performance performance
Study v.2
Study v.2
analysis analysis
Ontology
Re-design
Requirements Workshop 2:
Study v.1
Outstanding Challenges
• Refines lexicon
Video Concept
and ontology for
Ontology
broadcast news
v.2
video domain
Gap
Analysis
Input
Task
• Identifies and
s
• Recommendations defines
Recommendations Research technology gaps
for ontology
v.1 Challenge v.1 and challenges
exploitation and
Output
for future
solution design
Documents
research
Page
PNN MITRE

89. Domain and Data Sets
• Candidate data set:
- TRECVID Corpus (>200 hours of video broadcast news from CNN
and ABC). Has the following advantages
• availability
• generalization capability better with than other domains
• # of research groups up to speed on this domain for tools/detectors
• TREC established some benchmark and evaluation metrics already.
- Will avoid letting domain specifics influence the design of ontology to an extent where
the ontology starts catering to artifacts of the BN domain.
- Will seek other sources such as FBIS, WNC etc.
• Annotation issues:
- Plan to leverage prior video annotation efforts where possible (e.g.,
TRECVID annotation forum)
- Hands-on annotation effort will induce discussions and requires
refinements of concepts meanings
Page
PNN MITRE

90. Evaluation Methods
• Require benchmarks and metrics for evaluating:
- Utility of ontology – coverage of queries in terms of quality and quantity
- Feasibility of ontology:
• Accuracy of concept detection and degree of automation (amount of
training)
• Effectiveness of query systems using automatically extracted
concepts
• Metrics of Retrieval Effectiveness
- Precision & Recall Curves, Average Precision, Precision at Fixed Depth
• Metrics of Lexicon Effectiveness
- Number of Use Cases that can be answered by lexicon successfully
- Mean average precision across the set of use cases
• Evaluate at multiple levels of granularity:
- Individual concept, classes, hierarchies
Page
PNN MITRE

91. Confirmed Participants – Knowledge Experts and Users
Page
PNN MITRE

92. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge
representation (definition of
lexicon):
 Corrine Jorgensen, School of
Information Studies, Florida State
University
 Barbara Tillett, Chief of Cataloging
Policy and Support, Library of
Congress
 Jerry Hobbs, USC / ISI
 Michael Witbrock, Cycorp
 Ronald Murray, Preservation
Reformatting Division, Library of
Congress
Page
PNN MITRE

93. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge
representation (definition of
lexicon):
 Corrine Jorgensen, School of
Information Studies, Florida State
University
 Barbara Tillett, Chief of Cataloging
Policy and Support, Library of
Congress
 Jerry Hobbs, USC / ISI
 Michael Witbrock, Cycorp
 Ronald Murray, Preservation
Reformatting Division, Library of
Congress
R&D Agencies
 John Prange, ARDA
 Sankar Basu, Div. of Computing and
Comm. Foundations, NSF
 Maria Zemankova, Div. of Inform. and
Intell. Systems., NSF
Page
PNN MITRE

94. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge Standardization and Benchmarking
representation (definition of (theoretical and empirical
lexicon): evaluation):
 Corrine Jorgensen, School of  Paul Over, NIST
Information Studies, Florida State  John Garofolo, NIST
University  Donna Harman, NIST
 Barbara Tillett, Chief of Cataloging  David Day, MITRE
Policy and Support, Library of  John R. Smith, IBM Research
Congress
 Jerry Hobbs, USC / ISI
 Michael Witbrock, Cycorp
 Ronald Murray, Preservation
Reformatting Division, Library of
Congress
R&D Agencies
 John Prange, ARDA
 Sankar Basu, Div. of Computing and
Comm. Foundations, NSF
 Maria Zemankova, Div. of Inform. and
Intell. Systems., NSF
Page
PNN MITRE

95. Confirmed Participants – Knowledge Experts and Users
Library Sciences and Knowledge Standardization and Benchmarking
representation (definition of (theoretical and empirical
lexicon): evaluation):
 Corrine Jorgensen, School of  Paul Over, NIST
Information Studies, Florida State  John Garofolo, NIST
University  Donna Harman, NIST
 Barbara Tillett, Chief of Cataloging  David Day, MITRE
Policy and Support, Library of  John R. Smith, IBM Research
Congress
 Jerry Hobbs, USC / ISI
 Michael Witbrock, Cycorp
 Ronald Murray, Preservation
Reformatting Division, Library of
User Communities (interpretation of
Congress
use cases for lexicon definition,
broadcasters help getting query logs
for finding useful lexical entries)
R&D Agencies
 Joanne Evans, British Broadcasting
 John Prange, ARDA
Corporation
 Sankar Basu, Div. of Computing and
 Chris Porter, Getty Images
Comm. Foundations, NSF
 ARDA and analysts
 Maria Zemankova, Div. of Inform. and
Intell. Systems., NSF
Page
PNN MITRE

96. Confirmed Participants – Technical Team
Theoretical Analysis: Experimentation: (Help Prototyping: (Help with
address evaluation issues prototyping tools for
(Help conduct analysis
for lexicon, ontology and annotation, evaluation,
during initial lexicon and
concept evaluation) querying, summarization
ontology design)  Alexander and statistics gathering)
 Shih-Fu Chang,
Hauptmann, CMU
 Milind R. Naphade, IBM
Columbia University
Research  Alan Smeaton, Dublin
 Ramesh Jain, Georgia  Edward Chang,
City University
Institute of Technology UCSB
 HongJiang Zhang,
 Thomas Huang, UIUC
 Nevenka Dimitrova,
 Microsoft Research
Edward Delp, Purdue
Phillips Research
University  Ajay Divakaran, MERL
 Rainer Lienhart, Intel
 Wessel Kraaij,
 Apostol Natsev, IBM
Information Systems
Research
Division, TNO TPD
 Tat-Seng Chua, NUS
 Ching-Yung Lin, IBM
 Ram Nevatia, USC
Research
 John Kender,
 Mubarak Shah,
Columbia University
University of Central
Florida
Page
PNN MITRE

97. Impact and Outcome
• First of a Kind Ontology of 1000 or more semantic concepts that have been
evaluated for their usability and feasibility by different communities including UC,
OC, MC.
• Annotated corpus (200 hours) and ontology can be further exploited for future
TRECVID, VACE, MPEG-7 activities. Core semantic primitives, that can be included
in various video description standards/languages such as MPEG-7.
• Empirical and theoretical study of automatic concept detection performance for
elements of this large ontology. Use of current state of the art detection wherever
possible. Use of simulation where the detection is not available.
• Use cases (queries) testing and expansion into ontology
• Reports documenting use cases, existing practices, research challenges and
recommendations
• Prototype systems and tools for annotation, query formulation and evaluation
• Guidelines on manual and automatic multimedia query formulation techniques going
from use-cases to concepts.
• Categorization of classes of concepts based on feasibility, detection performance
and difficulty in automation
BOTTOMLINE: All this is driven by the user
Page
PNN MITRE

98. Summary of Key Questions
• How easy was it to create annotations
- (man-hours/hr of video?)
• How well does the lexicon 'partition' the collection
• Given perfect annotations/classification:
- How well does the lexicon aid with queries/tasks
• How good is automatic annotation of the sample collection
- What fraction of perfect annotations accuracy is obtained for the
queries/tasks
• How much is automatic classification performance of a given lexical item
a function of training data
- Estimate how much training data would get this lexical item to 60%,
80%, 90%, 95%?
• What lexicon changes are necessary or desirable?
Page
PNN MITRE

99. Video Event Ontology (VEO) & VEML
• A Video Event Ontology was developed in the ARDA workshop on
video event ontologies for surveillance and meetings allows natural,
hierarchical representation of complex spatio-temporal events
common in the physical world by a composition of simpler (primitive)
events
• VEML – XML-derived Video Event Markup Language used to annotate
data by instantiating a class defined in that ontology. Example: We
will attempt to use or adapt their notation to the extent possible
• (http://www.veml.org:8668//space/2003-10-08/StealingByBlocking.veml)
• Broadcast video news ontology is likely to have little overlap with the
complex surveillance events described in the VEO, except for some
basic concepts. We expect our ontology to be broader, but much
shallower
• Our broadcast news ontology is largely applicable to any edited
broadcast video (e.g. documentaries, talk shows, movies) and
somewhat applicable to video in general (including surveillance, UAV
and home videos).
Page
PNN MITRE