semantic search engine in a multimedia knowledge base with natural language input

1. Natural language search in multimedia knowledge bases Serge Linckels & Christoph Meinel Hasso Plattner Institute at Potsdam University

2. Hasso Plattner Institute 2 Founded in 1998 10 professors 100 lectures, co-workers… IT-Systems Engineering (Bachelor, Master, PhD)

3. 3

5. Knowledge representation

6. Natural Language Processing

7. Machine Learning

8. …weak AI  artifical intelligence (AI) art or porn?

9. Web 2.0 approach 5 voting tagging art or porn? photography woman black & white 18-200mm VR best picture 2010 Statistical solution Requires critical mass of "good" users

10. Semantic Web approach 6 art or porn? artistic black and white picture of a nakedwoman in a narrowstreet

11. Keyword matching 7 user query black and white of a woman in picture naked narrow street photo nude outdoors daylight search engine data metadata + artistic black and white picture of a nakedwoman in a narrowstreet

12. Keyword search 8 too much information false interpretation false positive

13. Keyword matching problems 9 black and white photo of a nude woman outdoors in daylight of black woman in a white daylight photo and nude outdoors does order matters? black and white photo of a nude woman outdoors indaylight does size matters?

14. Syntax tree 10 Informal language a date Brown tag set noun phrase (NP), prepositional phrase (PP), adjective phrase (ADJP), verb phrase (VP) adjective (JJ), conjuncation (CC), preposition (IN), determiner (DT), noun (NN), verb (VBZ) 21 March 2011 a date presentation CRP-HT

15. Natural language processing 11 semantic interpretation S  Photo hasColor.BW photoOf.(Woman isNude)  isOutdoors.Daylight description logics remove “stop words” verbs, adjectives, adverbs  roles nouns  concepts

16. Martching of concept descriptions 12 artistic black and white picture of a naked woman in a narrow street P  Picture  hasColor.BW  isArtistic  pictureOf.(Woman  isNaked)  isLocated.(Street  isNarrow) ? similarity Q  Photo  hasColor.BW  photoOf.(Woman  isNude)  isOutdoors.Daylight

17. Ontological approach 13 picture image photo movie semantic resources e.g., WordNet pictureOf  photoOf isNude  isNaked equivalences Photo  Picture Photo is subsumed by Picture P  Picture  hasColor.BW  isArtistic  pictureOf.(Woman  isNaked)  isLocated.(Street  isNarrow) ? similarity Q  Photo  Picture  hasColor.BW  pictureOf.(Woman  isNaked)  isOutdoors.Daylight Q  Photo  hasColor.BW  photoOf.(Woman  isNude)  isOutdoors.Daylight

18. Semantic distance 14 Query Object 1 1 2 Object 2 no cover 3 Object 3 Object 4 Object 5 cover rest miss Best cover = object with smallest rest and miss preference is given to smallest miss miss

19. Annotating motion picture 15 Lecture "WWW Grundlagen" by Prof. Meinel #1 Intro #n How TCP/IP works #2 Protocols in general #3 Error-handling as task of a protocol #4 Error-handling LO3 Protocol hasTask.ErrorHandling This clip is about error-handling as a task of a protocol <owl:Class rdf:about="#LO3"> <owl:intersectionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Protocol" /> <owl:restriction> <owl:onProperty rdf:resource="#hasTask" /> <owl:someValuesFrom rdf:resource="#ErrorHandling" /> </owl:restriction> </owl:intersectionOf> </owl:Class>

20. Illustration 16