This document discusses using visual analysis techniques to access large audiovisual collections. It describes how digitization programs have created collections with over 800,000 hours of content. Automatic visual analysis techniques like face recognition, speech recognition and visual concept detection could help users find content in these large collections. The document outlines challenges like needing training examples to perform detection and managing user expectations about the capabilities and limitations of these techniques. It advocates exploring how these technologies could help with tasks like query by example for digital humanities researchers while being aware of limitations.

1. ACCESSING LARGE
AUDIOVISUAL COLLECTIONS
USING VISUAL ANALYSIS
AV IN DH WORKSHOP @ DH2014 LAUSANNE
ROELAND ORDELMAN

3. NETHERLANDS INSTITUTE FOR
SOUND AND VISION

4. BUSINESS ARCHIVE
DUTCH PUBLIC BROADCASTERS

5. LARGE DIGITIZATION PROGRAMS

6. CLARIAH PRESENTATIE 11 September 2013 6
+800.000 hours of audiovisual content
‘POTENTIAL’

7. Find what you were (not)
looking for

9. Browse video to find what you
were looking for
X

10. We need labels!

11. Labels connect
(content, context)

12. Labeling

13. CLARIAH PRESENTATIE 11 September 2013 13
BIG DATA!

14. CLARIAH PRESENTATIE 11 September 2013 14
INNOVATIVE PLATFORMS

15. We need USEFUL labels

16. 16
USEFUL?
Developer/
ICT researcher
DH Researcher

17. 17
FEEDBACK

18. Research & Education
Broadcast
Professionals
Hergebruik
Media Archivists
(documentalisten)
Beschrijven
Journalists Research
Academic researchers Investigate
Education Illustrate

19. 19
Use Scenarios
&
System
Requirements
Interview
&
elicitation
sessions
Mock-up
creation &
evaluation
Prototype
evaluation
System
evaluation
Surveys &
log analysis
Qualitative
Qualitative
QualitativeQuantitative
Quantitative/
Qualitative

21. BUILDING PROTOTYPES

22. 2012
2013
-PRO
-RES
Onderzoekers
Media Professionals

23. <nisv@axes> ls –l
Total 10
-r--r--r--. 1 nisv axes 301 Jun 26 2011 METADATA
-r--r--r--. 1 nisv axes 301 Jun 26 2011 SUBTITLES
-r--r--r--. 1 nisv axes 301 Jun 26 2011 SPEECH RECOGNITION
-r--r--r--. 1 nisv axes 301 Jun 26 2011 FACE RECOGNITION
-r--r--r--. 1 nisv axes 301 Jun 26 2011 VISUAL CONCEPT DETECT
-r--r--r--. 1 nisv axes 301 Jun 26 2011 EVENT DETECTION
-r--r--r--. 1 nisv axes 301 Jun 26 2011 LOCATION DETECTION
-r--r--r--. 1 nisv axes 301 Jun 26 2011 QUERY BY EXAMPLE
-r--r--r--. 1 nisv axes 301 Jun 26 2011 SEARCH
-r--r--r--. 1 nisv axes 301 Jun 26 2011 RECOMMENDATION
-r--r--r--. 1 nisv axes 301 Jun 26 2011 USER INTERFACE
<nisv@axes> |

24. Face Recognition

25. Query by example

26. DETECTION REQUIRES TRAINING
(EXAMPLES)
2nd EC review meeting – Hilversum – Mar
19th 2013

27. 2nd EC review meeting –
Hilversum – Mar 19th 2013
EXPECTATION MANAGEMENT

28. 2nd EC review meeting –
Hilversum – Mar 19th 2013

29. Expectation Management
• Expectation management:
– Training examples versus result list
– Google images search versus visual search in AV
• Understanding visual search:
– why something is hard to detect
• visual characteristics, training examples
– Noise is not bad per definition

30. DH perspective
• First explorations in various projects
– Requirements studies
– Demonstrations
– Prototypes
• Technology is ready to start exploring its use in
real use scenarios (e.g., query by example)
• Feed DH ideas into ICT research community

31. Technology exists that could help
Technology does not solve all
problems
Discuss with ICT experts
Technology has a price, what is the
RoI?
AWARENESS
How does technology fit in
How do limitations fit in
‘Technology Critique’ (Historian 2.0?)
ICT and curriculum
METHODOLOGY/TRAINING
What can it do?
How does it work?
How does it perform?
How can it be improved?
MICRO MACRO
How can we use it?
What do we need?
How does it scale?
Who could benefit as well?

32. www.axes-project.eu
roelandordelman.nl
Questions?
<nisv@axes>
<nisv@axes>