Audiovisual content exploitation JTS2010
•Descargar como PPTX, PDF•
2 recomendaciones•398 vistas
On audiovisual content exploitation at Netherlands Institute for Sound and Vision and the crowdsourcing application Pinkpop
Recomendados
Recomendados
Más contenido relacionado
Similar a Audiovisual content exploitation JTS2010
Similar a Audiovisual content exploitation JTS2010 (20)
Más de roelandordelman.nl
Más de roelandordelman.nl (13)
Último
Último (20)
Audiovisual content exploitation JTS2010
- 1. Audiovisual content exploitation in the networked information society Crowdsourcing Rock ‘n Roll Multimedia Retrieval Roeland Ordelman Research & Development Netherlands Institute for Sound and Vision rordelman@beeldengeluid.nl
- 2. contents AV content exploitation, annotation technology and user needs NISV context: digitization in Images of the Future Annotation technology for enabling access Annotation technology and user needs Example: Crowdsourcing Rock ‘n Roll Multimedia Retrieval
- 4. and growing:
- 5. digitally born television and radio programs made by the Dutch public broadcasting companies (video: ~15K/hours/year)
- 9. Three goals:
- 10. Safeguarding heritage for future generations
- 11. Creating social- economical value (“unlock the social and economic potential of the collections”)
- 14. A strong business model is necessary to support this kind of investment and prove that such an investment will result in long-term socio-economic returns
- 15. The outcome of a Cost-Benefit analysis was positive: “The total balance of costs and returns of restoring, preserving and digitising audio-visual material (excluding costs of tax payments) will be between: 20+ and 60+ million.’’
- 17. Direct effects of the investment are revenues from sales, access for specific user groups, the repartition of copyright for the use of the material and so on.
- 18. The indirect effects concern the product markets and labour market.
- 19. Social benefits:
- 20. conservation of culture, reinforcement of cultural awareness, reinforcement of democracy through the accessibility of information, increase in multimedia literacy and contribution to the Lisbon goals set by the EUhttp://www.prestoprime.org/project/public.en.html
- 21. Content exploitation: from content is king ...
- 22. ... to metadata rules
- 23. Manual annotation costly & limited
- 24. (SEMI) AUTOMATIC ANNOTATION DECADE+ RESEARCH efforts
- 26. visual features
- 28. crowdsourcing
- 29. deploying collateral data sources:
- 32. national: development of common services
- 33. OAI, Persistent Identifiers, ASR service, Vocabulary Repositories
- 34. commercial: uptake by MNCs (Google and Microsoft) and SMEs
- 35. individual: bring about a shift regarding defensive attitude of content owners towards
- 36. opening up their funded and protected archives
- 38. Video Active, MultiMATCH, VIDI-video, LiWA, P2P-Fusion, Sterna, EUscreen, PrestoPrime
- 39. Collaboration agreement with Dutch research institutes
- 40. Researchers stationed at Sound and Vision
- 41. Provide data (TRECVID, VideoCLEF)
- 42. Research environment: exact copy of iMMix production environment for testing new technology
- 44. video analysis
- 45. fingerprinting
- 48. Opening up
- 49. Focus on community specific requirements
- 50. search needs
- 53. 19 Broadcast professionals In: Huurnink, Hollink, van Den Heuvel 2009 (submitted)
- 54. User survey (broadcast professionals)
- 56. distributed collections (harvested via OAI)
- 57. search & interlink collections via centralized search
- 58. project goals: provide demonstrator portal to show how technology could help researchers acquire information on specific user requirements search collaboration linking privacy dedicated work space http://www.verteldverleden.org
- 59. draw communities into libraries
- 61. exploring the wisdom of crowds by hooking up with user communities (e.g., everyone-as-commentator, unexpected experts)
- 62. capturing relevant information from the internet and aligning this with archived items.
- 64. synchronization of web-content/UGC with AV documents
- 65. users in the loop: UGC for adapting/training analysis tools
- 66. technology aided annotation: Documentalist Support System
- 69. archive internet web content preserve broadcast related websites to use as context information for audiovisual data in the Sound and Vision archive
- 70. AUDIOVISUAL INTERNET CONTENT iMMix AV ARCHIVE CONTEXT CONTEXT BROADCAST RELATED INTERNET CONTENT WEB-ARCHIVE
- 72. internet search might fail as such information is typically available only for a limited amount of time
- 76. Crowdsourcing Rock N’ Roll Multimedia Retrieval Netherlands Institute for Sound and Vision University of Amsterdam – Visual Search (Cees Snoek) University of Twente – Speech Recognition (Franciska de Jong) VideoDock – User Interface (Bauke Freiburg)
- 78. from only summary to almost unabridged recordings, even including raw, unpublished footage as well as interviews
- 79. collection digitized in Images for the Future
- 81. Use speech recognition for browsing interviews
- 82. Exploit popularity of festival to get rock ‘n roll enthusiasts community into the loop:
- 83. general feedback on technology
- 84. improve and extend automatic labeling
- 86. Granted dispensation to use content to enable a large scale study of community-aided annotation and verification via an open internet platform
- 87. for a limited time period of three months,
- 88. video displayed in a secured player
- 90. number of concepts in concerts more or less fixed (in contrast to BN domain), 12 were choosen based on:
- 91. frequency
- 93. previous mentioning in literature
- 94. expected utility for users
- 98. fragment algorithm finds the longest fragments with the highest average scores for a specific concert concept
- 100. Words in transcripts have time-labels
- 101. Transcripts converted to filtered term frequency list on the basis of tf.idf statistics for generating a time-synchronized term cloud:
- 103. colored dots represent concepts
- 104. clicking dot starts playback
- 105. feedback window:
- 106. right/wrong label
- 107. comment
- 109. embed integrated video player,
- 112. full-length concert videos (no ‘commercials’)
- 113. no interruptions, only graphical overlays
- 114. participation threshold kept low:
- 115. no signing up
- 116. just click buttons (thumps-up/down)
- 119. visitors watched on average 3.5 pages, with an average viewing time of 4,57 minutes.
- 120. busiest day was December 3, with 1,566 visitors, immediately after launch and media attention
- 121. Most traffic (65%) originated from 255 referrer sites. The best referrer sites being:
- 124. google
- 125. users provided feedback more than 4000 times.
- 127. access is to a large extend technology driven
- 128. but next to technology development we need to make a shift:
- 129. from a ‘laboratory view’ on users to drawing users and communities into the loop
- 130. NISV is aiming towards this two-way strategy:
- 131. incorporate advanced access technology
Notas del editor
- € 173 mln in 7 jaar (looptijd 2007-2014)wel terugverdienverplichting. Tijdens looptijd € 19 mln
- in the contextdatabase also other information sources
- The amount of footage for each festivalyear varies from only a summary to almost unabridgedconcert recordings, even including raw, unpublished footage
- In contrast to domains like news video, where the numberof visual concepts is unrestricted, the number of conceptsthat may appear in a concert is more or less fixed. A bandplays on stage for an audience. Thus, major concepts arerelated to the role of the band members, e.g. lead singer, orguitarist, and the type of instruments that they play, e.g.,drums or keyboard. Although quite many instruments exist,most bands typically use guitars, drums, and keyboards.We chose 12 concert concepts based on frequency, visual detectionfeasibility, previous mentioning in literature [3, 10],and expected utility for concert video users. For each conceptwe annotated several hundred of examples using theannotation tool depicted in Figure 3 [1]. The 12 concertconcepts are depicted in Figure 4.
- Automatic speech recognition (ASR) technology was usedto attach browsing functionality to the interview fragmentsin the collection. Speech transcripts were generated usingthe SHoUT ASR toolkit [4] and post-processed to generatea filtered term frequency list that is most likely to representthe contents of the interviews, based on tf.idf statistics. Thislist was then used to create a time-synchronized term cloud.Each word in the cloud is clickable to enable users to jumpto the part of the interview where a word is mentioned.
- The main mode of user interaction with our video searchengine is by means of a timeline-based video player, seeFigure 2. The player enables users to watch and navigatethrough a single video concert. Little colored dots on thetimeline mark the location of an interesting fragment correspondingto an automatically derived label. To inspect thelabel and the duration of the fragment, users simply movetheir mouse cursor over the colored dot. By clicking the dot,the player instantly starts the specific moment in the video.If needed, the user can manually select more concept labelsin the panel on the left of the video player. If the timelinebecomes too crowded as a results of multiple labels, the usermay decide to zoom in on the timeline. Besides providingfeedback on the automatically detected labels, we also allowour users to comment on the individual fragments, sharethe fragment through e-mail or Twitter, and embed the integratedvideo player, including the crowdsourcing mechanism,on different websites.
- In order to find a balance between an appealing user experienceand a maximized user participation, we motivateonline users to participate by providing them with access toa selection of exclusive, full-length concert videos. The userswatch the videos without interruption and are encouragedto provide their feedback by graphical overlays that appearon top of the video, see Figure 1.The threshold to participate is deliberately kept low. Usersdo not need to sign up and can provide their feedback justby clicking buttons. With the thumbs-up button they indicatethat they agree with the automatically detected labelfor the video fragment. If they press the thumbs-down button,the user is asked to correct the label. Within a fewclicks the user can select another pre-defined label or createa new label on demand. In addition, the user are allowedto indicate whether the start or end of the fragment wasinconsistent with the label. All user feedback is stored ina database together with the users IP addresses and usersessions.