Near-Duplicate Video Detection Using Temporal Patterns of Semantic Concepts. Paper presented at The IEEE International Symposium on Multimedia (ISM2009) in San Diego, California, USA. http://www.computer.org/portal/web/csdl/doi/10.1109/ISM.2009.93

1. Near-Duplicate Video Detection UsingTemporal Patterns of Semantic Concepts IEEE International Symposium on Multimedia San Diego, California, USADecember 14-16, 2009 Hyun-seok Min, Jaeyoung Choi, Wesley De Neve, Yong Man Ro Image and Video Systems Lab Department of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea

2. Overview Introduction Near-duplicates Semantic video signatures Experimental results Conclusions 2 /20

3. Introduction Importance of duplicate video detection prevents cluttering of search results prevents copyright infringement 3 /20 Search results for the query “I will survive Jesus” A significant number of search results are near-duplicates!

4. Definition of Near-duplicates Identicalor approximately identical videos photometric variations e.g., change of color and lighting editing operations e.g., insertion of captions, logos, and borders speed changes e.g., addition or removal of frames semantic concepts e.g., ‘road’, ‘sand’, ‘snow’ , …. 4 /20

5. Examples of Near-duplicates original videos near-duplicates transformation (cam cording, insertion of subtitles) 5 /20 transformation (blur)

7. represents a video segment with a unique set of features

8. Conventional video signatures

9. often created by extracting low-level visual features fromvideo frames6 /20 video content featureextraction video signature …

10. Use of Low-level Visual Features forCreating a Video Signature Problem near-duplicates may not be visually similar original video near-duplicate transformation (cam cording, insertion of subtitles) Visual match? No! video signature video signature … … 7 /20

11. Semantic Similarity Observation near-duplicates often contain similar semantics original video near-duplicate transformation (cam cording, insertion of subtitles) Semantic match? Yes! Semantic concepts: Semantic concepts: indoor, man, face, … indoor, man, face, … 8 /20

12. Use of Semantic Concepts forCreating a Video signature Semantic concept detection traditionally used for classifying video clips into several predefined concepts Problem limited number of semantic concepts can be detected Solution use of temporal variation of semantic concepts different from video sequence to video sequence 9 /20

13. Semantic Video Signature Creation (1/2) Semantic video signature creation A1 A2 A3 … … Semantic video signature V video shots key frames … concept classification classifier for ‘Street’ classifier for ‘Beach’ classifier for ‘Tree’ Ai AN N: the number of shots M: the number of predefined semantics Ci: ith predefined semantic concept semantic video signature si … sN s2 s1 … 10 /20

14. Semantic Video Signature Creation (2/2) 11 /20 original video near-duplicate transformation … … … … Semantic video signature of original video Semantic video signature of near-duplicate

15. Matching Procedure 12 /20 Semantic video signature of near-duplicate Semantic video signature of original video

16. Experimental Setup (1/3) Reference database video sequences taken from TRECVID2007 over 9 hours of video data format: MPEG-1 resolution: 352X288 frame rate: 25 frame per second (fps) Screenshots 13 /20

17. Experimental Setup (2/3) Creation of query video (near-duplicate) set number of query video sequences 64 in total average length of the query video sequences 3 minutes Process for generating query video sequences original video sampling subvideoof original video transformation query video 14 /20

18. Experimental Setup (3/3) Transformations used spatial transformations Gaussian blur logo insertion letter-box resizing temporal transformations change of frame rate original 15 /20

19. Experimental Results: Spatial Transformation (1/2) 16 /20 The precision increases as the threshold value decreases, while in turn, the recall value decreases. blur letter-box

20. Experimental Results: Spatial Transformation (2/2) 17 /20 Ordinal measurement does not work well with logo insertion, compared to the proposed method.

21. Experimental Results: Temporal Transformation 18 /20 The proposed method is robust against temporally modified video sequences.

22. Conclusions Proposed the use of semantic video signatures for near-duplicate video detection relies on a number of semantic concepts detected along the temporal axis Experimental results indicate that the use of a semantic video signature looks promising Future work improving the accuracy of semantic concept detection use of additional semantic concepts 19 /20

23. Any questions or comments? 20/20