Presentation about computer vision using Processing and Jitter for the Interactive Art PhD program at the School of Arts (http://artes.ucp.pt)

1. Computer Vision Techniques forInteractive ArtUsing Processing and Max/Jitter Interactive Art Doctoral Program http://www.artes.ucp.pt/si/doutoramento/index.html 28-05-2011 Jorge Cardoso 1

2. Topics Techniques Framedifferencing(for motiondetection) Color detection(whereintheimage, does a given color appear – can beused for trackingobjects) Brightnessdetection(whatisthepositionofthebrightestpixels) Background subtraction(for objectsegmentation) Blobextraction(applied to theprevioustechniques to extractshapeparametersandvectorizingsilhuettes) Movementestimation (whatdirection are theobjectsmoving) Face detection(isthere a face intheimage? where?) 28-05-2011 Jorge Cardoso 2

4. Computer Vision for Interaction Use a computer and camera to extract information about what people are doing and use that as an implicit or explicit form of interaction “Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53, May 9, 2009, from http://en.wikipedia.org/w/index.php?title=Computer_vision&oldid=285848177 28-05-2011 Jorge Cardoso 4

5. 28-05-2011 Jorge Cardoso 5 Image Computer vision are processes applied to sequences of images An image is just a sequence of number (usually arranged in a matrix form) Eachnumberrepresents a colour 50 44 23 31 38 52 75 52 29 09 15 08 38 98 53 52 08 07 12 15 24 30 51 52 10 31 14 38 32 36 53 67 14 33 38 45 53 70 69 40 36 44 58 63 47 53 35 26 68 76 74 76 55 47 38 35 69 68 63 74 50 42 35 32

6. Image Each pixel (i.e., number in the matrix) isusuallystoredin RGB or ARGB formatso It can be decomposed: 28-05-2011 Jorge Cardoso 6 or

7. Frame Differencing If we subtract two consecutive video frames, threshold and binarize it, we get a rough view of the movement in the video Procedure: Go through the current frame image, pixel by pixel Calculate the distance between the color of the current frame’s pixel and the previous frame’s pixel If distance is smaller than a given threshold, assume no movement (black), otherwise assume movement (white) 28-05-2011 Jorge Cardoso 7

8. Frame Differencing Color difference Q: How do wecalculatethe “distance”betweentwocolors? A: Simpleversion: euclideandistance. Take a color as a pointin 3D space (RGB -> XYZ) Calculatethedistancebetweenthetwopoints Subtracteachcomponent (dR = R1-R2, dG = G1-G2, dB = B1-B2) Dist = Sqrt(dR*dR + dG*dG + dB*dB) A1: Slightly (perceptively) betterversion: weighteachcomponentdifferently: Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB) 28-05-2011 Jorge Cardoso 8

9. Frame Differencing Q: What’sitgood for? A: KALOETHOGRAPHIA (2010) Pedro Cascalheira http://artes.ucp.pt/blogs/index.php/vai/2011/02/14/alunos-2009-2010 aB (2011) Jorge Coutinho https://picasaweb.google.com/lh/photo/x9bGyxFsTLVFbkk9kD3xLA?feat=directlink WebCamPiano MemoAkten [video] 28-05-2011 Jorge Cardoso 9

10. Frame Differencing - Code ProcessingCode FrameDifferencing FrameDifferencingWithMotionDetection FrameDifferencingForAirPiano JitterCode FrameDifferencing.maxpat FrameDifferencingWithMotionDetection.maxpat 28-05-2011 Jorge Cardoso 10

11. Color detection Isolate color regions in an image Procedure: Go through the image, pixel by pixel Calculate the distance between the color of the current pixel and the reference color If distance is smaller than a given threshold, keep the pixel 28-05-2011 Jorge Cardoso 11

12. Color Detection Color Difference For a betterapproachthan RGB distance,use HSB Give more weight to theHue Dist = Sqrt(3*dH*dH + dS*dS + 2*dB*dB) 28-05-2011 Jorge Cardoso 12

13. Color Detection Q: What’sitgood for? A: Catchoftheday [video] PlayDohAsInterfaceKeyboard[video] 28-05-2011 Jorge Cardoso 13

14. Color Detection - Code ProcessingCode ColorDifference ColorDifferenceSequencer JitterCode ColorDifferenceWithFindbounds.maxpat ColorDifference.maxpat 28-05-2011 Jorge Cardoso 14

15. Brightenessdetection 15 Slightvariationon color detectionwhenwe’reonlyinterestedintrackingbrightpixels For example, to track a light Procedure: Same as color detectionbutextractthebrightnessofeach pixel andkeeponlythehighest Jorge Cardoso 28-05-2011

17. A:

18. In RGB:

19. (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B)

20. (physical/energy) Brightness = (R+G+B)/3

21. …

22. In HSB:

23. Brightness = B 

26. A:

27. DrawingWith Light [video]

28. BurningtheSound [video]Jorge Cardoso 28-05-2011

31. Background subtraction Q: How do we “subtract” two images? A: Pixel by pixel Q: How do we “subtract” two pixels? A: Channel by channel (color component by color component, usually R, G and B) 28-05-2011 Jorge Cardoso 21

32. Background subtraction Pixel subtraction alone doesn’t work very well due to noise or shadows in the image The resulting subtraction must be thresholded to eliminate noise 28-05-2011 Jorge Cardoso 22

33. Background subtraction After subtracting and thresholding, we can binarize the image to get a mask 28-05-2011 Jorge Cardoso 23

34. Background subtraction Q: What’sitgood for? A: Isolatingobjectsin a scenewhenyoudon’thavemuchcontroloverthe background. Background SubtractionProcessingDemo [video] 28-05-2011 Jorge Cardoso 24

35. Background subtraction - Code Processingcode: BackgroundSubtraction Jittercode BackgroundSubraction.maxpat 28-05-2011 Jorge Cardoso 25

36. Blob extraction All the previous techniques result in possibly many white areas distributed in the image 28-05-2011 Jorge Cardoso 26 http://en.wikipedia.org/wiki/Connected_Component_Labeling

37. Blob extraction Blob extraction (blob detection, connected component labeling) identifies those regions and and allows the extractions of some features area, perimeter, orientation, bounding box, Silhouette 28-05-2011 Jorge Cardoso 27

38. Blob extraction Procedure: See http://en.wikipedia.org/wiki/Connected_Component_Labeling 28-05-2011 Jorge Cardoso 28

39. Blob extraction Q: What’sitgood for? A: Loliness (2007) Telmo Rocha [video] Células (2009) Ana Lima [video] SilhouetteCollisions [video] 28-05-2011 Jorge Cardoso 29

40. Blob extraction - Code 28-05-2011 Jorge Cardoso 30 Processing libraries

42. Blob extraction - Code Jitter Code (com external cv.jit) BlobExtractionAfterBrightnessDetection 28-05-2011 Jorge Cardoso 32

44. Procedure (naive)

45. Divide bothimagesintoblocks

46. For eachblockinimageone, findtheclosest (more similar) blockinimagetwo

47. http://en.wikipedia.org/wiki/Lucas%E2%80%93Kanade_methodImage: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png Jorge Cardoso 28-05-2011

49. A:

50. opticalflow-particles [video]

51. opticalflow-navigation [video]Jorge Cardoso 28-05-2011

53. Face detection Notrecognition! Needs a configuration file thatdetermines what to detect (front faces, side faces, body, etc) Returnsthepositionandsizeofdetected faces 28-05-2011 Jorge Cardoso 36

54. Face Detection Q: What’sitgood for? A: Anatomias Urbanas (2008) Sara Henriques [video] Ex.001 (2010) Bernardo Santos [foto] https://picasaweb.google.com/lh/photo/_KCdCQGUeEw8lSzF8ZMPVw?feat=directlink 28-05-2011 Jorge Cardoso 37

55. Face Detection - Code Processing (with OpenCV processing library) FaceDetection FaceDetectionFunnyEyes Jitter (with cv.jit) FaceDetectionWithASmiley.maxpat 28-05-2011 Jorge Cardoso 38

57. The End 28-05-2011 Jorge Cardoso 40 http://slideshare.net/jorgecardoso (tag: ea-phd-ia-2011)