1. Hilbert problems and
challenges
Pietro Perona
California Institute of Technology
NSF Frontiers in Vision Workshop
Cambridge - 21-23 Aug. 2011
Friday, August 26, 2011

2. Scene understanding
Friday, August 26, 2011

3. Scene understanding
1
x= X
X Z
O
x
Friday, August 26, 2011

4. Vision in control loop
Friday, August 26, 2011

5. Other sensors...
Friday, August 26, 2011

6. scene
vision in the loop
understanding
ease of experimentation
goal representation decision/action
active perception
other sensors
Friday, August 26, 2011

7. Visual recognition
Friday, August 26, 2011

8. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

9. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

10. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

11. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

12. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

13. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

14. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

15. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

16. Recognizing..
Materials
Objects
Actions Scenes
Friday, August 26, 2011

17. Geometry and materials
Friday, August 26, 2011

18. How many?
Friday, August 26, 2011

19. Weakly
supervised
learning
Transfer
learning
Friday, August 26, 2011

20. Incremental
learning
...
Friday, August 26, 2011

21. Subordinate categorization
Friday, August 26, 2011

22. Organizing visual knowledge
Friday, August 26, 2011

23. Intentions, causes, consequences, ...
Friday, August 26, 2011

24. Behavior, events
Friday, August 26, 2011

25. Friday, August 26, 2011

26. Friday, August 26, 2011

27. Perception
PSYCHOLOGY
interaction, cooperation,
competition
plans, goals, behavior,
relationships ...
pose, movemes, actions,
activities, objects, scenes
SENSORY
images, trajectories
World
Friday, August 26, 2011

28. Action Perception
PSYCHOLOGY
interaction, cooperation,
PLANNING
group-level goals and plans
competition
SOCIAL NETWORK THEORY OF SOCIOLOGY
INDIVIDUAL
plans, goals, behavior,
individual goals and plans
relationships ...
PREFRONTAL CORTEX THEORY OF PSYCHOLOGY
pose, movemes, actions,
MOTOR
motor programs
activities, objects, scenes
SENSORY
MOTOR CORTEX RECOGNITION
sensor-based control images, trajectories
SPINAL CORD IMAGING,TRACKING
World
Friday, August 26, 2011

29. Behavior
• Hiearchical representation
• Interactions
• Beyond description: intentions, plans,
consequences,
Friday, August 26, 2011

30. Sharing visual
representations with
humans
Friday, August 26, 2011

31. Friday, August 26, 2011

32. Friday, August 26, 2011

33. Grand challenges
Friday, August 26, 2011

34. Drosophila behavior
[Dankert et al., Nature Methods, April 2009]
Friday, August 26, 2011

35. Drosophila behavior
[Dankert et al., Nature Methods, April 2009]
Friday, August 26, 2011

36. VISIPEDIA Users
Images, segments
annotations, links,
GUIs, diagnostics
Experts
Image databases
Annotators Automata Vision
scientists
Friday, August 26, 2011

37. Autonomous driving
<<show movie of traffic in India>>
Friday, August 26, 2011

38. Grand challenges
• Recognition -> Visipedia
• Behavior -> Fly behavior (mouse, ...human)
• Scene understanding -> ???
• Vision for action -> Autonomous driving
(how about manipulation?)
• Sustainable economy -> ???
Friday, August 26, 2011

39. Summary
• Scene understanding vs vision-for-action
• Recognition: just started, much to be done
• Behavior: ditto
• Sharing visual knowledge with humans??
• Grand challenges: many, fun and worthwhile
Friday, August 26, 2011