Preliminary slides providing an overview of the Python Computer Vision Framework.

1. PyCVF
Overview of
the Python Computer Vision Framework
(preliminary version)
Bertrand NOUVEL
Japanese French Laboratory for Informatics
bertrand.nouvel@gmail.com
May 2010

2. Traditional Framework
Machine
Create Compute Extract Compute Fusion
Learning
Video DB Shots Keyframes Features Rendering
Indexing
Program 1 Program 2 Program 3 Program 4 Program 5 Program 6
Program 7
● Too many steps
● Many access to files : low performances
● Difficult to check and to run
● Not easy to perform testings

3. Models We Need to
Represent
● Learning all along the process;
● Many class­specific learners;
● Link with highlevel ontologies
● Ability to provide top­down and bottom­up
communications
Too complex fort traditional frameworks !
→ Need for improvement

4. Can We Do Better ?
● Avoid usage of archaic file formats
● Use better software design
● Inspire from what is meant to be ”easy”
Blender OpenCV
PureData

5. PyCVF is a Framework
OPENCV WEKA
Dependencies
Libraries MPT Orange
Toolkits
(features) Marsyas R
Framework
PyCVF
(uniformize concepts)
Applications
Analyze Mosaicing
Synthetize Indexer Recognizer
Textures Loop generator
(use and
extend the framework)

6. Essential PyCVF Concepts
DB
Database Models Nodes
Machine Datatypes Structures Experiment
Learning
Models

7. PyCVF Datatypes
● Images → 3d ­ Numpy Array
● Video → Video Player Object (play/pause/seek)
● Audio → Audio Player Object (play/pause/seek)
● Vectors → 1d ­ Numpy Array
● Vectorset → 2d – Numpy Array
Datatypes can solve the following questions ?
● How to store the data ?
● How to display the data ?

8. PyCVF Databases
Supported operations
● Get Datatype
● List Items
DB
● Query
● List Keys
● List Labels
● Query Label

9. Example PyCVF
Image Databases
Image Databases Online Databases
Database :
Mathematical functions
Cellular Automatas...
Pseudo-database :
device
Folder-based Databases

10. Example Meta databases
● Limit : Limit the number of elements
● Transformed : Apply a transformation
● Exploded : Obtain sub­elements
● TrainDB : Create a training associated with a seed
● TestDB : Create a test set associated with a seed
● Randomized : Shuffle the elements of a database
● Agglomerated : Merge databases together
● As_once : Return databases as elements

11. Nodes
● Processing units
● Dynamic typing
● Associated Storage
● Interconnectable
● Labels
● May raise Exceptions
(cf. bag_of_words)

12. Model=PyCVF Dataflow
pycvf_features_view -m ”image.laplace()-image.keypoints.harris()”
Laplace Harris
Corner
Detector

13. PyCVF Dataflow
pycvf_features_view -m ”image.laplace()-image.keypoints.harris()+image.watershed()”
Laplace Harris
Corner
Detector
WaterShed

14. Essential Applications
db_view installdb model_run
evaluate_db features_view
PyCVF
create_index
confusion_matrix distance_matrix
nearest_neighbors
run_experiment remote_job_server compute_features

15. Evaluate Database
pycvf_dbevaluator ....
● PyCVF include a
generic tool to help
you to label your
database
● Works with any
datatypes and any
database.

16. Observe Features
pycvf_model_features_view –db 'image.kanji()' --model 'image.keypoints.sift()'
● Easy debugging
● Better intuition
● Portable

17. Nearest Neighbors Query
pycvf_neareastneighbors_qtapp –db ”image.kanji()” -m ”free('x.ravel()')” -n 5 -b25 -s kanjiidx

18. Nearest Neighbors Query
pycvf_neareastneighbors_panda –db ”image.kanji()” -m ”free('x.ravel()')” -n 5 -b25 -s kanjiidx

19. Extract and Analyze
Nearest Neighbor Graphs
pycvf_neareastneighbors_qtapp –db ”image.kanji()” -m ”free('x.ravel()')” -n 5 -b25 -s kanjiidx

20. Distance Matrices
pycvf_distance_matrix –db ”from_trackfile('HOG')”

21. Making Experiments
● Easy sampling of
parameter space
● Automatic
parallelization of
runs
● Specific graphics
● Specific feature
● Minimization
Performances of PyCVF of index queries

22. NEW
DJANGO
+
PyCVF
=
pycvf_remote_job_server &
python manage.py runserver &

23. Browsing Databases
● On­the­fly
transformations
● Easy queries to already
computed index

24. Viewing Element 1 by 1
● On­the­fly
transformations
● Easy queries to already
computed index

25. Revision Plan
0.2 0.3
● Do call for volunteers ● Add Features
● Ensure global design ● Improve feature
● Easier install stability
● Increase core­robustness
● Provide exhaustive set
of basic features
● Regression tests
● Start add new features

26. See you soon

27. Included Wrappers
● PyFFMPEG is able to
PyFFMPEG play in realtime 121
videos at the same
time.

28. Included Wrappers
Implements
●
OpenCV-Cython exhaustively C API
● Numpy compatibility
● Accessible pointers
● May be deprecated due
to new C++ wrappers

29. Included Wrappers
● Fast nearest neighbor
PySASH search
● Patented + NOT GPL

30. Included Wrappers
● Fast nearest neighbor
PyLSH search with LSH
● Different
implementations
● Still Buggy
● Contributions are
welcome

31. Included Wrappers
PySIFT ● Numpy compatible
version of PySIFT
●