COMP 4026 Lecture 5 OpenFrameworks and Soli
•
3 recomendaciones•912 vistas
COMP 4026 Advanced HCI lecture 6 on OpenFrameworks and Google's Project Soli. Taught by Mark Billinghurst at the University of South Australia on August 25th 2016.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a COMP 4026 Lecture 5 OpenFrameworks and Soli
Similar a COMP 4026 Lecture 5 OpenFrameworks and Soli (20)
Más de Mark Billinghurst
Más de Mark Billinghurst (20)
Último
Último (20)
COMP 4026 Lecture 5 OpenFrameworks and Soli
- 1. LECTURE 5: OPENFRAMEWORKS AND SOLI COMP 4026 – Advanced HCI Semester 5 - 2016 Mark Billinghurst University of South Australia August 25th 2016
- 2. RECAP
- 3. Advanced Interface Technology • Wearable Computers • Augmented Reality • Virtual Reality • Invisible Interfaces • Environment Sensing • Physiological Sensing
- 4. Class Project 1. Pick Advanced Technology 2. Brainstorm use case 3. Develop conceptual design 4. Prototype interface/experience design 5. Conduct user evaluation 6. Repeat steps 3-5 7. Write report
- 5. Wearable Computing ▪ Computer on the body that is: ▪ Always on ▪ Always accessible ▪ Always connected ▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
- 7. Google Glass
- 10. 1977 – StarWars
- 11. Augmented Reality Definition • Defining Characteristics [Azuma 97] • Combines Real andVirtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 12. Virtual Reality • ImmersiveVR • Head mounted display, gloves • Separation from the real world
- 13. AR vsVR
- 14. Early Examples • Interaction without devices: • BodySpace [Strachan 2007]: Functions to body position • Abracadabra [Harrison 2007]: Magnets on finger tips • GesturePad [Rekimoto 2001]: Capacitive sensing in clothing • Palm-based Interaction • Haptic Hand [Kohli 2005]: Using non-dominant hand in VR • Sixth Sense [Mistry 2009]: Projection on hand • Brainy Hand [Tamaki 2009]: Head worn projector/camera
- 15. ImaginaryPhone • Gustafson, S., Holz, C., & Baudisch, P. [2011]
- 17. Invisible Interfaces – Gestures in Space • Gustafson, S., Bierwirth, D., & Baudisch, P. [2010] • Using a non-dominant hand stabilized interface.
- 18. Project Soli • Using Radar to support free-hand spatial input
- 19. Google Tango • Tablet based system • Android OS • Multiple sensors • RGBD Sensor • IR Structured light • Inertial sensors • High end graphics • Nvidia tegra chip
- 20. Physiological Sensors • Sensing user state • Body worn devices • Multiple possible sensors • Physical activity • Eye tracking, gaze • Heart rate • GSR • Breathing • Etc
- 21. Tobii Eye Tracker • Wearable eye tracking system • Natural data capture • Scene camera capture • Recording/streaming eye gaze, 60 Hz sampling
- 22. OPENFRAMEWORKS
- 23. OpenFrameworks (www.openframeworks.cc) • Open source toolkit designed for creative coding • Developed by Z. Lieberman,T.Watson and A. Castro • Framework – collection of libraries • Written in C++ • More powerful than Processing, but more complicated • Must use IDE for development • Xcode,Visual Studio, Code::Blocks • Runs on Mac,Windows, Linux platforms
- 26. Why use oF instead of Processing • Speed • Accessibility of low level information • Debugger • C++ • Version control • Cross Platform
- 27. OpenFrameworks vs.Processing • Making project visible on Internet - Processing • Make a project with lots of 3D graphics - OpenFrameworks • Make a project for lots of different computers/OS – Processing • Make a project using an external library like the OpenCV computer vision library – OpenFrameworks • Make a project that interfaces with the Arduino board - Either
- 28. OpenFrameworks Installation • addons: added libraries from user community. Must be explicitly included in programs using them • apps: store your programs here.Also contains example code. • libs: where the core libraries of OpenFrameworks are stored. Also contains core openFrameworks folder
- 40. testApp.cpp
- 44. Typical FunctionTypes • setup( ) • load assets • Initialize values • Initialize addons or components • update( ) • calculations • increment video frames • draw( ) • draw shapes/images/videos • use GLSL Shaders
- 47. Classes in C++ • C++ classes comprise of two files. It helps to think of these two files as a recipe. • The header file (.h) is like the list of ingredients, and contains: • Any preprocessor statements there to prevent multiple header definitions • Any include statements to other classes • Any class extension statements • Any variables local to the class • Prototypes of any functions to be contained in the class • Security settings of these functions and variables (e.g. public, private, protected, etc). • and a body file (.cpp) which is like the instructions on what to do with the ingredients and contains: • An include statement that references the .h file • All of the code to fill in the function prototypes.
- 48. Class Extending • Take one class and add functionality to it with a new class • Eg enemy class for video game !class Enemy {! ! !int x, y; //position! ! !.. .. ! ! !public void draw() {! ! !//draw my picture to the screen at the proper location }! !}! • Want to draw enemy twice – create new class !//on a "DoubleEnemy.h" file! !class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}! !{! ! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file! !}; // note the ";" at the end of the class statement!
- 49. Pass byValue vs.by Reference • void functn(int num) – pass by value • void functn(<class> test) – pass by reference • sends address of where class stored • use pointers to pass arrays back and forth through functions int num = 5; value stores address of variable value
- 50. void setup() { int num = 1; addOne(num); print(num); } void addOne(int num) { num++; } class Test { int num=0; } void setup() { Test test = new Test(); test.num=1; addOne(test); print(test.num); } void addOne(Test test) { test.num++; } Pass by Value Pass by Reference
- 51. & and * • In C++ you need to explicitly state whether you are passing something by value or by reference. • Use & (referencing) and * (dereferencing) symbols • the & symbol is used to acquire the memory address of a variable or function b=1;! a = &b; // a now equal to memory address of b! a++; // memory address of b + 1! *a++; // value a +1 (increments b as well)!
- 52. Example • What does this code do? ! ! !int x;! ! ! !int *ptr;! ! ! ! !x=5;! ! ! !ptr = &x;! ! ! !*ptr = 10;!
- 53. 2D Image Functions • Colors • ofFill(); • ofCircle(100,400,80); • ofSetHexColor(0x000000); ofSetColor(255,0,0,127); • Primitives • ofCircle(100,400,80); • ofRect(400,350,100,100); • ofLine(600,300,800, 250); • ofDrawBitmapString("rectangles", 275,500);
- 55. Circle Grid • Setting the size of the window. • Processing: • size(800, 600, OPENGL); • openFrameworks: • ofSetupOpenGL(&window, 800, 600, OF_WINDOW); • function is called in main() in the file main.cpp.
- 56. Circle Grid • DeclaringVariables • Processing: • Declare the variables you need right after you import the libraries you need. • openFrameworks: • Declare variables in the file testApp.h, after the line void windowResized(int w, int h);.
- 57. Circle Grid • Background Color • Processing: • background(0); will set the background of your sketch to black. You need to call the function inside draw() to draw the background each frame. • openFrameworks: • Call ofBackground(0, 0, 0); once inside the setup() method. openFrameworks will draw the background automatically each frame.You can disable this by calling ofSetBackgroundAuto(false) within setup() in the file testApp.cpp.
- 58. Circle Grid • Drawing Circles • Processing: • after you have set the stroke and fill, use ellipse(50, 50, 20, 20); to draw a circle with a diameter of 20 at (50, 50). • openFrameworks: • you can use ofCircle(50, 50, 10); to draw the same circle.You could also use ofEllipse(50, 50, 20, 20);. If you want to draw a circle with a stroke you will need to call the function to draw the circle two times. Once for the fill and once for the stroke. ofSetColor(255, 255, 255);! ! !ofFill();! ! !ofCircle(50, 50, 20);!
- 59. Graphics Demo - graphicsExample • setup( ) method • draw( ) method
- 60. Drawing Polygons • Must begin and end a shape • ofVertex, ofCurveVertex, ofBezierVertex !ofBeginShape();! ! ! !ofVertex(200,135);! ! ! !ofVertex(15,135);! ! ! !ofVertex(165,25);! ! ! !ofVertex(105,200);! ! ! !ofVertex(50,25);! !ofEndShape();! •
- 61. polygonExample
- 62. ofBoxDemo
- 63. Importing Libraries • Large set of oF addon libraries (> 450) • http://ofxaddons.com/ • Just download library to addons directory, then include library #include “myLibrary.h”! • Sample libraries • ofxOpenCv • ofxVectorGraphics • ofxVectorMath • ofxNetwork • ofvOsc
- 64. Examples • VectorGraphicsExample • 3DModelLoaderExample • Loading 3D models • assimpleExample • 3D animation • openCVExample • Hand segmentation
- 65. Projects
- 68. Nike + Paint With Your Feet +openFrameworks +GPS
- 69. Resources • Main website • http://www.openframeworks.cc/ • Forum • http://forum.openframeworks.cc/ • Addons • http://ofxaddons.com/ • Roxlu s website • http://www.roxlu.com/
- 70. PROJECT SOLI
- 71. Overview • Soli uses radar to detect fine scale finger motion
- 74. Radar Fundamentals • Radar tracks moving objects • Measures response to Radar waves sent from transmitter
- 75. Radar Reflections from Hand • Multiple reflection points
- 76. Signal Processing • Signal received combination of slow and fast time
- 78. Processing Pipeline • From raw hand motion to recognized gestures
- 79. Soli Hardware • Miniaturized Radar
- 80. Signs vs. Actions • Soli recognizes hand actions
- 81. Virtual Tools • Use virtual tool metaphor • Change with distance
- 82. Types of Virtual Tools • asdasf
- 83. Basic Gesture Movement • Easily recognize distinctive gesture motions
- 84. • > 90% accuracy on filtered results (Bayesian Filter) Recognition Results
- 85. Applications • Gesture interaction with objects • Smart watch, car console • Gesture interaction with environment • Furniture, walls • Other applications • Material recognition, Gaming, Object scanning
- 89. Future Research • Radar Sensing • Radar clutter, multi-path reflections, occlusion, etc • Machine Learning • New gesture recognition approaches • Human Factors • Measuring human performance abilities, requirements • Interaction Design • New interaction modalities, metaphors
- 90. Background Reading Lien, Jaime, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. "Soli: ubiquitous gesture sensing with millimeter wave radar." ACM Transactions on Graphics (TOG) 35, no. 4 (2016): 142.