In these slides you can find the basic concepts of natural user interfaces. From the evolution of the classic desktop centered applications to the more intuitive and natural ones.
2. Luigi Oliveto
Master of Science at Politecnico di
Milano
Researcher at Politecnico di Milano
IT Consultant
email: luigi.oliveto@gmail.com
twitter: @LuigiOliveto
linkedin:
https://it.linkedin.com/in/luigioliveto
Nice to Meet You
3. Lots of words…
Ambient Intelligence
Internet of Things
Pervasive Computing
Physical Computing
Ubiquitous computing
Augmented reality
Human-centered computing
Smart device
4. • No more desktop-centered computation, but
distributed computation(“ubiquitous”)
• Objects become more “intelligent” and “smart”
• New information’s model
• New possibility of interaction with information
• Machines fit the human environment instead
of forcing humans to enter theirs
… One concept
15. Natural…experience!
“Natural interaction is defined in terms of
experience: people naturally communicate
through gestures, expressions, movements, and
discover the world by looking around and
manipulating physical stuff.”
Alessandro Valli
https://www.linkedin.com/in/alessandrovalli
17. What is changed?
AFTERBEFORE
• Single device
• Collaborative experience
• More points
• Multiple devices
• Individual experience
• One click
22. • Microsoft "Surface SDK" and "Windows Presentation
Foundation" include API, documentation and tool to
develop multi-touch apps on Windows 7 and Surface
• "Cocoa Touch" is a library to develop software for iPhone,
iPod Touch, e iPad. "Cocoa Touch" is included in iPhone
SDK.
• Android SDK include tool, emulator, debugger and library
to develop App for Android
• Gestureworks (by Ideum) is an interesting Flash multi-touch.
The Gestureworks software allows to develop multiuser and
multi-touch-enabled applications with Adobe Flash.
Multi-Touch Software
26. • Touch-based applications introduce new important
constraints in design of interface and interaction:
• Target’s dimensions must be fit to fingers’ dimension
(min 10mm)
Problems: Usability
28. Gesture’s design: some actions can hide part of information,
too.
Problems: Usability (3)
29. Iceberg Tips: create a wider
invisible area
Adaptive Targets: device
tries to guess next button
pressed by user and zoom it Hell
Tips & Tricks
30. • Don’t assume that people will know that they can touch a
screen.
• Create an “attract state” that demonstrates interactivity while
nobody is using the device
• Make touchable things look touchable
• Design for fingers
• Make sure hands don’t cover up information necessary for
interaction
• Don’t rely on traditional mouse-based interactions, such as
hover & double click
• Use consistent and familiar gestures
Tips & Tricks (2)
31. The Power of Microsoft® PixelSense™
https://www.youtube.com/watch?v=58dsqozft3k
Samsung SUR40 with Microsoft® PixelSense™
https://www.youtube.com/watch?v=kmOku92MlQc
Microsoft Surface wine-tasting demo
http://www.youtube.com/watch?v=Y3KzprGxpZU&feature=related
Patient Consultation Interface Surface Application
http://www.youtube.com/watch?v=Qf0WxOo3O4g&feature=related
Videos
32. Microsoft Surface Application- Barclay's
https://www.youtube.com/watch?v=cBF5BI5H7vs
Playing with Microsoft Surface
http://www.youtube.com/watch?v=SUfRSZppUYs&feature=related
Touch2Much - Microsoft Surface Museum/Gallery Application
http://www.youtube.com/watch?v=DDrCq9632YY
AR.Drone Quadrotor Flight via Microsoft Surface
http://www.youtube.com/watch?v=x1bbT8M6uRs
Video (2)
34. • Camera
• Monitor
• Microphone
QuiQui’s Giant Bounce
Game for children (4 - 9 years)
Game’s paradigm: story telling with animated characters
The actions of child activate specific behaviors of avatar
35. • The EyeToy is a color digital camera device, similar to a
webcam, for the PlayStation 2.
• The technology uses computer vision and Gesture recognition
to process images taken by the camera.
• This allows players to interact with games using motion, color
detection and also sound, through its built-in microphone.
• Limited success due to the low precision
Sony Eye Toy
36. • The console was released on November 19,
2006. About eight days after, 600,000 Wii’s were
reported to be sold.
• It has revolutionized game play and has
impacted society: anyone can play!
Nintendo Wii
37. • The Wii remote, or “Wiimote”,
interacts with a sensor bar by
using accelerometers, infrared
LED’s, and triangulation.
• In general, a player’s Wiimote
movements would determine
their character’s actions. A
gamer would have to move in
order to play.
Wii Technologies
38. • Wii and wiimote comunicate by Bluetooth
• TED 2008: Johnny Lee show how is possible
connect wiimote with a normal pc and use them
in innovative application:
– interactive whiteboard
– 3D head tracking
– finger tracking
• Many others researcher start to use wiimote in
academia projects: http://hackaday.com
Wii Hacks
39. • It is a motion sensing input device by Microsoft for the Xbox
360/XBOX ONE console.
• It enables users to control and interact with the Xbox without
the need to touch a game controller, through a natural user
interface using gestures and spoken commands.
Microsoft Kinect
42. Cursors (hands tracking)
Target an object
Avatars (body tracking)
Interaction with virtual space
• Depend by the tasks
• Important aspect in design of UI
Interaction metaphors
43. The shadow/mirror effect
Shadow Effect:
• I see the back of my avatar
• Problems with Z
movements
Mirror Effect:
• I see the front of my
avatar
• Problem with mapping
left/right movements
52. Power Comes from the Sum
• Any single technology on its own – can create
good experiences
• The sum: This is where the magic is
• Tons of opportunities ahead
53. Some selection criteria…
1° 2° 3° 4° 5°
2 or more users Kinect Intel Leap - -
Full body interaction Kinect - - - -
Hand Gesture Recognition Myo Intel Leap Kinect -
Accuracy Leap Intel Kinect Myo -
Voice command Intel Kinect - - -
Face Tracking Tobii Intel Kinect - -
Commecial use Kinect Intel Tobii Leap Myo
Compatibility Leap MYO Tobii Intel Kinect
54. Costs
Cost Buy Link
Kinect 1 100€ [???]
Kinect 2 150€ http://goo.gl/rskPuD
Real Sense 99$ http://goo.gl/G67TVy
Leap Motion 90€ http://goo.gl/zyVXZZ
Myo 199$ https://goo.gl/ubv6wV
EyeX 99€ http://goo.gl/oGD3Ds
59. Sensor with Camera use IR light and Sunlight can blind the camera!!!
• Check exposition during all day
• Verify that there isn’t direct light on the camera
These devices aren’t a Rugged devices:
• Check temperatures (+3/33°)
• Check humidity
Indoor/Outdoor
64. … where actions performed for some other purpose or unconscious
signs are interpreted in order to influence/improve/facilitate the actors'
future interaction or day-to-day life (from Alan Dix)
• The interaction is not purposeful from the person side, but it is
designed “to happen”
• It “happens” in relation to signs which are not done for that (body
temperature, unconscious reactions such as blink rate, or
unconscious aspects of activities such as typing rate, vocabulary
shifts (e.g. modal verbs), actions done for other purposes, …
• It is designed for people acting
Manage Incidental Interaction