SlideShare a Scribd company logo
1 of 65
Download to read offline
Luigi Oliveto
POLIMI
Natural User Interfaces
Luigi Oliveto
Master of Science at Politecnico di
Milano
Researcher at Politecnico di Milano
IT Consultant
email: luigi.oliveto@gmail.com
twitter: @LuigiOliveto
linkedin:
https://it.linkedin.com/in/luigioliveto
Nice to Meet You
Lots of words…
Ambient Intelligence
Internet of Things
Pervasive Computing
Physical Computing
Ubiquitous computing
Augmented reality
Human-centered computing
Smart device
• No more desktop-centered computation, but
distributed computation(“ubiquitous”)
• Objects become more “intelligent” and “smart”
• New information’s model
• New possibility of interaction with information
• Machines fit the human environment instead
of forcing humans to enter theirs
… One concept
Interface Evolution
Command Line Interface
CLI
Graphical User Interface
GUI
NUI
Natural User Interface
Natural User Interface
Computer
Vision
Facial
Recognition
Spatial
Recognition
Augmented
Reality
Gesture
Sensing
Audio
Recognition
Voice
Comma
nd
Natural
Speech
Ambien
t Noise
Touch
Single
Touch
Multi-
Touch
Pen
Input
Sensors
Geospatial
Sensing
Accelerometers
Biometrics
Ambient
Light
Brain
Waves
Mind
control
Mood
Recogniti
on
Multi-Touch interfaces
make a withdrawal…
megawatts86
Airplane check-in…
mrkathika
Shopping…
pin add
What’s happened?
Natural…experience!
“Natural interaction is defined in terms of
experience: people naturally communicate
through gestures, expressions, movements, and
discover the world by looking around and
manipulating physical stuff.”
Alessandro Valli
https://www.linkedin.com/in/alessandrovalli
Natural User Interfaces
What is changed?
AFTERBEFORE
• Single device
• Collaborative experience
• More points
• Multiple devices
• Individual experience
• One click
• Sensible to touch surface (touch screen)
Multi-Touch Hardware
RESISTIVE CAPACITIVE
• Camera-based technology:
Laser Line Plan
Frustrated Total Internal Reflection
Diffused illumination
Pixel Sense
Multi-Touch Hardware (2)
Multi-Touch Devices
Smartphone Tablet Monitor
Multi-Touch Devices (2)
Tangible Table / Wall
• Microsoft "Surface SDK" and "Windows Presentation
Foundation" include API, documentation and tool to
develop multi-touch apps on Windows 7 and Surface
• "Cocoa Touch" is a library to develop software for iPhone,
iPod Touch, e iPad. "Cocoa Touch" is included in iPhone
SDK.
• Android SDK include tool, emulator, debugger and library
to develop App for Android
• Gestureworks (by Ideum) is an interesting Flash multi-touch.
The Gestureworks software allows to develop multiuser and
multi-touch-enabled applications with Adobe Flash.
Multi-Touch Software
http://www.lukew.com/touch/TouchGestureGuide.pdf
Common Gestures
• Nails
• Gloves
• Dirty fingers
• Gestures (are they so easy?)
• Accuracy
Problems: Input
Gorilla arm problem
Problems: Accessibility
• Touch-based applications introduce new important
constraints in design of interface and interaction:
• Target’s dimensions must be fit to fingers’ dimension
(min 10mm)
Problems: Usability
Labels’ position: hands and fingers can hide information
Problems: Usability (2)
Gesture’s design: some actions can hide part of information,
too.
Problems: Usability (3)
Iceberg Tips: create a wider
invisible area
Adaptive Targets: device
tries to guess next button
pressed by user and zoom it Hell
Tips & Tricks
• Don’t assume that people will know that they can touch a
screen.
• Create an “attract state” that demonstrates interactivity while
nobody is using the device
• Make touchable things look touchable
• Design for fingers
• Make sure hands don’t cover up information necessary for
interaction
• Don’t rely on traditional mouse-based interactions, such as
hover & double click
• Use consistent and familiar gestures
Tips & Tricks (2)
The Power of Microsoft® PixelSense™
https://www.youtube.com/watch?v=58dsqozft3k
Samsung SUR40 with Microsoft® PixelSense™
https://www.youtube.com/watch?v=kmOku92MlQc
Microsoft Surface wine-tasting demo
http://www.youtube.com/watch?v=Y3KzprGxpZU&feature=related
Patient Consultation Interface Surface Application
http://www.youtube.com/watch?v=Qf0WxOo3O4g&feature=related
Videos
Microsoft Surface Application- Barclay's
https://www.youtube.com/watch?v=cBF5BI5H7vs
Playing with Microsoft Surface
http://www.youtube.com/watch?v=SUfRSZppUYs&feature=related
Touch2Much - Microsoft Surface Museum/Gallery Application
http://www.youtube.com/watch?v=DDrCq9632YY
AR.Drone Quadrotor Flight via Microsoft Surface
http://www.youtube.com/watch?v=x1bbT8M6uRs
Video (2)
Touchless interfaces
• Camera
• Monitor
• Microphone
QuiQui’s Giant Bounce
Game for children (4 - 9 years)
Game’s paradigm: story telling with animated characters
The actions of child activate specific behaviors of avatar
• The EyeToy is a color digital camera device, similar to a
webcam, for the PlayStation 2.
• The technology uses computer vision and Gesture recognition
to process images taken by the camera.
• This allows players to interact with games using motion, color
detection and also sound, through its built-in microphone.
• Limited success due to the low precision
Sony Eye Toy
• The console was released on November 19,
2006. About eight days after, 600,000 Wii’s were
reported to be sold.
• It has revolutionized game play and has
impacted society: anyone can play!
Nintendo Wii
• The Wii remote, or “Wiimote”,
interacts with a sensor bar by
using accelerometers, infrared
LED’s, and triangulation.
• In general, a player’s Wiimote
movements would determine
their character’s actions. A
gamer would have to move in
order to play.
Wii Technologies
• Wii and wiimote comunicate by Bluetooth
• TED 2008: Johnny Lee show how is possible
connect wiimote with a normal pc and use them
in innovative application:
– interactive whiteboard
– 3D head tracking
– finger tracking
• Many others researcher start to use wiimote in
academia projects: http://hackaday.com
Wii Hacks
• It is a motion sensing input device by Microsoft for the Xbox
360/XBOX ONE console.
• It enables users to control and interact with the Xbox without
the need to touch a game controller, through a natural user
interface using gestures and spoken commands.
Microsoft Kinect
Immersive user experience
Kinect’s magic
“Any sufficiently advanced technology is indistinguishable
from magic”
(Arthur C. Clarke)
Provided Data
Cursors (hands tracking)
Target an object
Avatars (body tracking)
Interaction with virtual space
• Depend by the tasks
• Important aspect in design of UI
Interaction metaphors
The shadow/mirror effect
Shadow Effect:
• I see the back of my avatar
• Problems with Z
movements
Mirror Effect:
• I see the front of my
avatar
• Problem with mapping
left/right movements
User Interaction
Game mindset ≠ UI mindset
Challenging = fun Challenging = easy and effective
Gesture semantically fits user task
Abstract Meaningful
Intel Real Sense
4 basic types of input
Categories of
Input
Capabilities Features
Hands • Hand and Finger
Tracking
• Gesture Recognition
• 22-point Hand and Finger Tracking
• 9 static and dynamic mid-air gestures
Face • Face Detection and
Tracking
• Multiple Face Detection and tracking
• 78-point Landmark Detection (facial features)
• Emotion Recognition (7 emotions, coming post-Beta)
• Pulse Estimation
• Face Recognition (Coming post-beta)
Speech • Speech Recognition • Command and Control
• Dictation
• Text to Speech
Environment • Segmentation
• 3D Scanning
• Augmented Reality
• Background Removal
• 3D Object / Face / Room Scanning (Coming post-beta)
• 2D/3D Object Tracking
• Scene Perception (coming post-beta)
https://www.youtube.com/watch?v=_d6KuiuteIA
https://airspace.leapmotion.com/
Leap Motion
Leap Motion - Field of View
150° - Long Side
120° - Short Side
Max 60 cm above the controller
Max 60 cm wide on each side
Thalmic Labs Myo
https://www.myo.com/
http://www.tobii.com/en/eye-experience/
Tobii EyeX
Power Comes from the Sum
• Any single technology on its own – can create
good experiences
• The sum: This is where the magic is
• Tons of opportunities ahead
Some selection criteria…
1° 2° 3° 4° 5°
2 or more users Kinect Intel Leap - -
Full body interaction Kinect - - - -
Hand Gesture Recognition Myo Intel Leap Kinect -
Accuracy Leap Intel Kinect Myo -
Voice command Intel Kinect - - -
Face Tracking Tobii Intel Kinect - -
Commecial use Kinect Intel Tobii Leap Myo
Compatibility Leap MYO Tobii Intel Kinect
Costs
Cost Buy Link
Kinect 1 100€ [???]
Kinect 2 150€ http://goo.gl/rskPuD
Real Sense 99$ http://goo.gl/G67TVy
Leap Motion 90€ http://goo.gl/zyVXZZ
Myo 199$ https://goo.gl/ubv6wV
EyeX 99€ http://goo.gl/oGD3Ds
Leap, Real Sense, Kinect ranges
2,5 cm 60 cm 2 m 4 m
Final considerations
Capture Volumes
The user is performing a hand gesture outside of the capture
volume. The camera will not see this gesture
Evaluate different settings and environment
Sensor with Camera use IR light and Sunlight can blind the camera!!!
• Check exposition during all day
• Verify that there isn’t direct light on the camera
These devices aren’t a Rugged devices:
• Check temperatures (+3/33°)
• Check humidity
Indoor/Outdoor
Comfortable positions
Your users are not GORILLAS!!!
User posture may affect design of a gesture
Input variability
Feedback, feedback, feedback,…
View of user:
• User Viewport
• User Overlay
… where actions performed for some other purpose or unconscious
signs are interpreted in order to influence/improve/facilitate the actors'
future interaction or day-to-day life (from Alan Dix)
• The interaction is not purposeful from the person side, but it is
designed “to happen”
• It “happens” in relation to signs which are not done for that (body
temperature, unconscious reactions such as blink rate, or
unconscious aspects of activities such as typing rate, vocabulary
shifts (e.g. modal verbs), actions done for other purposes, …
• It is designed for people acting
Manage Incidental Interaction
Luigi Oliveto
POLIMI
Natural User Interfaces

More Related Content

What's hot

Inter vehicle communication
Inter vehicle communicationInter vehicle communication
Inter vehicle communicationR prasad
 
Intermediate: 5G and Extended Reality (XR)
Intermediate: 5G and Extended Reality (XR)Intermediate: 5G and Extended Reality (XR)
Intermediate: 5G and Extended Reality (XR)3G4G
 
IoT services in the automotive sector
IoT services in the automotive sectorIoT services in the automotive sector
IoT services in the automotive sectorPRIME
 
seminar report on smart glasses
seminar report on smart glasses seminar report on smart glasses
seminar report on smart glasses Nipun Agrawal
 
Metaverse: A Convergence of Realities
Metaverse: A Convergence of RealitiesMetaverse: A Convergence of Realities
Metaverse: A Convergence of RealitiesTom
 
HCI : Virtual reality (AUGMENTED & MIXED REALITY)
HCI : Virtual reality (AUGMENTED & MIXED REALITY)HCI : Virtual reality (AUGMENTED & MIXED REALITY)
HCI : Virtual reality (AUGMENTED & MIXED REALITY)Yasmin Zahir
 
IoT Applications and Networks
IoT Applications and NetworksIoT Applications and Networks
IoT Applications and NetworksAbdulrahman Fady
 
IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...
IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...
IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...Edureka!
 
An IOT based Smart Traffic Management System
An IOT based Smart Traffic Management SystemAn IOT based Smart Traffic Management System
An IOT based Smart Traffic Management SystemAI Publications
 
Augmented Reality - PPT
Augmented Reality - PPTAugmented Reality - PPT
Augmented Reality - PPTRahul John
 
How Virtual Reality & Augmented Reality Technology is Disrupting The Business...
How Virtual Reality & Augmented Reality Technology is Disrupting The Business...How Virtual Reality & Augmented Reality Technology is Disrupting The Business...
How Virtual Reality & Augmented Reality Technology is Disrupting The Business...Stanford University
 
Augmented Reality
Augmented RealityAugmented Reality
Augmented RealityAjay Sankar
 
Internet of Things (IOT)
Internet of Things (IOT)Internet of Things (IOT)
Internet of Things (IOT)Kunal Adhikari
 
Artificial intelligence and IoT
Artificial intelligence and IoTArtificial intelligence and IoT
Artificial intelligence and IoTVeselin Pizurica
 
Augmented reality ppt
Augmented reality pptAugmented reality ppt
Augmented reality pptDark Side
 

What's hot (20)

Inter vehicle communication
Inter vehicle communicationInter vehicle communication
Inter vehicle communication
 
Intermediate: 5G and Extended Reality (XR)
Intermediate: 5G and Extended Reality (XR)Intermediate: 5G and Extended Reality (XR)
Intermediate: 5G and Extended Reality (XR)
 
IoT services in the automotive sector
IoT services in the automotive sectorIoT services in the automotive sector
IoT services in the automotive sector
 
seminar report on smart glasses
seminar report on smart glasses seminar report on smart glasses
seminar report on smart glasses
 
Metaverse: A Convergence of Realities
Metaverse: A Convergence of RealitiesMetaverse: A Convergence of Realities
Metaverse: A Convergence of Realities
 
HCI : Virtual reality (AUGMENTED & MIXED REALITY)
HCI : Virtual reality (AUGMENTED & MIXED REALITY)HCI : Virtual reality (AUGMENTED & MIXED REALITY)
HCI : Virtual reality (AUGMENTED & MIXED REALITY)
 
3 d password
3 d password3 d password
3 d password
 
Metaverse Infographics
Metaverse InfographicsMetaverse Infographics
Metaverse Infographics
 
What is a metaverse?
What is a metaverse?What is a metaverse?
What is a metaverse?
 
Technical seminar report
Technical seminar reportTechnical seminar report
Technical seminar report
 
IoT Applications and Networks
IoT Applications and NetworksIoT Applications and Networks
IoT Applications and Networks
 
IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...
IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...
IoT Tutorial for Beginners | Internet of Things (IoT) | IoT Training | IoT Te...
 
An IOT based Smart Traffic Management System
An IOT based Smart Traffic Management SystemAn IOT based Smart Traffic Management System
An IOT based Smart Traffic Management System
 
Augmented Reality - PPT
Augmented Reality - PPTAugmented Reality - PPT
Augmented Reality - PPT
 
How Virtual Reality & Augmented Reality Technology is Disrupting The Business...
How Virtual Reality & Augmented Reality Technology is Disrupting The Business...How Virtual Reality & Augmented Reality Technology is Disrupting The Business...
How Virtual Reality & Augmented Reality Technology is Disrupting The Business...
 
Augmented Reality
Augmented RealityAugmented Reality
Augmented Reality
 
Internet of Things (IOT)
Internet of Things (IOT)Internet of Things (IOT)
Internet of Things (IOT)
 
Artificial intelligence and IoT
Artificial intelligence and IoTArtificial intelligence and IoT
Artificial intelligence and IoT
 
Smartphone based ADAS
Smartphone based ADASSmartphone based ADAS
Smartphone based ADAS
 
Augmented reality ppt
Augmented reality pptAugmented reality ppt
Augmented reality ppt
 

Similar to Natural User Interfaces

Interaction modalities, technologies and tools for interactive art
Interaction modalities, technologies and tools for interactive artInteraction modalities, technologies and tools for interactive art
Interaction modalities, technologies and tools for interactive artJorge Cardoso
 
touchless touchscreen technology abstract
touchless touchscreen technology abstracttouchless touchscreen technology abstract
touchless touchscreen technology abstract14SYITVaishnaviDamar
 
Touchless Touchscreen Technology
Touchless Touchscreen TechnologyTouchless Touchscreen Technology
Touchless Touchscreen TechnologyAkshay Vasava
 
Touchless touchscreen
Touchless touchscreenTouchless touchscreen
Touchless touchscreenNaga Dinesh
 
Research on Detecting Hand Gesture
Research on Detecting Hand GestureResearch on Detecting Hand Gesture
Research on Detecting Hand GestureIRJET Journal
 
Immersive technologies.pptx
Immersive technologies.pptxImmersive technologies.pptx
Immersive technologies.pptxAnandSri5
 
Leap Motion seminar presentation ppt
Leap Motion seminar presentation pptLeap Motion seminar presentation ppt
Leap Motion seminar presentation pptAdarsh Kottur
 
Dev09 – la battaglia del touchless
Dev09 – la battaglia del touchlessDev09 – la battaglia del touchless
Dev09 – la battaglia del touchlessClemente Giorio
 
2022 COMP4010 Lecture2: Perception
2022 COMP4010 Lecture2: Perception2022 COMP4010 Lecture2: Perception
2022 COMP4010 Lecture2: PerceptionMark Billinghurst
 
Mixed reality the second generation is all about ux
Mixed reality   the second generation is all about uxMixed reality   the second generation is all about ux
Mixed reality the second generation is all about uxCameron Vetter
 
Touchless Touchscreen
Touchless TouchscreenTouchless Touchscreen
Touchless Touchscreenvishnu murthy
 
Mobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research DirectionsMobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research DirectionsMark Billinghurst
 
Touchless technology
Touchless technologyTouchless technology
Touchless technologyInternet User
 
Touchless technology
Touchless technologyTouchless technology
Touchless technologyInternet User
 
Gam02 kinect1, kinect2
Gam02   kinect1, kinect2Gam02   kinect1, kinect2
Gam02 kinect1, kinect2DotNetCampus
 
IoT Day Italy - Mixed Reality & IoT
IoT Day Italy - Mixed Reality & IoTIoT Day Italy - Mixed Reality & IoT
IoT Day Italy - Mixed Reality & IoTClemente Giorio
 

Similar to Natural User Interfaces (20)

Iotconf2015 mn pdf
Iotconf2015 mn pdfIotconf2015 mn pdf
Iotconf2015 mn pdf
 
Touchless touch screen
Touchless touch screenTouchless touch screen
Touchless touch screen
 
Interaction modalities, technologies and tools for interactive art
Interaction modalities, technologies and tools for interactive artInteraction modalities, technologies and tools for interactive art
Interaction modalities, technologies and tools for interactive art
 
touchless touchscreen technology abstract
touchless touchscreen technology abstracttouchless touchscreen technology abstract
touchless touchscreen technology abstract
 
Touchless Touchscreen Technology
Touchless Touchscreen TechnologyTouchless Touchscreen Technology
Touchless Touchscreen Technology
 
Touchless touchscreen
Touchless touchscreenTouchless touchscreen
Touchless touchscreen
 
Touchless Touchscreen
Touchless TouchscreenTouchless Touchscreen
Touchless Touchscreen
 
Research on Detecting Hand Gesture
Research on Detecting Hand GestureResearch on Detecting Hand Gesture
Research on Detecting Hand Gesture
 
Immersive technologies.pptx
Immersive technologies.pptxImmersive technologies.pptx
Immersive technologies.pptx
 
Leap Motion seminar presentation ppt
Leap Motion seminar presentation pptLeap Motion seminar presentation ppt
Leap Motion seminar presentation ppt
 
Dev09 – la battaglia del touchless
Dev09 – la battaglia del touchlessDev09 – la battaglia del touchless
Dev09 – la battaglia del touchless
 
2022 COMP4010 Lecture2: Perception
2022 COMP4010 Lecture2: Perception2022 COMP4010 Lecture2: Perception
2022 COMP4010 Lecture2: Perception
 
Mixed reality the second generation is all about ux
Mixed reality   the second generation is all about uxMixed reality   the second generation is all about ux
Mixed reality the second generation is all about ux
 
Touchless Touchscreen
Touchless TouchscreenTouchless Touchscreen
Touchless Touchscreen
 
Mobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research DirectionsMobile AR Lecture 10 - Research Directions
Mobile AR Lecture 10 - Research Directions
 
Touchless technology
Touchless technologyTouchless technology
Touchless technology
 
Touchless technology
Touchless technologyTouchless technology
Touchless technology
 
HoloLens
HoloLensHoloLens
HoloLens
 
Gam02 kinect1, kinect2
Gam02   kinect1, kinect2Gam02   kinect1, kinect2
Gam02 kinect1, kinect2
 
IoT Day Italy - Mixed Reality & IoT
IoT Day Italy - Mixed Reality & IoTIoT Day Italy - Mixed Reality & IoT
IoT Day Italy - Mixed Reality & IoT
 

Recently uploaded

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Alexander Turgeon
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdfPaige Cruz
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"DianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 

Natural User Interfaces

  • 2. Luigi Oliveto Master of Science at Politecnico di Milano Researcher at Politecnico di Milano IT Consultant email: luigi.oliveto@gmail.com twitter: @LuigiOliveto linkedin: https://it.linkedin.com/in/luigioliveto Nice to Meet You
  • 3. Lots of words… Ambient Intelligence Internet of Things Pervasive Computing Physical Computing Ubiquitous computing Augmented reality Human-centered computing Smart device
  • 4. • No more desktop-centered computation, but distributed computation(“ubiquitous”) • Objects become more “intelligent” and “smart” • New information’s model • New possibility of interaction with information • Machines fit the human environment instead of forcing humans to enter theirs … One concept
  • 5. Interface Evolution Command Line Interface CLI Graphical User Interface GUI NUI Natural User Interface
  • 6. Natural User Interface Computer Vision Facial Recognition Spatial Recognition Augmented Reality Gesture Sensing Audio Recognition Voice Comma nd Natural Speech Ambien t Noise Touch Single Touch Multi- Touch Pen Input Sensors Geospatial Sensing Accelerometers Biometrics Ambient Light Brain Waves Mind control Mood Recogniti on
  • 15. Natural…experience! “Natural interaction is defined in terms of experience: people naturally communicate through gestures, expressions, movements, and discover the world by looking around and manipulating physical stuff.” Alessandro Valli https://www.linkedin.com/in/alessandrovalli
  • 17. What is changed? AFTERBEFORE • Single device • Collaborative experience • More points • Multiple devices • Individual experience • One click
  • 18. • Sensible to touch surface (touch screen) Multi-Touch Hardware RESISTIVE CAPACITIVE
  • 19. • Camera-based technology: Laser Line Plan Frustrated Total Internal Reflection Diffused illumination Pixel Sense Multi-Touch Hardware (2)
  • 22. • Microsoft "Surface SDK" and "Windows Presentation Foundation" include API, documentation and tool to develop multi-touch apps on Windows 7 and Surface • "Cocoa Touch" is a library to develop software for iPhone, iPod Touch, e iPad. "Cocoa Touch" is included in iPhone SDK. • Android SDK include tool, emulator, debugger and library to develop App for Android • Gestureworks (by Ideum) is an interesting Flash multi-touch. The Gestureworks software allows to develop multiuser and multi-touch-enabled applications with Adobe Flash. Multi-Touch Software
  • 24. • Nails • Gloves • Dirty fingers • Gestures (are they so easy?) • Accuracy Problems: Input
  • 26. • Touch-based applications introduce new important constraints in design of interface and interaction: • Target’s dimensions must be fit to fingers’ dimension (min 10mm) Problems: Usability
  • 27. Labels’ position: hands and fingers can hide information Problems: Usability (2)
  • 28. Gesture’s design: some actions can hide part of information, too. Problems: Usability (3)
  • 29. Iceberg Tips: create a wider invisible area Adaptive Targets: device tries to guess next button pressed by user and zoom it Hell Tips & Tricks
  • 30. • Don’t assume that people will know that they can touch a screen. • Create an “attract state” that demonstrates interactivity while nobody is using the device • Make touchable things look touchable • Design for fingers • Make sure hands don’t cover up information necessary for interaction • Don’t rely on traditional mouse-based interactions, such as hover & double click • Use consistent and familiar gestures Tips & Tricks (2)
  • 31. The Power of Microsoft® PixelSense™ https://www.youtube.com/watch?v=58dsqozft3k Samsung SUR40 with Microsoft® PixelSense™ https://www.youtube.com/watch?v=kmOku92MlQc Microsoft Surface wine-tasting demo http://www.youtube.com/watch?v=Y3KzprGxpZU&feature=related Patient Consultation Interface Surface Application http://www.youtube.com/watch?v=Qf0WxOo3O4g&feature=related Videos
  • 32. Microsoft Surface Application- Barclay's https://www.youtube.com/watch?v=cBF5BI5H7vs Playing with Microsoft Surface http://www.youtube.com/watch?v=SUfRSZppUYs&feature=related Touch2Much - Microsoft Surface Museum/Gallery Application http://www.youtube.com/watch?v=DDrCq9632YY AR.Drone Quadrotor Flight via Microsoft Surface http://www.youtube.com/watch?v=x1bbT8M6uRs Video (2)
  • 34. • Camera • Monitor • Microphone QuiQui’s Giant Bounce Game for children (4 - 9 years) Game’s paradigm: story telling with animated characters The actions of child activate specific behaviors of avatar
  • 35. • The EyeToy is a color digital camera device, similar to a webcam, for the PlayStation 2. • The technology uses computer vision and Gesture recognition to process images taken by the camera. • This allows players to interact with games using motion, color detection and also sound, through its built-in microphone. • Limited success due to the low precision Sony Eye Toy
  • 36. • The console was released on November 19, 2006. About eight days after, 600,000 Wii’s were reported to be sold. • It has revolutionized game play and has impacted society: anyone can play! Nintendo Wii
  • 37. • The Wii remote, or “Wiimote”, interacts with a sensor bar by using accelerometers, infrared LED’s, and triangulation. • In general, a player’s Wiimote movements would determine their character’s actions. A gamer would have to move in order to play. Wii Technologies
  • 38. • Wii and wiimote comunicate by Bluetooth • TED 2008: Johnny Lee show how is possible connect wiimote with a normal pc and use them in innovative application: – interactive whiteboard – 3D head tracking – finger tracking • Many others researcher start to use wiimote in academia projects: http://hackaday.com Wii Hacks
  • 39. • It is a motion sensing input device by Microsoft for the Xbox 360/XBOX ONE console. • It enables users to control and interact with the Xbox without the need to touch a game controller, through a natural user interface using gestures and spoken commands. Microsoft Kinect
  • 40. Immersive user experience Kinect’s magic “Any sufficiently advanced technology is indistinguishable from magic” (Arthur C. Clarke)
  • 42. Cursors (hands tracking) Target an object Avatars (body tracking) Interaction with virtual space • Depend by the tasks • Important aspect in design of UI Interaction metaphors
  • 43. The shadow/mirror effect Shadow Effect: • I see the back of my avatar • Problems with Z movements Mirror Effect: • I see the front of my avatar • Problem with mapping left/right movements
  • 44. User Interaction Game mindset ≠ UI mindset Challenging = fun Challenging = easy and effective
  • 45. Gesture semantically fits user task Abstract Meaningful
  • 47. 4 basic types of input Categories of Input Capabilities Features Hands • Hand and Finger Tracking • Gesture Recognition • 22-point Hand and Finger Tracking • 9 static and dynamic mid-air gestures Face • Face Detection and Tracking • Multiple Face Detection and tracking • 78-point Landmark Detection (facial features) • Emotion Recognition (7 emotions, coming post-Beta) • Pulse Estimation • Face Recognition (Coming post-beta) Speech • Speech Recognition • Command and Control • Dictation • Text to Speech Environment • Segmentation • 3D Scanning • Augmented Reality • Background Removal • 3D Object / Face / Room Scanning (Coming post-beta) • 2D/3D Object Tracking • Scene Perception (coming post-beta)
  • 49. Leap Motion - Field of View 150° - Long Side 120° - Short Side Max 60 cm above the controller Max 60 cm wide on each side
  • 52. Power Comes from the Sum • Any single technology on its own – can create good experiences • The sum: This is where the magic is • Tons of opportunities ahead
  • 53. Some selection criteria… 1° 2° 3° 4° 5° 2 or more users Kinect Intel Leap - - Full body interaction Kinect - - - - Hand Gesture Recognition Myo Intel Leap Kinect - Accuracy Leap Intel Kinect Myo - Voice command Intel Kinect - - - Face Tracking Tobii Intel Kinect - - Commecial use Kinect Intel Tobii Leap Myo Compatibility Leap MYO Tobii Intel Kinect
  • 54. Costs Cost Buy Link Kinect 1 100€ [???] Kinect 2 150€ http://goo.gl/rskPuD Real Sense 99$ http://goo.gl/G67TVy Leap Motion 90€ http://goo.gl/zyVXZZ Myo 199$ https://goo.gl/ubv6wV EyeX 99€ http://goo.gl/oGD3Ds
  • 55. Leap, Real Sense, Kinect ranges 2,5 cm 60 cm 2 m 4 m
  • 57. Capture Volumes The user is performing a hand gesture outside of the capture volume. The camera will not see this gesture
  • 58. Evaluate different settings and environment
  • 59. Sensor with Camera use IR light and Sunlight can blind the camera!!! • Check exposition during all day • Verify that there isn’t direct light on the camera These devices aren’t a Rugged devices: • Check temperatures (+3/33°) • Check humidity Indoor/Outdoor
  • 60. Comfortable positions Your users are not GORILLAS!!!
  • 61. User posture may affect design of a gesture
  • 63. Feedback, feedback, feedback,… View of user: • User Viewport • User Overlay
  • 64. … where actions performed for some other purpose or unconscious signs are interpreted in order to influence/improve/facilitate the actors' future interaction or day-to-day life (from Alan Dix) • The interaction is not purposeful from the person side, but it is designed “to happen” • It “happens” in relation to signs which are not done for that (body temperature, unconscious reactions such as blink rate, or unconscious aspects of activities such as typing rate, vocabulary shifts (e.g. modal verbs), actions done for other purposes, … • It is designed for people acting Manage Incidental Interaction