This talk was presented at the 2015 HCIL Annual Symposium put on by the University of Maryland's Human Computer Interaction Lab.
We introduce the preliminary design of a novel vision-augmented touch system called HandSight intended to support activities of daily living (ADLs) by sensing and feeding back non-tactile information about the physical world as it is touched. Though we are interested in supporting a range of ADL applications, here we focus specifically on reading printed text.
The recent miniaturization of cameras has enabled finger- based reading approaches, which provide blind and visually impaired readers with access to printed materials. Compared to handheld text scanners such as mobile phone applications, mounting a tiny camera on the user’s own finger has the potential to mitigate camera framing issues, enable a blind reader to better understand the spatial layout of a document, and provide better control over reading pace. A finger-based approach, however, also introduces the need to guide the reader in physically navigating a document, such as tracing along lines of text. While previous work has proposed audio and haptic directional finger guidance for this purpose, user studies of finger-based reading have been limited to 3 or 4 participants.
To further investigate the feasibility of a finger-based sensing and feedback system for reading printed text, we conducted two lab studies: (i) a comparison of audio and haptic directional finger guidance with 20 blind participants using an iPad-based testbed, and (ii) a smaller proof-of- concept study with 4 blind participants using a preliminary wearable prototype called HandSight.
Findings from the first study show similar performance between haptic and audio directional guidance, although audio may offer an accuracy advantage for line tracing. Subjective feedback also highlights tradeoffs between the two types of guidance, such as the interference of audio guidance with speech output and the potential for desensitization to haptic guidance. While several participants appreciated the direct access to layout information provided by finger-based exploration, important concerns also arose about ease of use and the amount of concentration required.
We close with a discussion on the advantages of finger-based reading for blind readers, its feasibility, and potential design improvements to the HandSight prototype.
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
HandSight: The Design and Evaluation of a Finger-Mounted Camera and Feedback System to Enable Blind Persons to Read Printed Text
1. HandSight: The Design and Evaluation of a
Finger-Mounted Camera and Feedback System
to Enable Blind Persons to Read Printed Text
Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Yumeng Wang2,
Leah Findlater3, Rama Chellappa4, David A. Ross5, Jon E. Froehlich1
University of Maryland: Computer Science1, Architecture2, Information Studies3, Electrical Engineering4,
Atlanta VA R&D Center of Excellence in Vision and Neurocognitive Rehabilitation (CVNR)5
2. There are 285 million people with visual impairments
worldwide—including 39 million who are blind
3. There are 285 million people with visual impairments
worldwide—including 39 million who are blind
Visual impairments can negatively impact a
person’s ability to perform activities of daily living
(ADLs)
4. Previous research has explored using mobile cameras
with computer vision for at-a-distance tasks
7. Our Approach: HandSight
a vision augmented touch system
Tiny CMOS cameras,
micro-haptic actuators mounted
on one or more fingers
8. Our Approach: HandSight
a vision augmented touch system
Tiny CMOS cameras,
micro-haptic actuators mounted
on one or more fingers
SmartWatch for power,
processing, speech output
9. Our Approach: HandSight
a vision augmented touch system
Tiny CMOS cameras,
micro-haptic actuators mounted
on one or more fingers
SmartWatch for power,
processing, speech output
Computer vision, machine
learning algorithms to support
fingertip-based sensing
10. Our Vision for HandSight
Augment the sense of
touch with a device that
is unobtrusive, always
available, and that
allows seamless switching
between the physical world
and HandSight enabled
applications.
24. Touch-based Reading
• Potential advantages
• Immediate access, no alignment issues
• Spatial awareness and interactions—may improve
comprehension and document understanding
25. Touch-based Reading
• Potential advantages
• Immediate access, no alignment issues
• Spatial awareness and interactions—may improve
comprehension and document understanding
FingerReader
26. Touch-based Reading
• Problem: How can we guide a blind person’s finger?
• Following a line of text
• Locating the start of the next line
• Exploring a document’s layout
27. Continuous audio cues indicate content beneath the user’s finger
System Design: Exploration Mode
28. Continuous audio cues indicate content beneath the user’s finger
System Design: Exploration Mode
29. Audio or haptic directional guidance helps users to stay on the line or locate
the start of new line. Audio cues indicate start/end of a line or paragraph.
System Design: Reading Mode
30. Audio or haptic directional guidance helps users to stay on the line or locate
the start of new line. Audio cues indicate start/end of a line or paragraph.
System Design: Reading Mode
34. Study I Method
Simulated reading experience using an iPad
20 participants
Average Age 48.3 (SD=11.7, Range=26-67)
Gender 12 Male, 8 Female
Vision Level 11 Totally Blind, 9 Light Sensitive
35. Study I Method
Simulated reading experience using an iPad
20 participants
Within-subjects, 2 conditions: audio and haptic
36. Study I Method
Simulated reading experience using an iPad
20 participants
Within-subjects, 2 conditions: audio and haptic
Participants read 2 documents for each condition
plain magazine
37. Study I Method
Simulated reading experience using an iPad
20 participants
Within-subject, 2 conditions: audio and haptic
Participants read 2 documents for each condition
Analysis: reading speed and accuracy,
comprehension, subjective feedback
audio haptic
38. Study I Findings
audio haptic
Audio vs. Haptic:
Similar performance. Audio was significantly more accurate
for line tracing on the magazine documents (p = 0.018)
39. Study I Findings
Audio vs. Haptic:
Similar performance. Audio was significantly more accurate
for line tracing on the magazine documents (p = 0.018)
Preferences split (12 haptic, 7 audio, 1 equal preference)
40. Study I Findings
Audio vs. Haptic:
Similar performance. Audio was significantly more accurate
for line tracing on the magazine documents (p = 0.018)
Preferences split (12 haptic, 7 audio, 1 equal preference)
Participant comments: audio interferes with speech,
desensitization to haptics over time
48. Questions?
Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Yumeng Wang2,
Leah Findlater3, Rama Chellappa4, David A. Ross5, Jon E. Froehlich1
University of Maryland: Computer Science1, Architecture2, Information Studies3, Electrical Engineering4,
Atlanta VA R&D Center of Excellence in Vision and Neurocognitive Rehabilitation (CVNR)5
Thank you to our participants and the Maryland State Library for the Blind and Physically Handicapped.
This research was funded by the Department of Defense.
Images from Flickr used under Creative Commons license.
Contact: lstearns@umd.edu | jonf@cs.umd.edu
Notas del editor
Hello everyone. My name is Lee Stearns, and today I’m going to be presenting my group’s research on a system we call HandSight, which we’ve designed to help people with visual impairments perform everyday tasks like reading printed text.
According to a 2010 study, there are 285 million people living with visual impairments worldwide, including 39 million who are blind. In the United States and other countries with populations that are aging, that number is only increasing.
Visual impairments can reduce a person’s ability to perform activities of daily living, like reading or getting dressed.
There has been a lot of previous research that’s looked at using mobile cameras and computer vision to help people with visual impairments with tasks that are performed at a distance, like navigation or face recognition.
But very few have tried to support the direct touch-based interactions that make up so many of the activities we perform every day.
Without sight, touch is one of the primary means of acquiring information about the physical world. Many people with visual impairments have a highly attuned sense of touch.
Our approach augments the sense of touch, sensing and feeding back non-tactile information about the physical world as it is touched.
We envision a system called HandSight that’s made up of tiny cameras and haptics integrated directly with the user’s fingers.
It will be built into a smart-watch platform for power, processing, and speech output,
and use computer vision and machine learning algorithms to support fingertip-based sensing.
By putting the camera and feedback directly on the user’s fingers and augmenting the sense of touch, our hope is that we can create new and more intuitive assistive applications. Our goal is to offer a device that is unobtrusive, always available, and that allows seamless switching between the physical world and the applications that HandSight supports.
There are a wide range of applications for this technology. One of the first things that jumps to mind is to use it to read printed text in newspapers, bills, menus… not everything available digitally yet. You could also use it to explore maps and drawings, to get a better idea of their spatial layouts. Imagine moving your fingers over the page and being about to feel the texture of the lines, exploring the shapes and relative positions. Another application is getting dressed in the morning, quickly identifying colors and visual textures that you might not be able to distinguish by the feel alone. Even something as simple as shopping for groceries – imagine picking up a bell pepper in the store and being able to instantly identify by touch whether it’s red or green.
All images licensed for use under Creative Commons
USA image: Wikimedia Commons
Clothing image: Flickr user perspicacious
Bell pepper image: Flickr user Thomas Hawk
I’ll give a brief overview of the hardware we’re using and the design of our current prototype, and then the rest of the talk will be devoted to describing a recent study we completed on finger guidance for reading.
The cameras we’re using were originally designed to be used in minimally invasive surgeries, but their tiny size makes them perfect for our use as well. The images on the left show how small the sensors are, with and without a ring of tiny LEDs. We use the LED version in our prototypes to provide light for the camera.
We’re still exploring different form factors and camera positions, whether the camera should be on top of the finger or on the side, whether it should be up close to the touch surface or further back. We’re also looking at whether it might be useful to have these cameras attached to more than one finger.
Our goal is to power and control everything from a smart watch platform on the user’s wrist. Our early experiments with mobile phones have shown that they’re already powerful enough for the kinds of image processing tasks we’d need to do.
Our current design uses a computer for most of the processing, but it also includes a wrist-mounted microcontroller that communicates with the computer over Bluetooth and powers and controls the haptic feedback on the user’s finger.
We’re still experimenting with several different haptic actuators and other types of tactile feedback, which could be mounted on the fingers or the wrist. The motors we’re using now are larger than we’d like, but they’re cheap and reliable. We use them to augment the sense of touch by providing notifications or directional guidance.
We also provide speech feedback and other audio cues through built-in speakers or over a lightweight wireless headset.
Our objective is to create a platform that we can extend to support many different activities of daily life. We’re developing algorithms to support touch-based interactions using a finger-mounted camera. We’re also investigating different physical designs for the system and different types of haptic and audio feedback, with the goal of making the system intuitive and comfortable enough for everyday use.
We plan to demonstrate our system’s potential through three proof-of-concept applications. Reading and exploring printed documents. Identifying and conveying non-tactile information to help users coordinate clothing when they’re getting dressed. And enabling more intuitive access to cell phones and other technology through interactions on the user’s own body.
Next I’m going to talk about a recent study that we completed, where we were looking at the touch-based reading experience in general, and more specifically at two different types of directional finger guidance.
Since we’re trying to create a touch-based reading experience, we took a close look at braille. It has a high learning curve, but the immediate access to information and the spatial interactions through touch are very similar to the experience we want to provide.
We also looked at existing technology for reading printed documents. Desktop scanners and screen reader software work, but they’re not very portable.
Cell phone apps and wearable cameras can scan a document and read it back to you, but if you’re totally blind they can be difficult to line up with the document and know whether you’ve captured all of the text. None of them use the spatial layout information or allow for easy exploration.
Devices that help people with visual impairments read text are fairly common, but touch-based reading offers some potential advantages over the more traditional approaches. Reading by touch enables immediate access—you place your finger on the page, and you can immediately start getting feedback and exploring the document. You don’t need to worry about lining up the document with the camera or waiting for it to be scanned and processed.
It may also provide a more intuitive way for users to explore a document’s spatial layout, which we hope could improve comprehension and document understanding, especially for documents with complex layouts like newspapers or menus.
There’s a project from MIT called FingerReader, which also uses a finger mounted camera and haptic feedback to help users read printed text. Our goals are the same, but we provide some different features and evaluated the directional guidance much more extensively.
Directional finger guidance is also a basic problem for any touch-based application that we might support. With reading, guidance especially important, because unless you keep the camera centered over a line of text the system won’t be able to see what to read. We’ve explored a few different types of haptic and audio cues to help users stay centered over the line, and to locate the start of the next line. We’ve also developed cues to help users to explore a document’s layout, identifying the content that’s underneath their finger.
Our system has 2 modes: exploration mode and reading mode.
In the exploration mode, the user can move around the page however they’d like, and the system will provide continuous audio feedback to identify the content that’s underneath their finger. We chose distinctive high and low pitched sounds to represent text or pictures, and silence to represent white space.
[play video]
In the reading mode, the user moves sequentially along each line and the system reads each word out loud as their finger passes over it. Users can use their right index finger to read and their left to serve as an anchor to help them find the start of the next line. We provide directional guidance to help the user move vertically to stay on a line or to locate the start of a new line.
[play video]
Continuous audio cues indicate when the user’s finger drifts off of the line, and the frequency changes depending the direction and distance to the line. The haptic directional guidance we provide is very similar, but it uses the intensity and location of the vibrations in place of the audio frequency. We also play distinctive audio cues to identify the start and end of each line or paragraph.
We use the Tesseract OCR library to recognize text in the video frames from the camera, along with some custom preprocessing algorithms to clean up the images and track motion between frames.
We recently completed 2 studies on touch-based reading.
The first was focused on the feasibility of our approach and looked closely at the user experience. We also performed a detailed comparison of two types of directional finger guidance.
Our second study was a proof-of-concept evaluation of our prototype, with a much smaller set of participants.
For our first study we were focused just on the user experience and directional guidance, so we used an iPad to simulate the reading experience. That also allowed us to collected precise finger trace data to use in our analysis.
We recruited 20 participants, all of whom were totally blind or with only minimal light perception.
Our study used a within-subjects design with 2 directional finger guidance conditions: audio and haptic.
We asked each participant to explore and read 2 documents for each condition, a plain text document and a magazine-style document with a more complicated layout.
Throughout the study sessions we collected the participants’ reading speed and accuracy, their comprehension, and subjective feedback.
The speed and accuracy of the two types of guidance were very similar, but audio may offer some advantages to accuracy since it was significantly more accurate than haptics while reading the magazine documents.
The participants’ preferences were also split, although a majority of them preferred haptic guidance.
From our participants’ comments, both conditions may have some limitations; audio may interfere more with the speech feedback, and the haptics may lead to desensitization over time.
Many participants said that they appreciated the lower learning curve compared to braille, and some liked the direct access to spatial information.
But there were also some concerns about the ease of use and the amount of concentration that was required to complete the reading tasks.
We conducted a follow-up study with 4 participants randomly selected to return and test out a prototype version of our system. They also tried out the KNFB Reader iPhone app, which is the current state of the art.
Unfortunately, I don’t have time to go into any detail about the study, but our results showed that both approaches have some strengths and weaknesses. We’ll need to do further research to determine how to best support reading using our system.
These studies left a lot of room for future work, with several new questions to be answered.
We’ll need to do a more in depth study to see how useful the spatial layout information might be in everyday use.
We’ll also need to iterate on our design to reduce the cognitive and physical effort required. It might be that finger-based reading isn’t practical when reading longer passages, but maybe we can optimize it for quick access to smaller amounts of text.
We’ve also considered the idea that a hybrid system may be a good compromise. It could use a body mounted camera to determine the context and read back longer passages, but then use the finger-mounted camera for immediate access and exploration.
Finally, reading is only one of the applications we hope to support. We’ll continue to work towards a wearable device that can augment the sense of touch and help support people with visually impairments throughout a variety of activities of daily living.
Before I end my presentation, I’d like to take a moment to acknowledge everyone who has contributed to this project. We have a large and diverse research team, and it’s been a great experience working with all of them.
Thank you.
