1. Gesture phones
Krishna Kumar.S, Sai Venkata Vinay .C
1st
year- ECE,
Sona College of Technology, Salem, Tamil Nadu.
Abstract: There are thousands of cell
phone models available in the market, all
with varying features in recent years, cell
phones have also seen considerable
progress in terms of user interaction,
evolving all the way from QWERTY keypad
to touch screens, and the next transition in
user interface is gesture recognitions on
Smartphone, such phones are termed as
gesture phones. This document comprises
of gesture control technology which
explains how a Smartphone understands
gestures. The scenarios where smart
phones are operated by gesture recognition
on the day to day basis are also explained.
They are adding a whole new dimension to
multimedia via touch free technology.
Mobile operating like android and windows
8 and so on use gesture recognition with
their own features. With that, this new
experience can be offered to broader
audiences on a wider range of devices.
Gesture phones are enforcing many
innovations on because of its advantages to
mankind. But it’s not so simple for it has
many several challenges which remain a
mystery for gesture recognition. However,
these problems can be overcome with
different tracking and new technologies
with good reasoning. This paper precisely
includes the innovations that can be
implemented for more efficient use of
gesture phones.
1. How a Smartphone
Understands gestures
Gesture recognition is done through a few
types, among them a few are optical
tracking, inertial tracking, and calibration.
The device understands by computing the
position of the gesture, its orientation,
acceleration and also its angular speed. The
information collected is processed and
finally the gesture is recognized and the
necessary response is given. Fig. 1 explains
clearly the model of recognition of gestures
Fig.1: Prototype of gesture phones[1]

2. on Smart phones.
1.1 Optical Tracking
The gestures are tracked by analyzing the
variations in the color of each and every
pixel recorded through one or more
cameras. The different movement is
recognized by the camera and compares it
with saved gestures while the phone is in a
steady position and it gives vibration. This
buzz indicates that the movement has been
recognized. The gesture is saved and
implemented to carry out the corresponding
task(s) you have attached to its profile.[3]
There are three common technologies that
can acquire optical tracking that is 3-d
images, each with its own advantages.
Stereoscopic vision, structured light pattern
and time of flight (TOF) among which
stereoscopic vision is ideal for Smartphone,
tablets and other consumer devices. The
analysis of these 3-d images brings 3-d
technology into reality.[4]
1.1.1 Stereoscopic vision
It uses two cameras to obtain a left and right
stereo image. As the computer compares the
two different images, it develops a disparity
image that computes the displacement of the
objects in the images.
The views from each camera are different; it
becomes possible to determine the distance
to scene points by triangulation. Essentially,
if we can identify a pair of pixels that image
the same 3D point, one in the left camera
image and the other in the right image, then
the 3D coordinates of the point can be
calculated as the intersection of light rays
from that point to each of the pixels.
Fig.2: a rock on the surface of Mars, as imaged
by a stereo pair on one of the Mars Exploration
Rovers. Both the original (typically left camera)
intensity image and a false-color “elevation
map” image are shown.[1]
1.1.2 Pros and cons of stereoscopic
vision technology
Stereoscopic vision technology requires
amount of software complexity for highly
precise 3D depth data that can be processed
and analyzed in real time by digital signal
processors (DSPs).[1] Errors in camera
alignment as well as insufficient lighting
leads to significant errors in the calculation
of depth values. Precise camera
synchronization is required if the object is in
motion.
Advantages of the sensors are the use of
well established hardware (2D-cameras)
with a wide range of capabilities. Cost
effective and fit in a small form factor,
making them a suitable for Smartphone,
tablets and other consumer devices. It

3. cannot deliver high accuracy and response
time.
1.1.3 Triangulation
The coordinates and distance to a point can
be found by calculating the length of one
side of a triangle, given measurements of
angles and sides of the triangles formed by
that point and two other known reference
points.
Fig.3: two cameras focusing on the same point
1.1.3.1 Calculation:
𝒍 =
𝒅
𝒕𝒂𝒏𝜶
+
𝒅
𝒕𝒂𝒏𝜷
Therefore,
𝟏
𝒅
=
𝟏
𝒍
(
𝟏
𝒕𝒂𝒏𝜶
+
𝟏
𝒕𝒂𝒏𝜷
)
Using the trigonometric identities,
𝒕𝒂𝒏 𝜶 =
𝒔𝒊𝒏 𝜶
𝒄𝒐𝒔 𝜶
and
𝒔𝒊𝒏(𝜶 + 𝜷) = 𝒔𝒊𝒏 𝜶 𝒄𝒐𝒔 𝜷 + 𝒔𝒊𝒏 𝜷 𝒄𝒐𝒔 𝜶
This is equivalent to:
𝟏
𝒅
=
𝒔𝒊𝒏(𝜶 + 𝜷)
𝒍 𝒔𝒊𝒏 𝜶 𝒔𝒊𝒏 𝜷
From this way, it is easy to determine the
distance of the unknown point from
observation point, its north or south and east
or west offsets from the observation point,
finally its full coordinates.
1.2 Inertial Tracking
The motion of Smartphone and similar
devices is tracked with high precision,
without depending on external structure or
prior knowledge of the environment of
usage.
The camera was assumed to have a global
shutter, where all the pixels in an image are
captured at the same time. But cameras for
inertial tracking use CMOS sensors with
rolling shutters, capturing each row of pixels
at slightly different time instant. By this way
the variation in the position of the pixels in
comparison with the image with normal
shutters, the Smartphone understands its
own motions.[2]
These motions are recognized as inputs for
which the response is to perform some task.
Fig.4: Example of an image taken using rolling
shutters.[2]

4. Fig.5: These motions are recognized as inputs for
which the response is to perform some task.
Different movements are saved and each of them
can perform functions.
2. Cases of Gesture Recognition
Technology
Gesture recognition technology would be
widely spread and used in future on mobile
phones as a common feature in them. A few
examples of cases, which are enhancing the
user experience compared to what is
available today, are listed below as follows:
1. We can answer or unanswered an
incoming call with a wave of the hand while
driving and can make outgoing calls
similarly. This feature is termed as call
control. This comes under optical tracking
2. We can skip tracks on your media player
while listening songs using simple hand
motions either optically or inertial or two
dimensionally.
3. We can even operate the other devices
without using their remote control but by
using gesture phones.
4. We can turn down or increase the volume
phone or switch off the mobile through it.
5. The most interesting feature at future on
mobile phones would be browsing
WebPages with the hand movements
without operating a phone
6. We can scroll web pages, or within an
eBook with simple left and right hand
gestures.
7. We can switch off the mobile phone,
mute it or can change to vibration mode
through gestures.
8. Another interesting use case is when
using the Smartphone as a media hub; a user
can dock the device to the TV and watch
content from the device- while controlling
the content in a touch-free manner from a far
position.
Amongst the above mentioned cases, a few
are available in the market as a feature in
phones that come under the labels like
iPhone, Samsung, micromax, Kinect, Lava,
QUALCOMM, etc. These mentioned cases
are ideal when touching the device is a
barrier such as when hands are wet, with

5. gloves, dirty etc. (we all are familiar with
the annoying smudges on the screen from
touching). Moreover, these are done either
by two dimension gesture recognition or
optically, in an inertial way.
3. Applications of gesture
technology on android and windows
phones
The few above mentioned cases are inbuilt
features in mobile phones but a few are not
inbuilt in them. In order to overcome this
problems and give more emphasis on
gesture tool. Software developers have
developed and coined some apps which can
be operated on mobile OS’s. The apps which
are brilliant in use and usage are listed
below but for them to operate we need front
view camera at most one or sensors in them.
Of course, all most all smart phones have
them. Point Grab’s [5] solution is ideal for
use with tablets and smartphones in two
primary scenarios: when operating the
device closely without the possibility of
physically touching the screen (i.e. hands are
not clean), or when the user wishes to watch
the device content on a large screen that is
connected to the device. In both cases, Point
Grab’s solution brings next generation hand
gesture technology with gestural
interfaces to a diverse range of applications.
Air Swiper is one of the most feature rich
apps in this set. This application allows you
to work more effectively and faster with
your phone. It has a number of features
including unlocking and locking the device,
controlling SMS and sound control.
Hovering Controls is one other such app
which costs $1.34 and you can set it up to
launch predetermined apps by hovering over
the sensor, swiping once, or swiping twice.
Some of the things you can do with this app
include silencing an alarm, controlling
media playback like videos or photos with
just gestures. [9]
3.1 Modes of two dimensional app
The app has two modes. The Target mode
which opens a pre-determined app with a set
gesture and the carousel mode, which allows
you to select from a set list of apps. Scribble
to open any app-Google Gesture Search is
an app that has been around for some time
and even though it requires you to actually
touch the screen it is a really useful app. The
app allows users to quickly access contacts,
applications, settings, music and bookmarks
on your Android device by just drawing
letters or numbers on the screen. The app is
self- learning and becomes more useful the
more you use it.
4. Benefits of gesture technology
As gesture technology is a high level
processing vision algorithm. There are
various benefits available for this gesture
architecture and these benefits helped the
researchers to coin many innovations and
develop applications. These benefits are:
1. This gesture technology enables a
flexible programming environment to
the user.
2. It is very user friendly as it provides
multiple tasking, multiple programming,

6. multiple sharing, multiple instructions,
and multiple data transmission.
3. These gestures help in paralysing the
data and task with throughput which
includes fast prototyping and
optimisation.
4. Moreover, this gesture phone would be
highly portable than any other device.
As there are benefits, cons would also be
there running back of it.
5. Challenges of gesture
technology
1. The entire gesture technology depends
on compiler’s performance if he is slow,
then efficiency in fast prototyping would
decrease and may lead to misconception in
programming for what we program is what
we get.
2. Optical tracking consumes battery
highly due to high clock cycle so it affects
the fair performance.
3. If the inter processor is of low
standard then memory access would be
inefficient.
4. Optical tracking requires a camera and
with good camera resolution. More over this
may require good consistent lighting. Items
in the background or distinct features of the
users may make recognition more difficult.
5. In periods of prolonged use of hand
movements, may lead to disease called
gorilla arm[6] that is users’ arms began to
feel fatigue and/or discomfort.
These cons can be overcome but it requires
ultrasonic waves, which is called as
ultrasonic tracking. This feature in mobile
phones would be the future innovation.
6. Innovations:
 Ultrasonic waves[7][8] can be used
for gesture recognition where it contains
sensor and speakers which could produce
these waves and sensors would sense the
change in amplitude and follow the
command based on the profile given to it
.Since battery consumption is low and can’t
be affected by light. It can widely be used
like operating a machine far away from us.
 Infrared waves are also used for
gesture recognition but only for smaller
applications which can be achieved by
phone because infrared can’t travel through
obstacles where as there is no such problem
arises in ultrasonic waves but it produces
intense heat when passed through obstacle
but can be overcome.
 The Smartphone can pause sending
sound signals where we close the cell with
hand while someone else is talking. This
prevents the person on the other side from
entering into speakers personal issues.
 Sometimes people really don’t want
to reveal their background during video
chats. We can use the depth estimation
techniques to recognize the background and
only send a blurred or blackened or different
background.
 The Smartphone can understand
stealing gesture and can actually alarm you

7. about the burglar. This way it keeps itself in
safe hands.
7. Conclusions:
Gesture phones can recognise gestures by
two methods, optical and inertial tracking.
Optical tracking uses triangulation method
for the estimation of details including the
depth of the object. Inertial tracking is based
on the idea of rolling shutters. The most
widely accepted innovations on gesture
phones are also mentioned. Their
applications in android and windows phones
along with different modes of app are
explained. Challenges the gesture phones
will face are described. Considering
different areas where this technology can be
implemented for efficient use of
Smartphone, few innovations are stated.
References:
1. BongWhan Choe, Jun-Ki Min, and Sung-
Bae Cho; Online Gesture Recognition for
User Interface.
2. Mingyang Li, Byung Hyung Kim and
Anastasios I. Mourikis; Real-time Motion
Tracking on a Cellphone using Inertial
Sensing and a Rolling-Shutter Camera.
3. Mingyu Chen, Ghassan AlRegib, Senior
Member, IEEE, and Biing-Hwang Juang,
Fellow, IEEE; Feature Processing and
Modeling for 6D Motion Gesture
Recognition.
4. Dong-ik ko, Lead engineer, gesture
recognition and depth –sensing gaurav
agarwal, Manager, Gesture recognition and
depth-sensing; Gesture recognition:
Enabling natural interactions with
electronics.
5. Innovations on gesture technology,
http://www.pointgrab.com/276/gesture-
mobile/
6. Diseases related to gesture recognition on
phones.http://www.incosecc.org/http:/ww
w.incosecc.org/images/2012_01_18_Gestu
re-Recognition.pdf
7. Ultrasonic waves usage in gesture phones
http://phys.org/news/2013-11-spinoff-
ultrasonic-gesture-recognition-small.html
8. Ultrasonic waves usage in gesture phones
http://www.washington.edu/news/2014/02/
27/battery-free-technology-brings-gesture-
recognition-to-all-devices/
9. Li yin Kong 2009127643; supervisor: dr.
Kenneth Wong; fyp12026 gesture
recognition on Smartphone