Presentation at ACM CHI'13 in Paris by Antti Oulasvirta (Max Planck Institute for Informatics). Work done in collaboration with Keith Vertanen (Montana Tech) and Per Ola Kristensson (University of St Andrews)

1. Improving Two-Thumb Text Entry
on Touchscreen Devices
A. Oulasvirta,A. Reichel,W. Li,
Y. Zhang, M. Bachynskyi
K.Vertanen
P.O. Kristensson
MAX PLANCK INSTITUTE FOR INFORMATICS
MONTANA TECH
OF THE UNIVERSITY OF MONTANA
UNIVERSITY OF ST ANDREWS
10. toukokuuta 13

2. 10. toukokuuta 13

3. Text input,
more than anything else,
is the problem Apple needs
to solve
http://www.businessinsider.com/how-will-we-type-on-the-apple-tablet-2010-1?op=1
10. toukokuuta 13

4. iPad Dial KeysiPad Dial Keys Android qwerty
Windows 8 thumb kbd Windows 8 regular
Large variety of virtual
keyboards for tablets
10. toukokuuta 13

5. Two-thumb typing
10. toukokuuta 13

6. Survey of postures on
touchscreen devices
user had no experience and 7 indicated the user was an
expert at entering text on soft smartphones keyboards.
As shown in Figure 2, each of the three postures was used
at least “sometimes” by about 60% of the participants. On
the other hand, no single method was used by all
participants. Two thumbs, one thumb, and one finger were
“almost never” used by 37%, 35% and 41% of the
participants, respectively.
Figure 2. Use of an index finger, two thumbs, and one thumb
during text entry on smartphones.
The K
the de
to prev
user’s
words
perfor
almost always often sometimes almost never
0
10
20
30
#ofpeople
Frequency of use in text entry
[Azenkot & Zhai MobileHCI 2012]
N = 75
10. toukokuuta 13

7. Natural in mobile use
10. toukokuuta 13

8. 14 wpm
[Oulasvirta et al. 2010]
[Goel et al. 2012]
31 wpm
21 wpm
In-house testing N=6
...
Text editing
Presentations
Spreadsheets
Data entry
Coding
(Long) Emails
Short messages
Note-taking
Search queries
URLs
Addresses
Names
...
Reported rates
10. toukokuuta 13

9. 14 wpm
[Oulasvirta et al. 2010]
[Goel et al. 2012]
31 wpm
21 wpm
In-house testing N=6
...
Text editing
Presentations
Spreadsheets
Data entry
Coding
(Long) Emails
Short messages
Note-taking
Search queries
URLs
Addresses
Names
...
Reported rates
Following best practices,
how much can we boost typing performance?
10. toukokuuta 13

10. Design space
x-position
y-position
button widthC
error correction
width
height
A B
organization of letters
X
10. toukokuuta 13

11. Layout?
Letter-to-key
assignment?
Error correction?
Study of grips
[Wagner et al. CHI’12]
Model of two-thumb
tapping [Clawson et al. CHI’07]
U!
G! T! O!
K! A! L! Q!
I! E! !
J!
D! N! F! V!
R! Y! S! Z!
P! X! C!
M! B! W! H!
Optimization
[Zhai et al UIST’00]
Language & movement model
[Kristensson &Vertanen ISCA’11]
Training?
[Castellucci & MacKenzie CHI’11]
[Sears et al. B&IT ’01]
[Zhai et al. CHI’02]
10. toukokuuta 13

12. How to hold the device?
10. toukokuuta 13

13. Grip space
α
10. toukokuuta 13

14. Study 1: Grips
6
PROXIMAL DISTAL
RADIALULNAR
5
4
3
CORNER
EDGE
2
1
10. toukokuuta 13

15. 10. toukokuuta 13

16. Study 1:Task
+
+ appears randomly
on both sides on
reachable areas
6 participants
x 3,000 taps / grip
x 6 grips
= 108,000 taps
10. toukokuuta 13

17. Study 1: Results
1
2
3 4
5
6
~10% faster than the worst grip
~4% faster than a random grip
6
PROXIMAL DISTAL
RADIALULNAR
5
4
3
CORNER
EDGE
2
1
10. toukokuuta 13

18. Winner grip
57.6 mm
Split keyboard
10. toukokuuta 13

19. Constrained movement
10. toukokuuta 13

20. 58 mm
5 mm
Grid is incompatible with QWERTY
10 mm
99% CI for offset
10. toukokuuta 13

21. Letter-to-key assignment?
6 * 1033 alternatives!
A B C D E F G H I J K L M N O P
Q R S T U V X Y Z _
10. toukokuuta 13

22. A previous model for
physical thumb kbds
[Clarkson et al. CHI’07]
!!"##$ !"#!!!, !"#! = ! + !!!" = ! + !!!"#! !
!
!
+ 1 ,!!!(1)
where D is the distance between keys, W is the width of keyn,
ID is the index of difficulty derived from D and W, and a and
b are empirical parameters.
For alternate-side (switching) taps, the “idle” thumb is as-
sumed to approach its next target aggressively. Its movement
time is affected by not only ID but also the time elapsed,
telapsed, before its turn. After it presses keyn, the thumb imme-
diately starts to approach keyn. If it has not yet reached it
when its turn comes, the remaining movement is shorter than
if having to start from the beginning. If telapsed is long enough
for the thumb to reach keyn, it can rest over or on it. Then,
only a minimal time tmin is needed for pressing keyn. The total
time Tn for the nth letter in a word is:
!!!!!!!!!! =
!!!! + !!"##$ !"#!!! − !"#! !!!!!!!!!!!!!!"#$
!"# !!!!!!!"#
!!!!!!!"##$(!"#!!!!!"#!)
!!!!!!!""!#$%&
!!!!!!!!!!!!!(2)
In the case of touchscreens, resting on a key is impossible
because it would cause an erroneous tap. For one to benefit
their k
taps. U
users t
Each s
Task a
sequen
as poss
targets
plannin
becaus
redone
Proced
er-over
structe
while w
on key
(correc
screen
Modeli
lantic Drive, Atlanta GA 30332-0280
ark, kent, jamer, thad}@cc.gatech.edu
y introduced a
pert two–thumb
In this work we
m a longitudinal
pdate the model
aw
ategies
Figure 1. The mini–QWERTY keyboards
previous studies: Targus (left) and Dell (right).
approach
wait
press
10. toukokuuta 13

23. What about
touchscreens?
thumb cannot
rest on a target
10. toukokuuta 13

24. Model acquisition
the N-return task
2
4
3
1
5
20 participants 10 x 10 trials per sequence
1 < N < 9
10. toukokuuta 13

25. Predictive model
Left Right
Same-side taps
with taps longer than 1,000 ms. Because in Step 3 we use
pixel coordinates, we here report D in pixel units. In all
models, we use eight ID conditions. For modeling of side-
switch taps, we use six telapsed conditions.
Same-Side Taps
For same-side taps, the subset of the fastest 7% of tap se-
quences constitutes 25,296 data points, or 65% of all data.
This indicates that performance in this task improved quick-
ly, stabilizing near a user’s personal best. We model MT
with a polynomial:
!"!"#$ = 319.5 − 89.0!!" + 36.7!!"!
(3)
!"!"#!! = 237.3 − 7.6!!" + 13.8!!"!
(4)
R2 = .94 R2 = .95
10. toukokuuta 13

26. Hover-over behavior
1
5
hover and wait
approach
press
10. toukokuuta 13

27. 2. The lowest-ID targets are slower than medium-ID tar-
gets, in contrast to the standard Fitts’-law models.
The need for a squared term can be explained by the obser-
vation that a thumb at times occludes nearby targets (low-
ID) and it needs to be moved away for seeing the target. If
one limits to ID≥1.3, a first-order model suffices.
Alternating Taps
Out of 14,619 returning taps (the Nth tap) in data, filtering
to the best 15% within a condition yielded 5,105 data points
(35% of the total). The 5% threshold was chosen to address
the fact that reaching the best performance in alternating
taps requires quite a few repetitions, and we had fewer ob-
servations of returning taps per sequence. Our model is a
bivariate quadratic function with telapsed (see Background)
and ID as the predictive variables:
!"!"#$ = 265.286 − 9.501!!" − 0.024!!!"#$%!& + 2.003!!"!
!
−0.007!!!"#$%!&!!" + 3.322 ∗ 10!
!!!"#$%!&
!
(5)
!"!"#!! = 142.601 + 86.564!!" + 0.062!!!"#$%!& − 17.949!!"!
!
!!!!!!!!!!!!!!−0.035!!!"#$%!&!!" + 1.930 ∗ 10!
!!!"#$%!&
!
(6)
faster
switch
er-ove
toward
turn a
prising
The pr
decrea
telapsed
simila
due to
tention
tion gr
needs
Desig
We arr
1. M
2. In
3. W
cl
R2 = .79 R2 = .79
Alternating taps
10. toukokuuta 13

28. Optimization
Gradient)Descent))500#itera)ons#
Annealing))10#X#3,000#itera)ons#
Gradient)Descent)10,000#itera)ons#
5,000#random#layouts#
100#best#
10#best#
1)winner)
Optimization
Mobile Enron dataset
Predictive model
[Vertanen & Kristensson MobileHCI’11]
10. toukokuuta 13

29. Key row optimization
Dextr
Microsoft’s new curved kbd
A" A" A" A"
A" A" A" A"
A" A" A" A"
A" A" A" A"
A" A" A" A"
A" A" A" A"
A" A" A" A"
A" A" A" A"
All key row position combinations within the
reachable area 48
10. toukokuuta 13

30. U!
G! T! O!
K! A! L! Q!
I! E! !
J!
D! N! F! V!
R! Y! S! Z!
P! X! C!
M! B! W! H!
Predicted rate of 49.0 wpm
10. toukokuuta 13

31. U!
G! T! O!
K! A! L! Q!
I! E! !
J!
D! N! F! V!
R! Y! S! Z!
P! X! C!
M! B! W! H!
Analysis
4.1% better than qwerty
6.1% better than alphabetical layout
64% of long-grams
Fewer letters
Most vowels (a,e,u,o,i)
Spacebars centrally
located
54%46%
62% switching
10. toukokuuta 13

32. 10. toukokuuta 13

33. Error correction
Touch model
[Kristensson &Vertanen ISCA’11]
Language model
Normally distributed touch points Perplexity of 3.84
!∗
= arg!max! ! ! ! ! ! . (8)
Movement Model
Since KALQ is a new keyboard layout there is no straight-
forward method to collect representative touch point data.
We could not train a likelihood model on the evaluation
study’s touch point data as this would mean we would train
the model on the same subjects. Therefore, we instead esti-
mated the likelihood P(K|T) by using a prescriptive model
that assumes normal distribution of touch points [13],
which is justified by existing evidence [9]. The probability
of a touch point belonging to a particular key is
! ! ! = exp −
!!
!
!!
! , (9)
where !! is the Euclidean distance between the touch point
and the center of the key and !! is an estimate of the vari-
ance of the touch point distribution around that particular
key’s center. This parameter was estimated from training
data of Step 2 that is disjoint from the evaluation (Step 5).
Language Model
The prior probability P(K) was estimated using a statistical
and question
7-gram lang
no count cu
mobile devi
size. Our fi
probabilities
size of 9 MB
by using a
devices [12]
terms of ave
cates the av
possible nex
the MobileE
its small siz
pared to an
ters. This la
to 3.44.
STEP 5: TR
Empirical e
tions in the
turned out t
clustered around the spacebar, whereas the left thumb has
only a few fast-action keys while the rest are more diffuse.
This exploits the unique switching characteristics observed
in the N-return study. A typing example is given in Table 1.
STEP 4: ERROR CORRECTION
Previous work has shown improvements in text-entry accu-
racy on mobile devices through error-correction techniques
that consider linguistic context and movement characteris-
tics [6,9,13]. Ideally, error correction should operate in real
time, correcting erroneous characters as they are typed.
Building on previous work [13], we constructed an error-
correction technique for KALQ* that utilizes both linguistic
information and the movement model for two-thumb text
entry. For each touch point T, the error-correction model
finds the key !∗
that maximizes the posterior probability:
!∗
= arg!max! ! ! ! ! ! . (8)
Movement Model
Since KALQ is a new keyboard layout there is no straight-
tatistical properties of typing
argets (hue: slow fast)
sparency 1–100).
ovement times and the fre-
demonstrates how the right
ate, frequently pressed keys
whereas the left thumb has
le the rest are more diffuse.
ing characteristics observed
example is given in Table 1.
N
Letter% Hand% D%(px)% telapsed%(ms)% MT%
S* L* ]* ]*
O* R* **93* **266*
U* R* **93* ]*
N* L* **66* **485*
D* L* **66* ]*
S* L* **93* ]*
_* R* **66* **782*
G* R* 148* ]*
O* R* 132* ]*
O* R* ***0* ]*
D* L* **93* 1015*
Table 1. Predicted typing performance with K
! ! = ! ! ! ≈ ! ! C!!!
!!!
),
where C is all previously written text and !!!!
!!!
a
six characters written.
We trained our model on a sample of 778M mes
via Twitter (12/2010–6/2012). Duplicate tweets
and non-English-language tweets were elimina
language-identification module [18, 19] (with a C
We included only tweets written on mobile d
U!
G! T! O!
K! A! L! Q!
I! E! !
J!
778M tweets
Inserts,
deletes,
subs-tutes
le.ers
for
the
last
6
characters.
10. toukokuuta 13

34. Training
Saarland university (M 25 years, SD 3.52). They reported
having almost no experience with large touchscreen devices
such as tablets, and only one was a touch-typist on physical
QWERTY keyboards. The participants were compensated at
10€/hour, and the two best were given a bonus of €100.
Session% Contents% Test% Goal%
0% QWERTY*typing*test* I* Baseline**measurement*
1% Grip,*idle*thumb*tech]
nique,*spacebar*policy*
* Introduce*KALQ,*confirm*
understanding*of*the*basics*
1?3% The*alphabet*and*frequent*
words*
* Type*the*alphabet*without*
seeing*the*key*labels*
3?8% Frequent*bigrams*and*
words*
II,*III* Learn*motor*techniques*for*
frequent*text,*speed*up*
9?13% Full*sentences,*frequent*
bigrams*and*words*
IV* Speed*up*gradually*
13?19% As*above*but*extra*prac]
tice*with*error*correction*
* Speed*up*while*keeping*
error*rate*under*5%*
Final% Final*evaluation* Va,*Vb* Personal*best*with*KALQ*
6 students (non-natives in English!)
[Castellucci & MacKenzie CHI’11]
[Sears et al. B&IT ’01]
[Zhai et al. CHI’02]
10. toukokuuta 13

35. Results
Figure 9. Development of text entry speed throug the training
KALQQWERTY
1 4 8 11 13-19
Test day
wpm
37 wpm
5% CER
28 wpm
9% CER
10. toukokuuta 13

36. Conclusion
10. toukokuuta 13

37. 10. toukokuuta 13

38. Row shifting: pred < 1%
Grip: 4-10%
Optimization: pred. 4%
Error correction: 1.3 perc units in cer
Hover-over: 10-20%
28 wpm (9% cer)
37 wpm (5% cer)
See the paper for
details here
10. toukokuuta 13

39. The challenge
50 wpm
10. toukokuuta 13

40. Thanks!
oantti@mpi-inf.mpg.de
Will have KALQ, code, and data
http://www.mpi-inf.mpg.de/~oantti/KALQ/
10. toukokuuta 13