Tutorial on Model-Based User Interface Optimization. Part I: LETTER ASSIGNMENT by Anna Feit of Aalto University. Presented at SICSA Summer School on Computational Interaction in 2015 in Glasgow. Note: This one-day lecture is divided into multiple parts.

1. SICSA Summer School
on Computational Interaction
Letter Assignment
Anna Feit
23.06.2015

2. The Assignment Problem
Keyboard Optimization
Heuristic Approaches
Competition
Integer Programming
Overview

3. The Assignment
Problem
On secret agents and the choice of the right weapon …

4. Assignment Problem
Anna Feit – SICSA Summer School on Computational Interaction - 4
[Image: www.classtools.net]
[Image: paxsims.wordpress.com]

5. Assignment Problem
p11 p12 p13 p14
p21 p22 p23 p24
p31 p32 p33 p34
p41 p42 p43 p44
Assign each
task to exactly
one agent such
that we get
most profit.
Anna Feit – SICSA Summer School on Computational Interaction - 5

6. How to model the problem?
One variable xij for each Agent-Task pair:
xij = 1 if Agent i is assigned to Task j.
x11=1
Agent 1 is assigned to task 1
x12=0
x13=0
x14=0
Anna Feit – SICSA Summer School on Computational Interaction - 6

7. How to model the problem?
x11=1
x12=0
x13=0
x14=0
Profit for Agent 1:
Agent 1 is assigned to task 1
Anna Feit – SICSA Summer School on Computational Interaction - 7

8. How to model the problem?
Objective function:
Maximize the profit for all agents
Anna Feit – SICSA Summer School on Computational Interaction - 8

9. How to model the problem?
Objective function:
Maximize the profit for all agents
Constraints:
• Each task j must be assigned to exactly one agent
Anna Feit – SICSA Summer School on Computational Interaction - 9

10. How to model the problem?
Objective function:
Maximize the profit for all agents
Constraints:
• Each task j must be assigned to exactly one agent
• Each agent i must be assigned to exactly one task
Anna Feit – SICSA Summer School on Computational Interaction - 10

11. Mathematical formulation
Easy problem
Linear assignment problem: only linear terms in the
objective function.
Anna Feit – SICSA Summer School on Computational Interaction - 11

12. Application to real world problems
• Send taxis to customers:
Profit = route price – cost to reach customer
• Assign lectures to rooms
Cost = number of lectures that are assigned to the
same room
But: very limited problem description
• Same number of agents and tasks
• Assign only one task to each agent
• How about other criteria, e.g. preferences?
Anna Feit – SICSA Summer School on Computational Interaction - 12

13. Generalized Assignment problem
cij
pij
i j
ci1
ci2
ci3
Assigning a task i to an agent j
gives a profit pij but has a cost cij
An agent can work on multiple
tasks but has a work capacity tici1
ci2
ci3
ti
Anna Feit – SICSA Summer School on Computational Interaction - 13

14. Mathematical formulation
Assignment problem:
Generalized
Assignment Problem:
Maximize profit
The sum of task costs
assigned to agent i can
not be larger than his
working capacity ti
Assign each task to one agent
Anna Feit – SICSA Summer School on Computational Interaction - 14

15. Mathematical formulation
Assignment problem:
Generalized
Assignment Problem:
NP-hard problem
Anna Feit – SICSA Summer School on Computational Interaction - 15

16. Quadratic Assignment Problem
Relations within the two sets

17. Quadratic Assignment Problem
Flow between Factories
Anna Feit – SICSA Summer School on Computational Interaction - 17

18. Quadratic Assignment Problem
2
4
3
1
Distance between locations
Anna Feit – SICSA Summer School on Computational Interaction - 18

19. How to model this?
One variable xiv for each Factory-Location pair:
xiv = 1 if Factory i is assigned to Location v
1 2
x13 x24
f12
d34
Anna Feit – SICSA Summer School on Computational Interaction - 19

20. Mathematical formulation
Linear
assignment problem:
Quadratic
assignment problem:
Assign each Factory to
exactly one LocationMinimize the costs
between locations and
factories
Anna Feit – SICSA Summer School on Computational Interaction - 20

21. Mathematical formulation
Linear
assignment problem:
Quadratic
assignment problem:
NP-hard problem
Anna Feit – SICSA Summer School on Computational Interaction - 21

22. More Variations
• Stable Marriage Problem
– Takes into account preferences
– Application: Assign students to hospitals
• Stable Roommates Problem
– No separation into two sets,
assigns one element to any of the other elements
• Weapon-Target Assignment problem
– Model cooperation between agents
– Not all targets must be assigned to a weapon
Anna Feit – SICSA Summer School on Computational Interaction - 22

23. Keyboard optimization
Challenging the Qwerty Keyboard

24. Task: Design a smartphone keyboard
that is optimized for performance
[Image: www.theinq

25. What is wrong with Qwerty?

26. The Letter Assignment Problem
ABCDEFGHIJKLMNOPQRSTUVWXYZ
?!
Goal: Optimize performance
26 letters, 26 keys = 26!> ​10↑25 different mappings
More than stars in the
observable universe!
Anna Feit – SICSA Summer School on Computational Interaction - 26

27. How can we model this problem?
ABCDEFGHIJKLMNOPQRSTUVWXYZ
?!
Anna Feit – SICSA Summer School on Computational Interaction - 27

28. Letter Assignment Problem as QAP
Remember:
1 2
x13 x24
f12
d34
T H
14
7
pt,h
t14,7
Bigram probability
Pointing time
xt,14 xh,7
Anna Feit – SICSA Summer School on Computational Interaction - 28

29. Letter Assignment Problem as QAP
T H
14
7
pt,h
t14,7
Bigram probability
Pointing time
xt,14 xh,7pi,j
Bigram frequency based
on a language corpus
tv,w
Pointing time as given by
Fitts’ Law.
Anna Feit – SICSA Summer School on Computational Interaction - 29

30. Letter Assignment Problem as QAP
T H
14
7
pt,h
t14,7
Bigram probability
Pointing time
xt,14 xh,7
Each key assigned to exactly one letter
Each letter assigned to exactly one key
Anna Feit – SICSA Summer School on Computational Interaction - 30

31. Letter Assignment Problem as QAP
T H
14
7
pt,h
t14,7
Bigram probability
Pointing time
xt,14 xh,7
NP-hard problem
​ 𝟏 𝟎↑ 𝟐 𝟓 mappings
Anna Feit – SICSA Summer School on Computational Interaction - 31

32. Heuristic approaches

33. Heuristic Approaches
• Do not rely on mathematical formulation
• Approximate the global optimum
Good choice to:
• find some ”good” solution in a short time
• solve problems that are too complex to be solved
mathematically
Anna Feit – SICSA Summer School on Computational Interaction - 33

34. Heuristic Approaches
Here: two classes of heuristics:
• Greedy / Constructive heuristics
• Local search heuristics
Try to find a good solution within a given time,
no guarantee on how good it is.
Anna Feit – SICSA Summer School on Computational Interaction - 34

35. General Principle
Greedy approach
1. Start with an empty
solution S = Ø.
2. Iteratively add elements
to S that bring the
highest value.
3. Stop when no more
elements can be added
that would improve S.
Local search
1. Start with a feasible
solution S.
2. Iteratively move to a
better ”neighbour” by
changing S.
3. Stop if no better solution
can be found in the
neighborhood.
Heuristics that tell
you which element
to add next
Heuristics on
how to construct
a neighbour
Anna Feit – SICSA Summer School on Computational Interaction - 35

36. Greedy Heuristics: Example
Heuristic: What element should be added next?
Minimize pointing time for frequent bigrams
1. Start with the empty keyboard
2. Take the next most frequent letter pair
3. Assign the 2 letters such that the key distance is minimized
4. Stop when all letters are assigned
th 1.52
he 1.28
in 0.94
er 0.94
an 0.82
re 0.68
nd 0.63
at 0.59
on 0.57
nt 0.56
…
Anna Feit – SICSA Summer School on Computational Interaction - 36

37. Greedy Heuristics: Example
t h
th 1.52
he 1.28
in 0.94
er 0.94
an 0.82
re 0.68
nd 0.63
at 0.59
on 0.57
nt 0.56
…
Heuristic: What element should be added next?
Minimize pointing time for frequent bigrams
1. Start with the empty keyboard
2. Take the next most frequent letter pair
3. Assign the 2 letters such that the key distance is minimized
4. Stop when all letters are assigned
Anna Feit – SICSA Summer School on Computational Interaction - 37

38. Greedy Heuristics: Example
t
th 1.52
he 1.28
in 0.94
er 0.94
an 0.82
re 0.68
nd 0.63
at 0.59
on 0.57
nt 0.56
…
h e
Heuristic: What element should be added next?
Minimize pointing time for frequent bigrams
1. Start with the empty keyboard
2. Take the next most frequent letter pair
3. Assign the 2 letters such that the key distance is minimized
4. Stop when all letters are assigned
Anna Feit – SICSA Summer School on Computational Interaction - 38

39. Greedy Heuristics: Example
t
th 1.52
he 1.28
in 0.94
er 0.94
an 0.82
re 0.68
nd 0.63
at 0.59
on 0.57
nt 0.56
…
h e i n
Heuristic: What element should be added next?
Minimize pointing time for frequent bigrams
1. Start with the empty keyboard
2. Take the next most frequent letter pair
3. Assign the 2 letters such that the key distance is minimized
4. Stop when all letters are assigned
Anna Feit – SICSA Summer School on Computational Interaction - 39

40. Greedy Heuristics: Example
t
th 1.52
he 1.28
in 0.94
er 0.94
an 0.82
re 0.68
nd 0.63
at 0.59
on 0.57
nt 0.56
…
h e i n
Heuristic: What element should be added next?
Minimize pointing time for frequent bigrams
1. Start with the empty keyboard
2. Take the next most frequent letter pair
3. Assign the 2 letters such that the key distance is minimized
4. Stop when all letters are assigned
Anna Feit – SICSA Summer School on Computational Interaction - 40

41. Local Search Heuristics
Heuristic: how to choose a neighbourhood
Anna Feit – SICSA Summer School on Computational Interaction - 41

42. Local Search Heuristics
Heuristic: how to choose a neighbourhood
S
N(S)
Main components:
1. Goal function f(S) to
evaluate the solutions
2. Rules how to modify S to
obtain the
neighbourhood N(S)
Currently best:
Note: f(S) should penalize infeasibility
Anna Feit – SICSA Summer School on Computational Interaction - 42

43. Local Search Heuristics: Example
Keyboard Optimization
1. Start from a random keyboard S
2. Neighbourhood N(S):
All keyboards obtained from S by swapping 2 keys
3. Evaluation of neighborhood:
Compute the average pointing time of S,
sum of Fitts’ pointing time weighted by bigram frequency
4. Choose the best neighbour S* of S.
If f(S*) < f(S): go to 2.
Else: Stop, S is the local optimum.
Anna Feit – SICSA Summer School on Computational Interaction - 43

44. Problem
[Source: http://bit.ly/1SvePF6]
The optimum of a neighborhood may not be the
global optimum
Neighborhood
Anna Feit – SICSA Summer School on Computational Interaction - 44

45. Metaheuristics
• High-level heuristic that tells you how to change the
local search heuristic
• Strategy to overcome local optima
• Naive Example: Multistart local search
– Local search heuristic:
Obtain neighbordhood by swapping 2 keys of a keyboard
– Metaheuristic:
If a local optimum is found, start over with a completely new
randomized layout
Anna Feit – SICSA Summer School on Computational Interaction - 45

46. Tabu Search
Basic Principle:
1. Store a tabu list of the last t solutions.
2. Perform a local search where the solutions in the tabu list
cannot be chosen.
3. If a local minimum S is found proceed with the next best
S* ∈ N(S) not in the tabu list and add S to the tabu list.
4. At termination return the best solution found so far.
N(S)
S
Tabu:
N(S*)
S
Tabu:
S*
N(S)

47. Tabu search
Main components:
• How to define N(S) (Local search heuristic)
• Length of the Tabu list
• The stopping rule. For example:
– Total number of iterations
– Number of iterations without improvement
Anna Feit – SICSA Summer School on Computational Interaction - 47

48. Simulated Annealing
Idea:
• accept a worse solution from N(S) with a
probabilty 0 < p < 1
• the worse the solution the smaller p
• p decreases at each iteration
Inspired by metallurgy:
Heat a metal to a temperature T
and let it cool down slowly
at rate r
to soften it.

49. Simulated Annealing
Basic principle:
1. Start with a random solution S
Init Temperature T and cooling ratio 0 < r < 1
2. For L iterations:
a) Choose a random solution S* from N(S)
b) Let d = f(S*) – f(S)
• If d<0: proceed with S* (S* is better)
• Else: proceed with S* with probability p=​ 𝑒↑− 𝑑/ 𝑇
3. Reduce the temperature: T = rT and go to 2.
4. At termination return the best solution found so far.
Anna Feit – SICSA Summer School on Computational Interaction - 49

50. Simulated Annealing
Main components:
• How to define N(S) (Local search heuristic)
• Temperature T, cooling ration r, loop length L
• The stopping rule. For example:
– Total number of iterations
– Number of iterations without improvement
– Small Temperature T < ε
Anna Feit – SICSA Summer School on Computational Interaction - 50

51. Genetic Algorithms
Inspired by the natural evolution of biological systems
Anna Feit – SICSA Summer School on Computational Interaction - 51

52. Genetic Algorithms
Inspired by the natural evolution of biological systems
Idea:
• Start with a set of solutions (= population)
• At each iteration (= generation) create a new
population by combining the best solutions
Anna Feit – SICSA Summer School on Computational Interaction - 52

53. Genetic Algorithms
Basic principle:
1. Start with a random set of solutions (population)
2. Evaluate the fitness f(S) of each solution
3. Repeat until a new population is created:
a) Select a pair of parent solutions based on their fitness
b) With probability pc crossover the parents to form two offspring
solutions. Otherwise create two offsprings as copies of the parents
c) With probability pm mutate the offsprings
4. Repeat from step 2.
At termination report the best solution found so far.
Anna Feit – SICSA Summer School on Computational Interaction - 53

54. Genetic Algorithms
Main components:
1. Population size and termination criterium
2. Fitness evaluation: f(S) similar to the goal function in local search
(incl. penalty for infeasibility)
3. Parent selection: probability proportional to its fitness. Variations:
e.g. choose one parent randomly
4. Crossover: implemented as string combinations.
E.g. combine parents at random point xk in string:
5. Mutation: random modifications, e.g. swap elements
(also occurs in nature)
Anna Feit – SICSA Summer School on Computational Interaction - 54

55. A good heuristic ensures…
Communication: Definition of neighborhood should allow to
reach any optimal solution
Anna Feit – SICSA Summer School on Computational Interaction - 55

56. A good heuristic ensures…
Communication: Definition of neighborhood should allow to
reach any optimal solution
S
Anna Feit – SICSA Summer School on Computational Interaction - 56

57. A good heuristic ensures…
Diversification:
e.g. through high initial temperature, large tabu list
S
Anna Feit – SICSA Summer School on Computational Interaction - 57

58. A good heuristic ensures…
Intensification: e.g. use bigger N(S) if S is good
S
Anna Feit – SICSA Summer School on Computational Interaction - 58

59. Optimized keyboard
The metropolis keyboard [Zhai, Hunter and Smith, 2000]
Variation of simulated annealing
• Goal function: Fitts’ pointing time
• Neighbour construction: No fixed layout, move a random
key in a random direction by a random amount
• Several annealing and cooling iterations until ”a sufficiently
efficient keyboard was found”
e
b
c
h
t
Anna Feit – SICSA Summer School on Computational Interaction - 59

60. Metropolis Keyboard
Diversification and Communication:
high temperature, keys move to
random directions
Intensification: Low temperature,
movement toward optimal keyboard
43.1 wpm
(vs. 36 for Qwerty)
Anna Feit – SICSA Summer School on Computational Interaction - 60

61. Optimizing for more than one Finger
The Kalq keyboard [Oulasvirta et al. CHI 2013]
Keyboard optimization for two-thumb text entry
Goal function:
• Adapted Fitts’ Law modeling
performance with two thumbs
• Thumb alternation:
+ Preparation gain
- Recover cost after long time
Optimization procedure is the same,
only goal function exchanged
Anna Feit – SICSA Summer School on Computational Interaction - 61

62. The Kalq Keyboard
Combination of Local Search and Simulated annealing:
1. Diversification:
Local optimization for 5000 random layouts
500 iterations
2. Communication:
Simulated annealing starting from the 100 best solutions,
3000 iterations
3. Intensification:
Local optimization for the 10 best layouts,
10.000 iterations
49 wpm
(vs. 47 for split Qwerty)
Anna Feit – SICSA Summer School on Computational Interaction - 62

63. Optimizing for multiple criteria
Multiobjective Keyboard Optimization
[Dunlop and Levine, 2012]
Multiple goal functions
• Fitts’ pointing time
• Spell correction
• Qwerty similarity in vs. on
if vs. of
…
Anna Feit – SICSA Summer School on Computational Interaction - 63

64. Optimizing for multiple criteria
Multiobjective Keyboard Optimization
[Dunlop and Levine, CHI 2012]
Variant of Local search:
Goal: find a set of optimal solutions that dominate each
other in one of the 3 goal functions
• Start with locally optimized set of solutions
• Neighborhood:
– randomly swap two keys
– randomly swap extra keys with 25% probability
• Termination: 2000 iterations
Anna Feit – SICSA Summer School on Computational Interaction - 64

65. Optimizing for multiple criteria
~37 wpm
(vs. 33 for Qwerty)
Anna Feit – SICSA Summer School on Computational Interaction - 65

66. Summary
• Heuristics approaches to find a good solution in
reasonable time
Ø Constructive vs. Local search
• Metaheuristics to overcome local minima
Ø Tabu search
Ø Simulated Annealing
Ø Genetic algorithms
• Define main components:
Ø Local search: Goal function, Neighborhood, Termination criterium,
Ø Meta heuristics: length of tabu list, temperature, crossover
Anna Feit – SICSA Summer School on Computational Interaction - 66

67. COMPETITION

68. Exact methods
Integer programming

69. Integer Programming (IP)
Linear Assignment problem
All decision variables are (binary) integers
Anna Feit – SICSA Summer School on Computational Interaction - 69

70. Integer Programming (IP)
Integer variables can be used to model
• indivisibilities,
e.g. number of to-be-ordered products
• on/off decisions (binary variables),
e.g. hiring or assigning
Anna Feit – SICSA Summer School on Computational Interaction - 70

71. Integer Programming (IP)
Integer programming is about solving IP problems.
In comparison to Heuristic approaches they
• increase the chance to find the global optimum
• are mathematically exact:
• Proven to find the global optimum or
• Give a guarantee on how good the solution is in
comparison to the global optimum
• But: need a lot of time and computing power, efficient
solution only for few ”easy” problems.
Anna Feit – SICSA Summer School on Computational Interaction - 71

72. Linear vs. Integer programming
Example
Anna Feit – SICSA Summer School on Computational Interaction - 72

73. Linear vs. Integer programming
Linear program (LP)
Easy!
(1.95, 4.92)
Example
Anna Feit – SICSA Summer School on Computational Interaction - 73

74. Linear vs. Integer programming
Integer program (IP)
Hard!
(1.95, 4.92)
(5,0)
Example
Anna Feit – SICSA Summer School on Computational Interaction - 74

75. How to solve an IP
Basic principle:
• Find lower and upper bounds for the problem.
• Iteratively compute tighter and tigher bounds by
– Finding good but approximate solutions
– Solving easier problem
– Refining the problems
Anna Feit – SICSA Summer School on Computational Interaction - 75

76. Linear vs. Integer programming
(1.95, 4.92)Example
Upper bound:
Optimal LP solution
Anna Feit – SICSA Summer School on Computational Interaction - 76

77. Linear vs. Integer programming
Example
Upper bound:
Optimal LP solution
Lower bound:
Any feasible integer solution
(1,1)
Optimal value:
Anna Feit – SICSA Summer School on Computational Interaction - 77

78. Branch and Bound
Divide and Conquer approach
Basic idea:
• Divide the problem into easier subproblems (Branch)
• Solve subproblems to obtain upper and lower bounds
• Use bounds to exclude subproblems that cannot lead to an
optimal solution (Bound)
Typically applied via an enumerative tree
Anna Feit – SICSA Summer School on Computational Interaction - 78

79. Example: Keyboard Optimization
[Karrenbauer and Oulasvirta, UIST 2014]
S
S1
S2
xa,1 = 1 xa,1 = 0
a a
Anna Feit – SICSA Summer School on Computational Interaction - 79

80. Example: Keyboard Optimization
[Karrenbauer and Oulasvirta, UIST 2014]
S
S1
S2
xa,1 = 1 xa,1 = 0
a a
xb,1 = 1 xb,1 = 0
S11 S12
bb
Anna Feit – SICSA Summer School on Computational Interaction - 80

81. Example: Keyboard Optimization
[Karrenbauer and Oulasvirta, UIST 2014]
S
S1
S2
xa,1 = 1 xa,1 = 0
xb,1 = 1 xb,1 = 0
S11 S12
ub = 60 wpm
lb = 30 wpm
…
L
30 wpm
Anna Feit – SICSA Summer School on Computational Interaction - 81

82. Example: Keyboard Optimization
[Karrenbauer and Oulasvirta, UIST 2014]
S
S1
S2
xa,1 = 1 xa,1 = 0
xb,1 = 1 xb,1 = 0
S11 S12
ub = 60 wpm
lb = 30 wpm
Solving the LP:
upper bound = 28 wpm
…
L
30 wpm
Best solution:
30 ≤ S ≤60
Anna Feit – SICSA Summer School on Computational Interaction - 82

83. Example: Keyboard Optimization
[Karrenbauer and Oulasvirta, UIST 2014]
S
S1
S2
xa,1 = 1 xa,1 = 0
xb,1 = 1 xb,1 = 0
S11 S12
ub = 60 wpm
lb = 30 wpm
xb,1 = 1
S22
Solving the LP:
ub= 58 wpm
…
L
30 wpm
Anna Feit – SICSA Summer School on Computational Interaction - 83

84. Example: Keyboard Optimization
[Karrenbauer and Oulasvirta, 2014]
S
S1
S2
xa,1 = 1 xa,1 = 0
xb,1 = 1 xb,1 = 0
S11 S12
ub = 58 wpm
lb = 30 wpm
xb,1 = 1
S22
Solving the LP:
ub= 58 wpm
…
L
30 wpm
Anna Feit – SICSA Summer School on Computational Interaction - 84

85. IP solution to Keyboard Optimization
[Karrenbauer and Oulasvirta, UIST 2014]
42.65 wpm,
within 7% of optimum
Metropolis:
43.1 wpm
Qwerty:
36 wpm
Exact but slow: no optimal solution after 15h
But guranteed to find the optimal solution after exponential
time Anna Feit – SICSA Summer School on Computational Interaction - 85

86. Summary
• Many Optimization methods applied in Keyboard
Optimization
• Heuristic approaches: reasonable solution in fast time
• Metaheuristics: non-zero probability to find the global
optimum in infinite time
• Integer Programming:
• Slow and complex
• Guaranteed optimal solution after exponential time
• Bounds on how good the current best solution is

87. Further readings
• Dunlop M. and Levine J. (2012). Multidimensional pareto optimization of
touchscreen keyboards for speed, familiarity and improved spell checking.
In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (CHI '12). ACM
• Karrenbauer, A., and Oulasvirta, A. Improvements to keyboard optimization with
integer programming. Proceedings of the 27th annual ACM symposium on User
interface software and technology. ACM, 2014.
• Oulasvirta A. et al. (2013). Improving two-thumb text entry on touchscreen
devices. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI '13). ACM
• Wolsey, L. A. (1998). Integer programming
• Zhai, S. , Hunter, M. and Smith B. A. (2000). The metropolis keyboard - an
exploration of quantitative techniques for virtual keyboard design.
In Proceedings of the 13th annual ACM symposium on User interface software
and technology (UIST '00). ACM
Anna Feit – SICSA Summer School on Computational Interaction - 87