The document proposes a new method to identify aggregate scanning strategies from eye tracking data by automatically clustering and aggregating groups of matching scanpaths. It begins by converting scanpaths to sequences of viewed area names. These sequences are then represented in a dotplot where matching sequences are identified using linear regressions. Matching sequences are then used to hierarchically cluster the scanpaths. Aggregate scanning strategies are generated for each cluster and presented interactively to identify strategies within groups and conditions. This allows discovery of both bottom-up and top-down matching strategies to better understand group scanning behaviors.

1. Scanpath Clustering and Aggregation
Joseph H. Goldberg and Jonathan I. Helfman
Applications User Experience
Oracle USA
joe.goldberg@oracle.com; jon.helfman@oracle.com
Abstract data, to identify aggregate scanning strategies within and among
observers, groups, and conditions.
Eye tracking specialists often need to understand and represent
aggregate scanning strategies, but methods to identify similar 1.1 Scanpaths
scanpaths and aggregate multiple scanpaths have been elusive. A
new method is proposed here to identify scanning strategies by Most eye trackers sample gaze locations 50-120 times a second,
aggregating groups of matching scanpaths automatically. A then reduce the samples to sequences of fixations and saccades
dataset of scanpaths is first converted to sequences of viewed called ‘scanpaths.’ Several algorithms are available for this re-
area names, which are then represented in a dotplot. Matching duction [Salvucci and Goldberg 2000]. Scanpaths are essentially
sequences in the dotplot are found with linear regressions, and records of visual attention while performing a task. Scanpaths
then used to cluster the scanpaths hierarchically. Aggregate are typically represented as a single visual image with a se-
scanning strategies are generated for each cluster and presented quence of connected nodes and edges, where node diameter is
in an interactive dendrogram. While the clustering and aggrega- often displayed as proportional to fixation duration (within de-
tion method works in a bottom-up fashion, based on pair-wise fined minimum and maximum bounds) and edges connect suc-
matches, a top-down extension is also described, in which a cessive fixations. Figure 1 displays a hypothetical scanpath with
scanning strategy is first input by cursor gesture, then matched 5 fixations and 4 saccades. Scanpath tracings from real data can
against the dataset. The ability to discover both bottom-up and be quite complex, with frequent revisiting of AOIs and overlap-
top-down strategy matches provides a powerful tool for scanpath ping saccades.
analysis, and for understanding group scanning strategies.
CR Categories: H.5.2. User Interfaces: Evalua-
tion/methodology, /Theory and methods, I.5.3. Pattern Recogni-
tion: Clustering/ algorithms, I.5.5. Pattern Recognition: Imple-
mentation/Interactive systems
Keywords: Eye Tracking, Usability Evaluation, Pattern
Analysis, Dotplot, String Analysis, Sequential Clustering, Se-
quence Analysis Figure 1. Scanpath showing a sequence of fixations and sac-
cades. Longer fixation durations are denoted by larger diameter
circles.
1 Introduction
1.2 Areas of Interest
Eye tracking studies collect large amounts of fixation and sac-
cade data while observers complete tasks or view scenes. To Areas of interest (AOIs) are regions that associate scanpath data
improve the design of software and visual media, investigators with features of the scanned scene, including task target areas.
make usability inferences by comparing task or interface condi- Recent studies have defined AOIs, for example, as separate list-
tions between groups of users. The challenge for the researcher ings on a search results page [Cutrell and Guan 2007] and as
is to quickly comprehend both individual and group scanning lines of text [Beymer and Russell 2005]. Scanpath data is typi-
strategies in each tested condition. cally combined with a set of AOIs by determining the contain-
ment of fixations within AOIs. AOI statistics can include fixa-
The number of studies using eye tracking methods have in- tion time within AOIs, time to first view an AOI, order of AOI
creased dramatically in the past few years, due to improvements viewing, transitions among AOIs, time to reach specific AOIs,
in hardware, ease of calibration, and rapidly reported results. and other data.
While new eye-tracking technologies have solved many niggling
issues, there is still a significant gap between the results and a Figure 2 illustrates the relationship between scanpaths and AOIs.
real understanding of users’ visual strategies while solving tasks. It shows a design wireframe with 5 AOIs, A-E, and a back-
The present paper proposes a method to cluster users’ scanning ground area, F. Two scanpaths are shown. One scanpath, shown
Copyright © 2010 by the Association for Computing Machinery, Inc. as solid red, contains 9 fixations. When each fixation is repre-
Permission to make digital or hard copies of part or all of this work for personal or sented by its AOI identifier, the solid red scanpath’s fixations
classroom use is granted without fee provided that copies are not made or distributed
for commercial advantage and that copies bear this notice and the full citation on the
form the sequence: CCABDFEFF. The other scanpath, in dashed
first page. Copyrights for components of this work owned by others than ACM must be blue, contains 8 fixations, which form the sequence: DCDAB-
honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on FEE. Note that some analysis methods eliminate repeated, con-
servers, or to redistribute to lists, requires prior specific permission and/or a fee. secutive AOI visits, and may also eliminate background areas,
Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
which would reduce these strings to CABDE and DCDABE,
permissions@acm.org.
ETRA 2010, Austin, TX, March 22 – 24, 2010.
© 2010 ACM 978-1-60558-994-7/10/0003 $10.00
227

2. respectively. Although not immediately obvious, the two scan- Sequence alignment techniques first attempt to align sequences
paths share the sequence of AOIs: CABFE. Techniques for as closely as possible, before computing their dissimilarity [Jo-
identifying shared sequences (also called ‘sequential matches’ or sephson and Holmes 2002]. Alignment techniques, however,
‘alignments’) are described in the following section. depend on initial pair-wise alignments and have difficulties
aligning sequences of vastly different lengths or with very long
distinct subsequences [Higgins, et al. 1996].
Scanpath sequences typically consist of AOI names as tokens,
but other tokens are also possible, such as:
• Saccade Angles. By coding each successive saccade as an
angular direction of travel, path direction sequences can
be compared. The direction of travel could be measured in
either absolute angles (with respect to a common coordi-
nate system) or relative to the current direction of motion.
• Fixation Durations. Longer fixation durations may indi-
cate greater stimulus complexity or observer confusion
[Goldberg and Kotval 1999; 1998]. Scanpaths with longer
fixation durations may indicate problematic aspects of an
interface.
Figure 2. Two hypothetical scanpaths browsing a page with six • Saccade Distances. Coding the distances between fixa-
defined AOIs. Each scanpath started at the ‘+’. tions as tokens could potentially locate denser areas of in-
terfaces that result in very short saccades.
AOIs may overlap or may be nested within each other. When
AOIs overlap it is still possible to establish a process for deter- Finding scanpath differences among groups of users and condi-
mining which AOI corresponds to a particular fixation. One tions is somewhat more difficult than comparing two scanpaths.
such process, for example, is to sort AOIs by area, smallest first, One algorithm first computes the pairwise sequence alignment
and then iterate through the sorted list, stopping at the first (i.e. difference between scanpaths within each compared group, then
the smallest) AOI containing the fixation. computes the difference between groups. A reference distribu-
tion is generated by Monte Carlo simulation, allowing the defi-
Spatial clustering techniques can also be used to empirically nition of statistically significant differences among scanpaths
define AOIs [Salvucci and Goldberg 2000], in one case from [Feusner and Lukoff 2008]. Matching alignments from multiple
over 5000 scanpaths [Wooding 2002]. A mean shift approach sequences may also represent the ‘averaged’ scanpath from a set
can generate AOIs by iteratively moving sampled gaze locations of users [Hembrooke, et al., 2006]. Unsupervised learning algo-
to locations of higher gaze density on a page [Santella and De- rithms can classify natural groups of scanpath sequences, form-
Carlo 2004]. Similarity coefficients within and between partici- ing a hierarchical clustering of sequences in a hierarchy tree
pants and images can then be used to generate a parsing tree to [West, et al. 2006].
assign AOIs automatically [Heminghous and Duchowski 2006].
Hidden Markov modeling has been used to model scanpaths, by
1.3 Comparing and Aggregating Scanpaths developing probability distributions for sequences of AOI transi-
tions [Salvucci and Goldberg 2000]. While these models can
Although individual scanpaths can appear to be extremely ran- determine the overall transition probabilities among AOIs, the
dom and noisy, methods are available to compare them, and to composite probabilities don’t necessarily represent the aggregate
aggregate them to find group trends or to uncover cognitive sequence across observers. This is due, in part, because Hidden
strategies. Markov models are usually only first or second order, including
only the prior one or two fixations in successive probability
String comparison methods are often used to compute the simi- estimates [Josephson and Holmes 2002].
larity between two scanpaths. Scanpaths are first coded as a
string of AOI names, numbers or letters. The Levenshtein dis- Aggregate representations of group sequential scanning strate-
tance between the strings is then computed as the minimum gies are not easily developed from current methods. While
number of substitutions, insertions, and/or deletions required to heatmaps provide a view of aggregated visual attention over a
transform one string into the other [Levenshtein 1966; Smith and specific time period, they cannot adequately convey user and
Waterman 1981]. Dynamic programming methods have been group scanning strategies. A single heatmap cannot show
used to help determine minimum Levenshtein distances between changes over time and, therefore, cannot show sequential infor-
scanpath sequences [Josephson and Holmes 2002]. A cost is mation about scanpath fixations. Sequential analysis of scan-
assigned to each of the operations to result in a dissimilarity paths is required to understand the flow of visual attention on a
value that ranges from 0 (identical scanpaths) to 1 (completely task.
different scanpaths). For example, Figure 2’s blue scanpath
(DCDABFEE) can be transformed into the red scanpath Current scanpath comparison and aggregation methods suffer
(CCABDFEFF) with 5 substitutions and 1 addition, a total of 6 from several drawbacks:
operations. Costs may be differentially assigned to each opera-
tion type, but can be difficult to define objectively, especially • Scanpath Length. Scanpaths of different lengths have
when scanpaths are of vastly different lengths. very different similarity scores, and comparison of scan-
paths of differering lengths can throw off alignment calcu-
lations.
228

3. • Intervening Tokens. In some cases, different length that smaller AOIs will result in greater precision for representing
scanpaths are due to non-matching tokens that interrupt the aggregate strategy.
what would otherwise be long matching sub-sequences.
The pair-wise comparison of scanpaths is scaled to an entire
• Transformation Costs. The relative cost of string opera- study by: (1) concatenating the scanpath AOI sequences into one
tions is hard to define objectively, and can greatly alter the sequence, (2) plotting the sequence with a dotplot, and (3) using
similarity metric. linear regression to identify pair-wise matches in the plot. Large
• Scaling. String analysis methods work well for comparing datasets result in a dense dotplot, such as that shown in Figure 4.
a limited number of sequences, but they don’t scale effec- Green lines indicate boundaries between scanpaths. Red lines
tively to comparisons of all scanpaths in all conditions of indicate a sequential match between two scanpath segments. The
a study. dotplot is symmetric, so only matches in the upper half are
shown. The plot density has also been relieved somewhat using
• Sequential Aggregation of Multiple Scanpaths. While inverse frequency weighting, in which extremely frequent
heatmaps aggregate positional fixation information, they matches (e.g., revisits to the background AOI) have been down-
do not represent sequential scanning sequences effec- weighted [Church and Helfman 1993].
tively.
1.4 Dotplots
A dotplot is a graphical technique for visualizing similarities
within a sequence or between two or more concatenated se-
quences. Dotplots have been used to find insertions, deletions,
matches, and reverse matches in genetic sequences [Huang and
Zhang 2004], and have been applied to finding repetition in
literature, detecting plagiarism, aligning translated documents, A. B.
and identifying copied source code [Church and Helfman 1993;
Helfman 1994]. The dotplot can also be considered as an inter- Figure 3. Finding matching sequences using a dotplot. A. Dot-
mediate representation that is used for finding patterns algo- plot of sequences from Figure 2, with collinear data forming a
rithmically; it does not need to be exposed to an eye tracking matching sequence. B. Aggregate strategy sequence ‘CABFE’
researcher directly. plotted on task background.
Advantages of dotplots over string editing and alignment tech- Additional dotplots may contain only selected scanpaths. An
niques include: example is shown in Figure 5. Regression lines are again shown
in red, with stronger matching (i.e., a greater number of match-
• Dotplots do not require a cost matrix to judge the similar- ing AOI names) noted by darker cell backgrounds. Regressions
ity between two strings. with negative slopes indicate forward matching patterns (e.g.,
CABFE from Figure 3). Positive slopes indicate reverse match-
• Dotplots support the calculation of millions of matches at ing patterns (e.g., EFBAC).
near interactive rates by pre-computing positions and fre-
quencies of distinct tokens [Church and Helfman 1993].
• Dotplots provide a visual representation that is useful for
interactive exploration and validation.
• Doplots can robustly handle non-matching tokens that in-
terrupt what would otherwise be long matching sub-
sequences.
2 Pattern Analysis Method
2.1 Sequence Matching using Dotplots
Dotplots can be applied to pattern finding among scanpaths by
listing one sequence of scanpath AOI names on the horizontal
axis, and one sequence on the vertical axis of a matrix. A dot is
placed in the intersecting cells of any matching AOIs. Figure 3A
shows an example of a dotplot for the scanpaths from Figure 2.
The red scanpath sequence (CCABDFEFF) is plotted on the
horizontal axis, and the blue sequence (DCDABFEE) is on the
vertical axis of the matrix. A linear regression identifies a se-
quence of five dots, representing the sequence of matching
AOIs, CABFE.
An aggregate scanning strategy can now be represented by the Figure 4. Dotplot resulting from concatenated sequence of
sequence of matching AOIs, with each aggregate fixation lo- scanpaths across study participants. Green lines separate each
cated near the center of its associated AOI (Figure 3B). Note scanpath, and red marks indicate matching sequences.
229

4. The matching sequences that are discovered by the dotplot re-
gression now become a reference for hierarchical clustering and
aggregation. The dotplot and its matches need only be computed
once for a dataset, a computational advantage.
2.3 Hierarchical Strategy Clustering
Matching scanpaths can be hierarchically clustered to find strat-
egies – sets of scanned regions that match increasingly greater
numbers of scanpaths and individuals. The process starts by
considering every individual scanpath as a leaf node cluster. At
each iteration of the process, the two ‘closest’ clusters are
merged into a single cluster. Clustering ends when only one
cluster remains. The concept of ‘closeness’ is determined from
the dotplot matches between the sequences associated with each
cluster according to Algorithm 1. Because closeness between
clusters must be determined repeatedly, the algorithm includes a
basic optimization: closeness results between a pair of clusters
are cached using a key formed from the two cluster IDs, and
closeness is only calculated if it is not already in the cache.
Algorithm 1. Computation of cluster distance.
if(no matches between cluster sequences)
Figure 5. Subset of previous dotplot, showing matching se-
quences as red lines. Darker cells represent those patterns that distance = MAXIMUM; //not close
matched a greater number of AOI names.
else distance =
2.2 Regressions on Dotplots
matchLength/maxMatchLength;
Like heatmaps and other research-oriented visualizations, dot-
plots show patterns that are recognized by the human visual where:
system quickly, but are much less discernable to a machine. The
present approach uses linear regression to pull out statistically • ‘matchLength’ is the number of sequentially matching
significant sequence matches from a dense dotplot. It uses ad- AOI names between the pair of sequences,
justable threshold R2 and residual data distances to find signifi-
cant matches, as follows: • ‘maxMatchLength’ is the largest number of sequen-
tially matching AOI names in the entire dataset (i.e.
1. Start with an inverse-frequency filtered dotplot comparing between each pair of sequences).
multiple concatenated scanpath sequences.
At each step of the clustering process, the two closest clusters
2. Iterate over the dotplot cells for each pair of scanpaths, are merged into a new cluster. The merged cluster stores a
identifying the ‘darkest’ dots in the cell, which correspond unique name, references to its two child clusters, and the ‘dis-
to sub-sequences with significant matches. High-pass fre- tance’ between the child clusters. For clustering to continue,
quency thresholds of 0.1-0.5 work well, using a 1µ+1σ cri- each new cluster must also be assigned a sequence of AOI
names. One possibility would be to form a sequence from the
terion.
matching AOI names. We have found, however, that because the
matches are typically shorter than either child sequence, this
3. Fit a linear regression to the darkest points, moving to the approach guarantees that cluster hierarchies will be small and
next cell if R2 value is too low (e.g., <0.5). shallow. In contrast, we have found that deeper clusters are
obtained when the merged cluster is assigned one of the child
4. Compute a 1µ+1σ threshold from the Euclidean distances sequences. Because each child sequence contains the matching
between the regression line and the data, and identify those sequence of AOI names, either one will match at least as many
other scanpaths as the matching sequence. In fact, we have
data within the threshold.
found that the deepest clusters are obtained when the merged
cluster is assigned the child sequence that matches the most
5. Recompute the regression only for data within the thresh- other sequences in the dataset.
old. Dots falling on (or close to) the regression line are con-
sidered elements of the matching sequence. 2.4 Aggregation of Strategies
6. (Optionally) return to (2) after removing matching dots, and An aggregate representation is assigned to each cluster of scan-
re-compute the regression to find further matches between ning strategies. The representation uses the sequence of match-
ing AOI names between the sequences associated with the clus-
the pair of scanpaths.
ter’s two children. If the sequences associated with the cluster’s
children have no matches, the aggregate strategy uses the se-
230

5. quence of the child cluster with the greatest distance value (i.e. relative angle radial histograms can also provide informa-
the child cluster with the best matching children of its own). tion about the shape of a scanpath or aggregate strategy.
Choosing an aggregate strategy in this way ensures that the ag-
gregates will correspond to the closest actual scanpath matches
in their clusters whenever possible. The choice of aggregate
strategy representation depends entirely on the sequential
matches found between scanpath pairs: if the scanpaths have few
matches, the results of the clustering and aggregation methods
will be dubious. Dotplot regression is parameterized, however, Figure 6. Compact visual scanpath representations: A. Scaled
and it is usually possible to vary the parameters to find addi- trace, B. Vertical time expansion, C. Horizontal time expansion,
tional sequential matches (See Section 3.2). The present aggre- D. Radial histogram, based on absolute angles, E. Radial histo-
gation method is a proposed solution, and further evaluation is gram, based on relative angles.
required to fully judge the validity of this approach.
3 Implementation of Eye Tracking Pattern
2.5 Visual Representation of Aggregate Analysis Tool
Strategies
The dotplot-based, pattern analysis method was implemented in
Aggregate sequences can be represented in multiple ways to a tool that clusters, aggregates, and represents scanning strate-
highlight different properties. Five example visual representa- gies across datasets of scanpaths collected from multiple studies.
tions are shown in Figure 6, and described below. Further back- The tool was written in HTML/Javascript, and runs within the
ground on these representations are available in a companion Firefox and Chrome web browsers. Sub-queries may be con-
paper [Goldberg and Helfman, in press]. ducted within each dataset, such as including only correct trials
or only certain targets. The tool is intended for usability profes-
• Scanpath Trace. Like those shown by Yarbus [1967], sionals who are using eye tracking methods to discover scanning
these scaled down scanpath traces show the general distri- strategies of groups of individuals, and to compare design alter-
bution and extent of scanning. While fixations are not ex- natives.
plicitly shown, a general impression of fixation density,
scanpath complexity, and individual and aggregated scan- A dataset is developed from data that are exported from a Tobii
ning strategy can be obtained (Figure 6A). T60 eye tracker, running Tobii Studio 1.15 software. The fol-
lowing elements are imported into the pattern analysis tool:
• Vertical Time Expansion. A record of horizontal scanning
is provided by substituting time for the vertical coordinates • Background image from each task.
of each fixation, allowing for analysis of horizontal scan-
ning trends and backtracking frequency (Figure 6B). The • AOI names and coordinates for background task images.
analyst can determine, for example, whether scanning was • List of tasks by participants, including (if available) in-
primarily in one direction or whether it moved right and left tended target AOI, AOI of selected response, and whether
frequently. the response was an error.
• Horizontal Time Expansion. Replacing the horizontal • List of fixations by participant by task, including time-
coordinates of each fixation with time provides a record of stamp, fixation duration, and x/y location.
vertical scanning within the overall scanpath or aggregate
strategy (Figure 6C). Similar representations have been 3.1 Analysis Parameters
used, in larger scale, for scanning lines of computer code
[Uwano, et al. 2006] and listings of search results [Raiha, et A dialog panel (Figure 7) is available in the tool to modify pa-
al. 2005, Aula, et al. 2005]. rameters of the regressions, in order to increase or decrease the
• Radial Histogram—Absolute Angles. A radial histogram number of matches found in the dotplot. Clusters that are too
shallow to be meaningful can be modified by resetting the re-
counts the number of saccade angles associated with each
gression values and re-conducting the regression. More liberal
of several predefined angle ranges (Figure 6D). Angles are
regression parameters result in a greater number of matching
measured with respect to an absolute scale (e.g. counter-
sequences in the dotplot, which in turn can result in greater
clockwise, with 0o on the positive X-axis). A red dot indi- matching and deeper clusters. More liberal regressions, how-
cates the mean shift, computed as the spatial average of the ever, also run the risk of matches between insignificant or overly
histogram. This representation provides angular informa- frequent sequences, resulting in non-representative aggregate
tion about the scanpath, including number of different an- strategy clusters. The proper settings of regression parameters
gles, and whether scanning was direct or convoluted. result in conservative clusters that are sufficiently deep. Regres-
• Radial Histogram—Relative Angles. Similar to Absolute sion parameters that can be modified include:
Angle radial histogram, except each angle is determined
relative to its prior saccade direction (Figure 6E). Moving • Xµ+Yσ, where X and Y adjust the threshold Euclidean
to the left from the prior direction defines a 90o relative an- distance of data points, to be included in the regression.
gle, but could define a very different absolute angle. Radial Choices include 0, 0.5, 1, 2. Larger values result in greater
histograms made from relative angles are more sensitive to number of significant regressions.
small directional shifts in scanpaths, than are those made • Minimum number of collinear points required for a suc-
from absolute angles. Comparison between absolute and cessful regression. Choices include 3, 4, 5, 10. Larger val-
ues result in fewer, but longer, sequential matches.
231

6. • Minimum R2 value required for a significant regression. ters at a specified depth level. Scanpath and aggregate strategy
Choices include: 0, .001, .01, .1, .5. Larger values result in traces (in red) are also shown within the dendrogram.
fewer sequential matches.
Selecting a cluster in the dendrogram updates a detail pane (Fig-
• Qµ+Rσ, where Q and R adjust the threshold distance of ure 9) with both the aggregated strategy and the individual scan-
data points from the initial regression line that are used to paths that match the aggregate strategy. Figure 9 displays a se-
compute the final regression line. Choices include: 0, .5, 1, lected cluster (-31), its aggregate strategy cluster, and the 7
2. Larger values result in greater number of significant scanpaths contained in the cluster. The aggregate strategy is also
matches. shown on the task background image in Figure 10, both with
(Figure 10A) and without (Figure 10B) its constituent scanpaths.
The aggregate matching strategy necessarily provides the lowest
common denominator among the matching scanpaths, so ap-
pears as a simpler, more compact scanpath. In this case, it re-
veals a matching strategy that starts at the left-most column
header, scans to the middle of the table, then scans leftward. In
general, larger clusters result in less complex aggregate scanning
strategies that match more scanpaths than those derived from
smaller clusters.
Figure 7. Parameter setting panel for dotplot analysis in the
pattern analysis tool. Default values are shown.
The content of the sequences evaluated by the dotplot can also
be selected. Sequence content choices include region names,
saccade lengths, fixation durations, absolute saccade angles, and
relative saccade angles. Although not the focus of the present
paper, sequences composed of these alternate metrics provide
additional strategic value for eye tracking analysis. While se-
quences of AOI names are dependent on the specific placement
of AOIs on a background image, sequences of saccade angles
can be matched and aggregated independently from background
images. For example, the sequence 90o, 0o, 270o (absolute an-
gles) defines a scanpath that moves up, right, then down, regard-
less of AOIs or background image. Matching scanning se- Figure 8. Interactive dendrogram shows results of the clustering
quences could be identified across images and tasks using sac- process. Clicking selects a cluster and its aggregate strategy,
cade angles. Similarly, sequences of fixation durations could which is viewed in other panes. Darker circles indicate stronger
potentially identify cognitive states or task complexity [Marshall matches, and a cut-plane of strategy clusters is selected by
2007, Goldberg and Kotval 1999]. dragging the vertical dotted line.
When based on quantitative metrics, sequences for dotplot
analysis and scanpath representation are tallied into user-
definable bins. For example, saccade angles have default bin
sizes of 20o, but larger or smaller sizes can be selected. Larger
bin widths provide coarser analyses, but generally make it easier
to form clusters of similar sequences. Smaller bin widths pro-
vide a higher resolution of analysis, but may make it more diffi-
cult to form clusters.
3.2 Interactive Dendrogram
The results of hierarchical clustering are shown in an interactive
dendrogram, a type of binary tree diagram (Figure 8). The hier-
archical pattern of sequential matches is represented graphically
by the linear branches of the tree diagram. Matching distance
(‘closeness’) within each cluster is represented by a number (0-
1) and a dot, where darker dots are associated with stronger
matching distance. Strategy clusters can be viewed by vertical
scrolling, and any cluster can be selected by clicking on it. A Figure 9. Detail pane showing selected aggregate strategy clus-
dotted blue line can also be dragged horizontally to slice across ter (-31) and the 7 individual scanpaths that matched the aggre-
the cluster hierarchy, allowing the selection and viewing of clus- gate strategy. Additional visual representations are also pro-
vided to the right of the scanpaths.
232

7. 3.3 Positing Strategies via Gesture matching would find other similar saccade angle sequences,
saccade distances, fixation durations, or other metrics.
Presentation of the pattern analysis tool to this point has focused
on its ability to find and cluster matching strategy sequences 4 Discussion
within an eye tracking dataset. This ‘bottom-up’ analysis essen-
tially finds ad hoc matches in a dataset. The pattern analysis tool
also includes a method to input (or posit) a ‘top-down’ strategy,
4.1 Pattern Analysis Method
which generates a new dotplot and regressions. The new strategy
may be input by dragging the mouse cursor over a scaled-down The present work was motivated by the need for a method to
version of the background task image. The input strategy is con- extract group scanning strategies from large eye tracking data-
sets. While heatmaps and cumulative fixation times provide
verted to a sequence of AOI names, much like a regular scan-
path, and a dotplot is used to match the posited strategy against valuable insight into which AOIs are viewed, they do not pro-
the dataset. Scanpaths that match the strategy are treated as a vide sufficient detail about the sequential scanning strategies on
a page. Sequential information is needed to understand common
new query result, clustered, and presented in a new interactive
dendrogram. scanning behavior between groups of observers, between differ-
ent interface designs, or between interface conditions.
Although other researchers have also proposed solutions to the
problem of defining the aggregate scanpath for a group [e.g.,
Hembrooke, et al. 2006, West, et al. 2006, Feusner and Lukoff,
2008], the present approach provides certain advantages. (1) It is
scalable to large numbers of participant scanpaths. (2) The un-
derlying dotplot representations have already been validated for
sequence matching in other domains, such as biology and pro-
gram code comparison. (3) Unlike other string analysis and se-
quential alignment methods, no cost matrix is required for dot-
Figure 10. Background task image with superimposed aggre- plots. (4) Matching sequences can be discovered even with the
gate strategy with (A), and without (B) individual scanpath data. presence of intervening non-matching tokens. (5) The method is
efficient, requiring that a dotplot be computed only once for a
As an example, Figure 11 displays a cursor-input path that traces dataset. (6) The matching process is highly tunable by modify-
a scanpath traveling down the left column, then rightward across ing regression properties. Although default values usually result
the bottom of the presented table. Upon mouse-up, the query is in a wide range of discovered strategies, adjusting these values
input, and a new result set is clustered and presented in a den- can create deeper clusters with scanning strategies that match
drogram. Selection of a cluster in the dendrogram, shown at the greater numbers of scanpath segments across a dataset. (7) The
left in Figure 12, displays the aggregate strategy, along with the dotplot method can be extended to other sequences besides AOI
four matching scanpaths. The aggregate strategy is shown super- names, such as saccade angles, or fixation durations. Each of
imposed on the task background image at the right of the figure. these can provide additional information about strategies.
In this case, the aggregate strategy matched a portion of the
downward scanning, and much of the rightward scanning. 4.2 Pattern Analysis Tool
The methods described in this paper were implemented within
an eye tracking pattern analysis tool to provide researchers a
way to rapidly discover sequential group scanning strategies in
studies. While details about the dotplot matching process are
easily hidden from an end-user, a selectable dendrogram allows
interactive exploration of the hierarchically clustered scanning
strategies. A range of scanning strategies and cluster depths are
visible by scrolling the dendrogram. Slicing the dendrogram at a
Figure 11. Cursor-input query to filter the dataset. particular depth provides a rapid way to control which clusters
are shown in a detail panel. Scanpaths and aggregate strategies
can be selected for display on the background task image. The
pair-wise sequential matches that underlie the aggregate strate-
gies can also be easily viewed.
Graphic representations of scanpaths and aggregate strategies in
our tool extend others’ ideas and introduce new functionality.
They allow the researcher to quickly visualize similarities and
differences between scanpaths. The representations provide a
Figure 12. Resulting portion of dendrogram, scanpaths, and
diagnostic aid to help understand why scanpaths were included
aggregate strategy for selected cluster, based upon the input
within selected clusters. Although not explored in the present
posited strategy of Figure 11.
paper, features of these representations (e.g., bin lengths in ra-
dial histograms, mean shift distances from origins) can poten-
Although not yet implemented in the present tool, it is also pos- tially aid the algorithmic discovery of larger scanning strategies,
sible to convert the gesture input to other sequences besides AOI such as clockwise, downward, rightward, even more complex
names. Examples include saccade angles, saccade distances, scanning tendencies [Goldberg and Helfman, in press].
fixations durations, and other metrics. In these cases, dotplot
233

8. Gesture-based strategy input supports a valuable top-down GOLDBERG, J. AND HELFMAN, J. In Press, in press. Visual Scan-
analysis by imposing a hypothesized strategy, then discovering path Representation. In Proc. ETRA 2010, ACM Press.
matching sequential strategies in the dataset. Top-down analysis
enables the researcher to find out whether observers collectively GOLDBERG, J., AND KOTVAL, X. 1998. Eye Movement-based
scanned a stimulus image in a specified way, such as left column Evaluation of the Computer Interface. In Kumar, J. (Ed.),
downward, or bottom row rightward. Ultimately, a library of Advances in Occupational Ergonomics and Safety, IOS
hypothesized strategies can be developed and saved, then com- Press, 529-532.
pared against scanpath collections. A higher-level description of GOLDBERG, J., AND KOTVAL, X. 1999. Computer Interface
individual and aggregate scanning strategies could ultimately Evaluation using Eye Movements: Methods and Constructs.
result from a pattern of these matches. Int. J. Industrial Ergonomics, 14, 631-645.
4.3 Areas for Future Enhancements HELFMAN, J. 1994. Similarity Patterns in Language. In Proc.
1994 IEEE Symp. Vis. Lang., IEEE Press, 173-175.
Although our pattern analysis tool makes a large contribution to HEMBROOKE, H., FEUSNER, M., AND GAY, G. 2006. Averaging
the empirical analysis of collections of scanpath data, there are Scan Patterns and What They Can Tell Us. In Proc. ETRA
several areas for improvement, such as: 2006, ACM Press, 41.
• The current hierarchical clustering process forces each HIGGINS, D., THOMPSON, J., GIBSON, T. 1996. Using CLUSTAL
scanpath to be in only one cluster. Scanpaths, however, for Multiple Sequence Alignments. Methods Enzymol., 266,
may match strategies from multiple clusters. While our 383-402. http://www.ncbi.nlm.nih.gov/pubmed/8743695.
technique of associating clusters with sequences that
HUANG, Y., AND ZHANG, L. 2004. Rapid and Sensitive Dot-
match the most other sequences in the dataset is an at-
Matrix Methods for Genome Analysis. Bioinformatics, 20,
tempt to mitigate this problem, future work will evaluate
4, 460-466.
alternative modifications of the clustering algorithm to al-
low non-matching sub-sequences of scanpaths to contrib- JOSEPHSON, S., AND HOLMES, M. 2002. Visual Attention to Re-
ute to other clusters. peated Internet Images: Testing the Scanpath Theory on the
World Wide Web. In Proc. ETRA 2002, ACM Press, 43-49.
• The present strategy matching process is efficient and can
form relatively deep clusters. As one explores a strategy LEVENSHTEIN, V.I. 1966. Binary Codes Capable of Correcting
cluster tree, parent strategies typically look like common Deletions, Insertions and Reversals. Doklady Physics, 10,
subsets of child strategy representations. However, with 707-710.
the present algorithm, it is possible to find parent strate-
gies that resemble deeper ancestors, rather than direct MARSHALL, S.P. 2007. Identifying Cognitive State from Eye
child strategies. Metrics. Aviation, Space, and Environmental Medicine,
78(5), B165-B185.
• While the pattern analysis tool presently supports match-
ing and clustering of any sequences, it can only define ag- RAIHA, K., AULA, A., MAJARANTA, P., RANTALA, H., AND KOI-
gregate strategies for clustered sequences of AOI names. VUNEN, K. 2005. Static Visualization of Temporal Eye-
Aggregate strategies that are based on sequences of fixa- Tracking Data. In Proc. INTERACT 2005, Int. Fed. Info
tion durations, absolute/relative saccade angles, saccade Proc, 946-949.
lengths, and other properties can potentially match scan- SALVUCCI, D., AND GOLDBERG, J. 2000. Identifying Fixations and
paths that are not tied to particular tasks and background Saccades in Eye-tracking Protocols. In Proc. ETRA 2000,
images. Future work will investigate clustering and aggre- ACM Press, 71-78.
gation methods based upon these and other criteria.
SANTELLA, A. AND DECARLO, D. 2004. Robust Clustering of Eye
Movement Recordings for Quantification of Visual Interest.
In Proc. ETRA 2004, ACM Press, 1-8.
References
AULA, A., MAJARANTA, P., AND RAIHA, K. 2005. Eye-tracking SMITH, T. AND WATERMAN, S. 1981. Identification of Common
Reveals the Personal Styles for Search Result Evaluation, Molecular Subsequences. J. Molecular Biol., 147, 1, 195-
In Proc. INTERACT 2005, Int. Fed. Info Proc, 1058-1061. 197.
BEYMER, D., AND RUSSELL, D. 2005. WebGazeAnalyzer: A Sys- UWANO, H., NAKUMURA, M., MONDEN, A., AND MATSUMOTO, K.
tem for Capturing and Analyzing Web Reading Behavior 2006. Analyzing Individual Performance of Source Code
using Eye Gaze. In Proc. CHI 2005, ACM Press, 1913- Review Using Reviewers’ Eye Movement. In Proc. ETRA
1916. 2006, ACM Press, 133-140.
CHURCH, K., AND HELFMAN, J. 1993. Dotplot: A Program for WEST, J., HAAKE, A., ROZANSKI, E., AND KARN, K. 2006. EyePat-
Exploring Self-similarity in Millions of Lines of Text and terns: Software for Identifying Patterns and Similarities
Code. J. Comp. and Graphical Statistics, 2, 153-174. across Fixation Sequences. In Proc. ETRA 2006, ACM
Press, 149-154.
CUTRELL, E., AND GUAN, Z. 2007. What are You Looking For?
An Eye-tracking Study of Information Usage in Web WOODING, D. 2002. Eye Movements of Large Populations: II.
Search. In Proc. CHI 2007, ACM Press, 407-416. Deriving Regions of Interest, Coverage, and Similarity us-
ing Fixation Maps. Behavior Research Methods, Instru-
FEUSNER, M., AND LUKOFF, B. 2008. Testing for Statistically ments & Computers, 34, 4, 518-528.
Significant Differences Between Groups of Scan Patterns.
In Proc. ETRA 2008, ACM Press, 43-46. YARBUS, A.L. 1967. Eye Movements and Vision. New York,
Plenum Press.
234