1. Visual Analytics as a Cognitive
Science
Brian Fisher
SFU School of Interactive Arts & Technology and
Program in Cognitive Science
UBC Media & Graphics Interdisciplinary Centre (MAGIC)

2. My Background
• UG Biology, Medical Biophysics tech at CWRU Med
• Scientific Programmer, Varian
• Ph.D Experimental Psychology, UCSC
• UWO & Rutgers Centres for Cognitive Science
• Human-Information Interaction AKA “Cyberpsychology”
• Institute for Robotics and Intelligent Systems Networks of Centres of
Excellence “a Cognitive Basis for the Design of Intelligent User
Interfaces to Complex Systems” (SFU-UWO)
• 1999- Associate Director, UBC Media And Graphics
Interdisciplinary Centre, Computer Science, Psychology,
Institute for Computing, Information and Cognitive Systems
• 2004- SFU School of Interactive Arts and Technology and
Program in Cognitive Science

3. My double life
• Ph.D in Experimental Psych
• Postdoc w Cognitive
Science society founder &
president
• Psychonomics Fellow
• VIS-related symposia at
Cogsci, & APS, papers on
cogsci of interaction
• Fuzzy-logic/Bayes models
• Postdoc funded by Inst. for
Robotics and AI
• VAST SC, VEC, VACCINE
• Led Dagstuhl “Interaction with
Information for Visual
Reasoning”, Cogsci-based
papers at VIS, CHI, BELIV.
Participants

4. “Big Data”: Volume, Velocity &
Variety
• In 2011, data expected to be
about 1.8 zettabytes (1021).
• In 2013, Internet traffic to reach
667 exabytes (1018)/yr.
• Comparison: US Library of
Congress
is ~10 petabytes (1015).
• “By 2018, the United States alone could face a shortage
of 140,000 to 190,000 people with deep analytical skills
as well as 1.5 million managers and analysts with the
know-how to use the analysis of big data to make
effective decisions.” (McKinsey Global Institute 2011)

5. Challenges for computational
approaches
• “3 Vs” challenge
• Relevance, validity, reliability of data uncertain
• Insufficient time to reach solution
• Model challenges
• Multiple models to chose from
• Assumptions may or may not hold
• “Wicked problems” challenge (Rittel)
• Lack criteria to evaluate solution
• Each problem is unique (no population)
• Problem is not understood until solution is found
The information needed to understand the problem depends upon
one’s idea for solving it. -- Rittel & Webber 1973

6. The ultimate challenge
In business, government & the professions
specific people are responsible for the
decisions that are made and how they are
executed. These people are derelict in their
duties if they only accept what the model tells
them. Either we re-engineer society to accept
a computer model as the ultimate authority or
we find a way that human decision makers
can exercise due diligence for computational
as well as informational aspects of the
problem.

7. An prehistory of visual
analytics

8. Visualization history
• NSF meeting: “Visualization
in Scientific Computing”
• Nov. 1987 Computer
Graphics
• First IEEE Visualization (now
SciVis) conference in 1990
“The purpose of [scientific] computing is insight, not numbers.”
Richard Hamming
“Visualization is a method of computing.” Authors of report

9. Information visualization
• 1990 Conference
on diagrammatic
reasoning
• 1995 InfoVis
Conference
• “Information
Visualization: Wings
for the Mind”
Keynote by Stuart
Card

10. Stuart Card’s view
•Increase the memory & processing
resources available to users
•Reduce the search for information by using
visual representations to enhance the
detection of patterns
•Engage perceptual inference operations
•Use perceptual attention mechanisms for
monitoring
•Support manipulation of information

11. Infographics
represent
information (data,
knowledge,
opinions, etc.) in
context to
support
understanding
Visual thinking:
Infographics

12. Visualization literacy
• Build a “language” of
collections of images
that support thinking
• Diagrammatic
reasoning science of
how we understand
complex diagrams

13. Thinking as “smart seeing and Projecting”
Actively looking at external representations and
projecting onto them makes us more powerful thinkers
than thinking in our heads alone.
David Kirsh example

14. Design Approach:
Bertin
6
Bertin Semiologie
Graphique (1967)
Cartographer, built
description of how data
should be represented
visually
Jock Mackinlay, Stanford
Ph.D. dissertation
Tableau software, used for
our work with Boeing etc.

15. Design Approach:
Edward Tufte

16. Big Data Example (Amaral)
Metabolism
Experts are overwhelmed by sheer
volume and complexity of data

17. Cartographic Representation
(Amaral)
Guimera & Amaral, Nature 433, 895 (2005)

19. http://www.visual-literacy.org/periodic_table/periodic_table.html

20. Role of psychology in VIS
• Design based on theories
(but no effective eval of
those theories)
• Adaptation of methods
from cogsci (but original
methods not well
understood)
• Rarely, ongoing
collaboration with
cognitive scientists

21. On the Death of Visualization
(Lorensen 2004)
Can It Survive Without Customers?
• Visualization, alone, is not a solution.
• Visualization is a critical part of many applications.
• Visualization, the Community, lacks application
domain knowledge.
• Visualization has become a commodity.
• Visualization is not having an impact in applications.

22. Visual analytics origins

23. Battelle PNNL R&D Agenda
Panel
• In US, Panel meeting in 2004
•Brown, GMU, Georgia Tech, OSU, Penn State, Purdue,
SFU , Stanford, UC, UI, UM, UNC, UU, WPI
•Boeing, Microsoft, PARC, Sandia Labs
•CIA, DHS, FBI, NIST, NSA, unspecified
• Gave rise to
•Industry Consortium
•DHS Centre of Excellence
• Ccicada (Rutgers DIMACS)
• VACCINE (Purdue et al)
• In Europe, EU Vismaster Coordination action
• DFG Scalable Visual Analytics Priority program

24. Visual analytics
“This science must be built on integrated perceptual
and cognitive theories that embrace the dynamic
interaction between cognition, perception, and action.
It must provide insight on fundamental cognitive
concepts such as attention and memory. It must build
basic knowledge about the psychological foundations
of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’
and ‘abstraction.’ “
“Illuminating the Path” (IEEE Press)
“The science of analytical reasoning
facilitated by interactive visual interfaces”

25. How are VA Information
systems different?
• Development based on understanding of
expert cognition in situ
• Informed by current cognitive & social science
• Engagement with community of experts
• Emergent cognitive science of expert reasoning
• Clear support for analytical processes--
reasoning, collaboration & interaction
• Graphical analog for analytic processes
• Support “Human-information discourse”
• Integrated across roles in the community

27. Visual analytics
Pg. 4 in Thomas, J., Cook, K., Institute of Electrical and Electronics Engineers, Dept
of Homeland Security, & United States. (2008). Illuminating the Path: The Research
and Development Agenda for Visual Analytics. IEEE Press. Retrieved from http://
www.osti.gov/energycitations/product.biblio.jsp?osti_id=912515

30. Cognitive
&
Perceptual
Sciences
Visual
Information
Systems
Graphic &
Interaction
Design
Mathematical
& Statistical
Methods
Effective
Situated
R&D

31. http://w

32. Visual analytics as a
translational cognitive science

33. How to bridge informatics &
psychology?
• VIS offers:
• Implementations
• Funding
• Awesome
Research
Questions
• Psych offers
• Methodology
• Theory
• Phenomena
• Cheap talent
Challenges: defining boundary
objects, culture clash,
publication venues, academic
jobs for Cogs grads…

34. My approach: start in the middle!
• Develop bridging
“Cyberpsychology”
theory & methods
• Hope is that…
• They are building
from each shore
• Somehow we will be
able to align things
http://www.magic.ubc.ca
http://www.icics.ubc.ca
http://interaction-science.iat.sfu.ca

35. Bridging ideas from D-Cog
• Visualization literacy is a form of “Smart seeing
and projecting” w external representation
• We propose 2 additional D-cog perspectives:
• Agent-machine coupling: coordination of thought and
action in dynamic artificial environments
•Cognitive architecture modeling and characterizing
“personal equation” of individual differences (e.g.
perceptual expertise)
• Socially-distributed cognition
•Grounded theory in Clark’s Joint Activity Theory
framework analysis of pair/group collaborative decision
making

36. Agent-machine coupling in air
traffic control
• Cognitive architecture from psychology
• Extend to expert human performance
• Cognitive expertise
• Visual expertise
• Visuomotor expertise
• Multimodality & modularity
• Test human capabilities in dynamic display
environments

37. Controller/display systems in air
traffic control
• NextGen ATC
“fishtank”
projection
• Change camera
position for better
view
• How will global
motion affect
tracking?
Liu, G.Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005)
Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of
Experimental Psychology: Human Perception and Performance. 31(2),Apr 2005,
235-247.
http://www.youtube.com/watch?v=tKJVB4id_TY

38. FINST theory of spatial
indexing

39. Multiple object tracking
(Pylyshyn)

40. Fit human tracking function
(Lui)

41. ... Then add display motion

42. Tracking vs object speed

43. Tracking in warped space

44. Tracking in warped space

45. Conclusion: We track in allocentric
space
• Retinal speed of targets does not determine
performance
• Motion of targets relative to each other does
• But only if motion preserves good metric
characteristics of space
• Explanation is at the level of a human -
display cognitive system

46. • “Pair analytics” sessions
•Student visual analyst & trained
domain expert collaborate on
analytic task
•Student “drives”, expert
“navigates”
•Video session & capture screen
Social D-Cog in safety analysis

47. Joint Activity Theory (Clark)
• Language is an essentially collaborative
activity, like playing duet or paddling canoe
• We work to build common ground so as to
communicate effectively and efficiently
• Clark’s theory:
• Defines kinds of common ground
• Formalizes the notion of activity as a “joint action”
• Describes the processes by which common ground is
developed through joint action

48. Bird Strikes
• http://www.youtube.com/watch?
feature=player_detailpage&v=5jN0bqL9cM0

52. Tableau Bird Strike Pair
Analysis

53. IN-SPIRE Bird Strike Pair Analysis

54. Wade Internship
• Video recorded and screen captured over 10
Paired Analysis sessions using both Tableau
and IN-SPIRE
• Influenced design decisions on:
• 777
• P8-A
• 787
• 747-8
• Changes to pilot training manual

55. Wade Internship
• Presented work to:
• 787 Engineers
• Aviation Safety Community of Practice
• Aerodynamics, Performance, Stability and Control
flight data recorder analysis group
• Advanced Analytics group
• UW Aeronautics and Astronautics students
• Boeing Educational Network webcast (400+)
• 500+ people exposed to Visual Analytics,
Paired Analysis for Aviation Safety

56. Joint activity analysis
• Observe joint attention management by
project markers
• Multimodal markers — mouse gesture & verbal
(content & prosody)
• Vertical and horizontal markers
• We define event structure for analysis based
on markers & actions
• Social constraints seen in self-talk incidents,
provides effective verbal protocols for protocol
analysis methods

57. Metrics include:
• Fluidity of the process.
• Management of joint attention.
• Situational awareness.
• Cognitive economics.

58. Publications
• Kaastra, L. T. and Fisher, B. (2014) Tracking Joint Activities in Visual Analyses.
Proceedings of the Psychonomics Society meeting
• Kaastra, L. T. and Fisher, B. (2014) Pair Analytics as a Field Experiment Methodology.. In
Proceedings of the 2014 BELIV Workshop: Beyond Time and Errors-Novel Evaluation
Methods for Visualization
• Kaastra, L. T., Arias-Hernandez, R., & Fisher, B. (2012). Evaluating analytic performance.
In Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors-Novel Evaluation
Methods for Visualization
• Arias-Hernandez, R., Green, T. M., & Fisher, B. (2012). From Cognitive Amplifiers to
Cognitive Prostheses: Understandings of the Material Basis of Cognition in Visual
Analytics. In Interdisciplinary Science Reviews, Vol. 37 No. 1.
• Arias-Hernandez, R., Kaastra, L. T., & Fisher, B. (2011). Joint action theory and pair
analytics: In- vivo studies of cognition and social interaction in collaborative visual analytics
In Proceedings of the 33rd Annual Conference of the Cognitive Science Society
• Arias-Hernandez, R., Kaastra, L. T., Green, T. M., & Fisher, B. (2011). Pair Analytics:
Capturing Reasoning Processes in Collaborative Visual Analytics. In (HICSS), 2011 44th
Hawaii International Conference On System Sciences.

59. Cognition
Perceptual
Science
Methods
Social
Science
Methods
Computation and
Visualization
Methods
Graphic &
Interaction
Design
Methods
Analysis
“in the
wild”
What we need

60. How to do it
• Institutes/centres/programs that combine:
• Application experts
• VIS people
• Cognitive scientists
• Designers
• Publication venues for translational research
• Two-part conference approach
• Outreach to application venues
• Return to VIS community

61. Thanks to SCIENCElab!
• Dr Richard Arias-
Hernández
• Dr. Nathalie Prevost
• Dr. Linda Kaastra
• Dr. Payam Rahmdel
• Samar Al-Hajj
• Nadya Calderón
• Tera Marie Green
• Ethan Soutar-Rau
• Ali Khalili
• Barry Po
• Aaron Smith
• Andrew Wade
• Doug Mackenzie

62. Cogs/IAT 885 Visually Enabled
Reasoning
• Cognitive theory
• Psychology of human reasoning
• Modelling causality
• Alternative (modal/hybrid) logics
• Analytic practice
• Problems and datasets from SEMVAST, Boeing
• Teams w paper, IN-SPIRE, Tableau Jigsaw
• Outcomes
• Learn reflective analytic practice
• Prepare for internship as analyst

63. Textbooks
• How We Reason. Philip Johnson-Laird,
Oxford Press
• Causal Models: How People Think about the
World and Its Alternatives. S. Sloman, Oxford
Press R
• Human Reasoning and Cognitive Science.
Keith Stenning & Michiel van Lambalgen, MIT
Press

64. Cogsci society logo