1. Q I N G A O 1, X U H U I Z H A N G 1, P E I - L U E N P A T R I C K R A U 1
1 INSTITUTE OF HUMAN FACTORS & ERGONOMICS, DEPT. OF INDUSTRIAL ENGINEERING,
TSINGHUA UNIVERSITY, BEIJING, 100084, CHINA
A N T H O N Y A . M A C I E J E W S K I 2, H O W A R D J A Y S I E G E L 2,3
2E L E C T R I C A L A N D C O M P U T E R E N G I N E E R I N G D E P A R T M E N T ,
3C O M P U T E R S C I E N C E D E P A R T M E N T
COLORADO STATE UNIVERSITY, FORT COLLINS, CO 80523 -1373 USA
PERFORMANCE VISUALIZATION FOR
LARGE-SCALE COMPUTING SYSTEMS
A Literature Review
HCI International 2011
9-14 July, Orlando, USA

2. CONTENT
• Motivation
• Approach to Performance Visualization
• Review of Performance Visualization Techniques for
Large-Scale Systems
• Future Work
Performance Visualization for Large-scale Computing Systems: A Literature Review 2

3. MOTIVATION
Exascale computers: 1000 times
faster than the current
Need for extreme scale
petascale systems
computing solutions
Immense volume and
Need to performance complexity of the
monitoring & tuning in run- performance data
time for extreme-scale
systems
Need for powerful and A review of existing
usable performance performance
visualization methods
visualization tool for extreme-
and tools for large
scale system scale systems
Performance Visualization for Large-scale Computing Systems: A Literature Review 3

4. PERFORMANCE VISUALIZATION
Program Visualization Visual
behavior Representations
Data View
Visual
transformation Transformation
Mappings
Raw Data Views
data tables
Source: Card, 2002
Human Interaction
• Goal:
• Augmenting cognition with the human visual system’s highly tuned ability to see
patterns and trends
• Aid comprehension of the dynamics, intricacies, and properties of program execution
Performance Visualization for Large-scale Computing Systems: A Literature Review 4

5. APPROACH TO PERFORMANCE
VISUALIZATION
Enabling access to performance data to be
Instrumentation
measured
Recording selected data during the run-time of the
Measurement
program
Data analysis Analyzing data for performance visualization
Mapping performance characteristics to proper
Visualization visual representations and interactions
Performance Visualization for Large-scale Computing Systems: A Literature Review 5

6. APPROACH TO PERFORMANCE
VISUALIZATION
• Instrumentation
• What to be instrumented?
Fidelity
Reflect application Minimizing
performance as perturbation of
Pertubation
closely as possible that behavior as
much as possible
• Approach
• Hardware
• Less performance degradation
• Poor portability
• Software
• Better portability
• Automation required for large-scale systems
Performance Visualization for Large-scale Computing Systems: A Literature Review 6

7. APPROACH TO PERFORMANCE
VISUALIZATION
• Measurement
• Tracing
• More detailed execution information
• Necessary for visualizing detailed program run-time behaviors
• E.g., Virtue, Pajé
• Profiling
• Collects only summary statistics, mostly with hardware counters
• Less pertubation by sacrificing fidelity
• Allow data collection with long execution time
• E.g., SvPablo
• Trigger for recording action
• Event-driven
• Periodically (sampling)
• Real-time or post-mortem?
• For distributed application, real-time measurement and visualization is
necessary
Performance Visualization for Large-scale Computing Systems: A Literature Review 7

8. APPROACH TO PERFORMANCE
VISUALIZATION
• Data analysis
• Microscopic and macroscopic metrics
• Method
• Data reduction
• Multivariate statistical analysis
• Application-specific analysis
• Bates, 1995: Recognizing high-level program behaviors
• AIMS: Pointing out causes of poor performance, generating
scalability trends
Performance Visualization for Large-scale Computing Systems: A Literature Review 8

9. APPROACH TO PERFORMANCE
VISUALIZATION
• Visualization
• Basic visual components involved in information visualization
(Card, 2002)
• Spatial substrate
• Marks
• Connections
• Enclosures Types of marks, source: Card, 2002
• Retinal properties
• Temporal encoding
Retinal properties, source: Card, 2002
Performance Visualization for Large-scale Computing Systems: A Literature Review 9

10. CLASSIFICATION OF PERFORMANCE
VISUALIZATION TECHNIQUES
Category Performance Visualization Example applications and studies
Techniques
Simple visual Pie charts, distribution, box plots, ParaGraph [2], PET [20], SvPablo [16],
structures kiviat diagrams VAMPIR [21], Devise [22], AIMS [9]
Timeline views Paje [23], AIMS [9], Devise [22],
AerialVision [24], Paraver [25],
SIEVE [14], Virtue [13], utilization and
algorithm timeline views in [17]
Information typologies SHMAP [26], Vista [4], Voyeur [27],
processor and network port display in
[28], hierarchical display in [12]
Information landscape Triva [29], Cichild [30]
Trees & networks Paradyn [18], Cone Trees [31],
Virtue [13], [32]
Composed visual Single-axis composition AIMS [9], Vista [4]
structures Double-axis composition Devise [22], AerialVision [24]
Case composition Triva [29]
Interactive visual Interaction through controls (data Paje[23], data input, filtering,
structure input, data transformation, visual and view manipulation in [28]
mapping definition, view operations) and [32]
Interaction through images Virtue [13], Cone Trees [31],
(magnifying lens, cascading displays, Devise [22], direct manipulation of the
linking and brushing, direct 3D cone and virtual threads in [32]
manipulation of views and objects)
Focus + context Macro-micro composite view Microscopic profile in [4],
visual structures PC-Histogram in [24]
Performance Visualization for Large-scale Computing Systems: A Literature Review 10

11. SIMPLE VISUAL STRUCTURES
• Statistical charts
• Provide an overview of
important performance
metrics
• Enable quick identification a. PET: Bar chart of resource utilization b. Pajé Pie chart representing the percentage
:
of major problems
percentage of different processors [22] of time with different number of active
threads at a node [17]
c. SvPablo: color matrix of metrics, each d. ParaGraph: Kiviat diagram showing load
column representing a performance metric, imbalance among different processors [7]
and color representing the value [13]
Performance Visualization for Large-scale Computing Systems: A Literature Review 11

12. SIMPLE VISUAL STRUCTURES
• Time-line views
• Showing the evolution of performance statistics over time
Utilization and overhead view
in Alexandrov et al., 2010
Time views of utilization/computation/communication
metrics of AerialVision AerialVision’s time view of
runtime warp divergence
breakdown
Performance Visualization for Large-scale Computing Systems: A Literature Review 12

13. SIMPLE VISUAL STRUCTURES
• Time-line views
• Describing run-time behaviors and communication paths
Virtue: time-tunnel display
Pajé visualization of program execution and communication
:
AIMS: visualization of program executions ParaGraph: Space-time diagram
Performance Visualization for Large-scale Computing Systems: A Literature Review 13

14. SIMPLE VISUAL STRUCTURES
• Time-line views
• • Facilitating source code level analysis
AerialVision: PC-Histogram SIEVE: Contour-plot showing
calls to a specific function
Performance Visualization for Large-scale Computing Systems: A Literature Review 14

15. SIMPLE VISUAL STRUCTURES
• Information typography
Proposed hierarchical views of a complex reconfigurable
Port display showing job computing application
allocation, communication traffic,
and route between nodes of a
cluster
Performance Visualization for Large-scale Computing Systems: A Literature Review 15

16. SIMPLE VISUAL STRUCTURES
• Information landscape
a. Triva: information landscape based on b. Triva: information landscape based
network typology on resource hierarchy
c. Cichild: interpolated surfaces showing network delays between different sites
Performance Visualization for Large-scale Computing Systems: A Literature Review 16

17. SIMPLE VISUAL STRUCTURES
• Trees and networks
a. Paradyn: Performance Consultant, b. Cone Trees: 3D visualization of tree
showing a search hierarchy [14] structures [31]
Virtue: Geographic network display [15]
Performance Visualization for Large-scale Computing Systems: A Literature Review 17

18. COMPOSED STRUCTURE
• Single-axis composition
• Multiple graphs sharing
single axis
• Double-axis composition
• Multiple graphs sharing
AIMS: composite view of procedure execution graph on
double axis each node and machine-load chart of each node
• Case compositions
• Two graphs having a single
mark for each case fused
Devise: message behavior visualization
Performance Visualization for Large-scale Computing Systems: A Literature Review 18

19. INTERACTIVE STRUCTURES
• Direct interaction through the
visualization
• Magifying lens
• Panning, selecting, re-positioning
• Cascading display (e.g., ConeTrees)
• Use of gestures (e.g., Virtue)
• Indirect interaction through controls
• Interactions with underlying computation,Virtue: Magnifying lens
such as data-related controls and
definitions of visual mapping
• View configurations
• Scroll-bars, zoom in/out, sliders…
Performance Visualization for Large-scale Computing Systems: A Literature Review 19

20. ATTENTION-REACTIVE VISUAL
STRUCTURES
• Limited usage in performance visualization systems
AerialVision: PC histogram
Vista: Filmstrip view of utilization
Performance Visualization for Large-scale Computing Systems: A Literature Review 20

21. SUMMARY & OUTLOOK
• Summary issues that need to be addressed
throughout the process of performance visualization
• Review performance visualization techniques from
21 systems
• Challenge: huge data size requires good scalability
• Data abstraction method from scientific visualization
• Visualization based on focus + context abstraction
• Challenge: ergonomics and usability issues
• Understanding of characteristics and limitations and human
sensory and cognition capabilities
Performance Visualization for Large-scale Computing Systems: A Literature Review 21

22. THANKS, AND QUESTIONS?
Performance Visualization for Large-scale Computing Systems: A Literature Review 22