A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]

Institut für Wirtschaftsinformatik/Informatik
5th International Conference on Information
Visualization Theory and Applications
Richard Müller1, Pascal Kovacs1, Jan Schilbach1, Ulrich
Eisenecker1, Dirk Zeckzer2, Gerik Scheuermann2
1Information Systems Institute
2Institute of Computer Science
University of Leipzig, Leipzig, Germany
January 5, 2014
A Structured Approach for Conducting a Series of Controlled
Experiments in Software Visualization

2
2D or 3D: State of the Art
© 2014 by R. Müller
 Scientific drivers
 Lack of empirical research in
software visualization
 Ongoing discourse: 2D vs. 3D
 Five papers about 3D software
visualizations at VISSOFT
2013
 Technical drivers
 Innovative methods to generate
software visualizations
 Increasing computing power
 Emerging technology
 High quality and low cost 3D
environments
3D-Vision Glasses and
Infrared Transmitter
3D-Vision Capable
Monitor or Projector
Quadro
GPU

3
A Series of Controlled Experiments
 A series of controlled
experiments is required
to determine the
circumstances when and
why a specific software
visualization is suitable
for a certain software
engineering task
Software
Visualization
User
Task
Software
Artifact
Represen-
tation
Navigation
&
Interaction
Implemen-
tation

4
Munzner‘s Model and its Extension
[Munzner 2009; Meyer, Sedlmair, Munzner 2012]
Block Guideline

5
Domain Specific Adaption of Munzner’s
Extended Model for Software Visualization

6
Possible Instantiations (1/3)
Factor/Sub-
Factor
Examples for Possible Instantiations
User
Role
Background
Knowledge
Circumstances
Manager, Requirements Engineer, Architect, Developer, Tester,
Maintainer, Reengineer, Documenter, Consultant, Team, Researcher
Age, Gender, Color Blindness, Ability of Stereoscopic Viewing
Education, Programming Experience, Domain Knowledge
Occupation, Familiarity with Study Object/Tools
Task
Problem Development, Maintenance, Re-Engineering, Reverse Engineering,
Software Process Management, Marketing, Test, Documentation
Operation Retrieve Value, Filter, Compute Derived Value, Find Extremum, Sort,
Determine Range, Characterize Distribution, Find Anomalies, Cluster,
Correlate

7
Factor/Sub-
Factor
Software
Artifact
Type
Size
Aspect
Requirements, Architecture, Source Code, Stack Trace, Revision History
Small, Medium, Large
Structure, Behavior, Evolution
Representation
Dimensionality
Technique
2D, 2.5D, Augmented 2D, Adapted 2D, Inherent 3D
Graph, Tree, Abstract/Real World Metaphor,
Decorational/Representational Animation

8
Factor/Sub-
Factor
Navigation &
Interaction
Technique
Input
Output
Overview, Zoom, Filter, Details-on-Demand, Relate, History, Extract
Keyboard, Mouse, Gamepad, Flystick, Kinect, Touch Device, Leap
Motion, Brain-Computer Interface
Paper, Monitor, Projector, Virtual Reality Environment, Oculus Rift
Implementation
Algorithm
Platform Dep.
Automation
Radial Layout, Balloon Layout, Treemap, Information Cube, Cone Tree
Platform
Platform Independent, Platform Dependent
Full, Semi, Manual
Data Famix, Dynamix, Hismo

9
Example Experiment: Overview
 Research question
 Does an inherent 3D software visualization reduce time to solve
software engineering tasks, compared to a 2D software visualization?
 Dependent variable
 time
 Independent variable
 dimensionality
 Between-subjects design
 control group (2D), experimental group (3D)

10
Example Experiment: Detail
User
Role
Background
Knowledge
Developer
Color Blindness, Ability of
Stereoscopic Viewing
Education, Programming
Experience, Domain
Knowledge (Virtual Reality,
Touch Devices, 3D)
Task
Problem Find a Bug,
Identify a Dominating Class
Operation Retrieve Value, Filter, Find
Extremum, Find Anomalies,
Correlate
Software Artifact
Type
Size
Aspect
Source Code
Medium
Structure
Representation
Dimensionality
Technique
2D vs. Inherent 3D
Graph (Nested Node-Link
Technique)
Navigation & Interaction
Technique
Input
Output
Overview, Zoom, Filter,
Relate
Touch Device
Virtual Reality Environment
Implementation
Algorithm
Platform Dep.
Automation
Force Directed Layout
Plaform Independent
Full
Data Abstraction Famix
Vary different factors in different experiments
while keeping other relevant factors constant or
measure their influence on the result

11
Summary and Conclusion
 The structured approach to conduct a series of controlled
experiments in software visualization is based on Munzner‘s model
and its extension
 The approach…
 Provides a comprehensive overview of relevant influence factors and
their possible interactions
 Helps to plan new experiments, to categorize previous experiments and
to assess their results
 Supports researchers to identify research gaps

12
Your questions…

13
References
 Meyer, M., Sedlmair, M. & Munzner, T., 2012. The four-level nested model revisited: blocks
and guidelines. In Workshop on BEyond time and errors: novel evaLuation methods for
Information Visualization (BELIV). pp. 1–6.
 Munzner, T., 2009. A nested model for visualization design and validation. IEEE
Transactions on Visualization and Computer Graphics, 15(6), pp.921–928.

A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]

Recommended

Recommended

More Related Content

Similar to A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]

Similar to A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014] (20)

Recently uploaded

Recently uploaded (20)

A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization [IVAPP 2014]