This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master, Software Engineering and Green IT track of the Vrije Universiteit Amsterdam: http://masters.vu.nl/en/programmes/computer-science-software-engineering-green-it/index.aspx http://www.ivanomalavolta.com

2. Vrije Universiteit Amsterdam

3. Vrije Universiteit Amsterdam
Experiment principles

4. Vrije Universiteit Amsterdam
Image encoding: which algorithm should I
use to optimize battery usage?
○ PNG
○ JPEG
○ …

5. Vrije Universiteit Amsterdam
MAIN DRIVERS of experimentation:
● to be able to scientifically assess an idea
● generalize

6. Vrije Universiteit Amsterdam
Theory
Observation
Cause EffectCausation
e.g. green encoding algorithm e.g. energy efficiency
Treatment Experiment Outcome
e.g. JPEG, PNG e.g. energy consumption per image

7. Vrije Universiteit Amsterdam
Terminology

8. Vrije Universiteit Amsterdam
● Dependent variables: quantities observed in the study
(a.k.a. response, output variables)
○ e.g. energy consumption, image quality
● Independent variables: quantities that we are able to
manipulate/control (a.k.a. input variables)
○ e.g. encoding algorithm, size of image, operating system

9. Vrije Universiteit Amsterdam
● Factor: an independent variable that we deliberately
manipulate/control
○ e.g. image encoding algorithm
● Treatment: a specific value of a factor
○ e.g. JPEG, PNG

10. Vrije Universiteit Amsterdam
● Subject: who applies the treatment
○ e.g. software developer, software architect, user
● Object: the receiver of the treatment
○ e.g. software application

11. Vrije Universiteit Amsterdam
● Trial: combination of subject, object and treatment
○ e.g. a software developer develops an app that
encodes images using the JPEG algorithm
● Experiment: a combination of several trials to observe the
effects of the treatments
○ e.g. ???

12. Vrije Universiteit Amsterdam

13. Vrije Universiteit Amsterdam
The experimental process

14. Vrije Universiteit Amsterdam
● Main idea behind the experiment
● The hypotheses must be stated clearly
○ Not formally, just clearly
Idea

15. Vrije Universiteit Amsterdam
Experiment
scoping
● We scope the experiment by stating
the problem, objectives, and goals
● We will use the Goal-Question-Metric
(GQM) framework
Idea

16. Vrije Universiteit Amsterdam
● Define context
● Formulate hypotheses
● Identify input and output variables
● Design the study
● Instrumentation
● Analyze validity threats
Experiment
scoping
Experiment
planning
Idea

17. Vrije Universiteit Amsterdam
● Preparation
○ Guidelines, subjects training,
code instrumentation, ...
● Execution
○ aka measurements
collection
● Data Validation
Experiment
scoping
Experiment
planning
Idea
Experiment
operation

18. Vrije Universiteit Amsterdam
● Understanding the data
○ descriptive statistics
○ exploratory data analysis
(e.g. box plots, scatter plots, … )
● Hypothesis testing
● Results interpretation
Experiment
scoping
Experiment
planning
Idea
Experiment
operation
Analysis &
interpretation

19. Vrije Universiteit Amsterdam
● Document the results
● Prepare replication package
● Sum up lessons learned
● Write down reflections
Experiment
scoping
Experiment
planning
Idea
Experiment
operation
Analysis &
interpretation
Presentation &
package

20. Vrije Universiteit Amsterdam
●
○
○ …
●
○
■

21. Vrije Universiteit Amsterdam
Chapter 6

22. Vrije Universiteit Amsterdam
Some contents of lecture extracted from:
● Giuseppe Procaccianti’s lectures at VU