These are the slides for a plenary talk I gave at GEC 11 in Denver in July 2011. The primary message is that we, the networking community need to fundamentally improve the way we do experimental research.

1. Will RSpec help us to do better Science?
Maximilian Ott
With lots of help from Thierry Rakotoarivelo & Guillaume Jourjon
© 2011 NICTA. All Rights Reserved. from imagination to impact

2. © 2011 NICTA 2

3. Why are we doing what we are doing?
© 2011 NICTA 3

4. Curious Bunch - Want to figure out new things
© 2011 NICTA 4

5. The advancement of science depends
in general on the interplay between
experimental studies and theory
Wikipedia
© 2011 NICTA 5

6. Many interesting Things only happen at Scale
© 2011 NICTA
Young Hyun, CAIDA
6

7. Need for LARGE
Experimental Facilities
© 2011 NICTA 7

8. © 2011 NICTA 8

9. © 2011 NICTA 9

10. Hypothesis
Analysis Experiment
Observation
© 2011 NICTA 10

11. Hypothesis
Analysis Experiment
Observation
© 2011 NICTA 11

12. How “well” do we do experiments today?
58%
(19)
30%
(10)
21%
12% 9% (7)
(4) (3)
Missing Missing Missing Missing OK
Model Experiment Replication Precision
Validation Description Information Information
“Evaluation” section of SIGCOMM 2010 papers
© 2011 NICTA 12

13. How can we improve this?
• The Art & Science of Experimentation
– The art of designing experiments
– Scientifically sound methodology & analysis
• Dissemination
– Overcoming the 6 page paper information barrier
• Learning
– Reflection on the methodology
– Developing Best Practices
• Tools
– To streamline processes
– To record processes and outcomes
© 2011 NICTA 13

14. Need for Methodologies & Best Practices
Observations
Model 1 … Model 2
Ha , H0 Ha , H0
Experiment Experiment
New Obs + Analysis
© 2011 NICTA 14

15. Need to teach Experimenting at Scale
Experiment
Queue
High Speed Student
Dedicated
Network
Network Internet
In the Box
Control
Remote
Network
Experiment
Server
Tested application and protocol
Measurement points
Student
ireel.npc.nicta.com.au
© 2011 NICTA 15

16. Need new ways to disseminate Findings
?
Authors are invited to submit
Full Papers of up to 10 pages
and Testbed Practices Papers
of up to 6 pages, including
references, figures and tables,
formatted according to the
IEEE conference format.
© 2011 NICTA 16

17. LabWiki: Publishing Your Lab Book
Portal Studied
Phenomenon
Model 1 ... Model i
HA , H 0 HA , H 0
Experiment Experiment
OEDL Experiment Experiment
Script Experiment Experiment
Measure-
ments Observations
database
200 350000
Distance
Bandwidth
180
300000
160
140 250000
R
Distance (m)
120
Analyses
Bandwidth
200000
100
150000
80
Script 60
40
100000
50000
20
0 0
Falsify and/or
0 50 100 150 200 250 300
Time (s)
refine the
models
© 2011 NICTA 17

18. Notes Experiment
Description
Observations
Model 1 … Model 2
Code Ha , H0 Ha , H0 Runs
Experiment Experiment
New Obs + Analysis
Analysis
Measurements
© 2011 NICTA 18

19. Notes Experiment
Description
Observations
Model 1 … Model 2
Code Ha , H0 Ha , H0 Runs
Experiment Experiment
New Obs + Analysis
Analysis
Measurements
© 2011 NICTA 19

20. Notes Experiment
Description
Observations
Model 1 … Model 2
Code Runs
Ha , H0 Summary Ha , H0
Experiment Experiment
New Obs + Analysis
Analysis
Measurements
© 2011 NICTA 20

21. Summary
Researcher
Comment
Summary
Reader
Review
Analysis
Reviewer
© 2011 NICTA 21

22. We are working on it
• OMF
– OEDL to describe experiments
– Experiment Controller to run them
• OML
– Capture measurements & meta-data
• Portal
– Wiki: Keep notes with references
– Schedule & monitor runs
– Analysis with integrated R server
– Repository for all artifacts
© 2011 NICTA 22

23. Portal
OMF
Wiki
OML
ON/OFF
e1 App 1
Measurement Library (OML)
Proxy
e2 Server Control
App 2 Network
OML SQL
c1
...
Server Database
...
Sensing
daemon
sensor device
System
Monitor
Experiment Node Control Node
OEDL Visualization R
Link Quality:
Very Good
Good
Poor
Very Poor
Y/Z Coordintate
X Coordinate
© 2011 NICTA 23

24. Call to Action
• The Art & Science of Experimentation
– The art of designing experiments
– Scientifically sound methodology & analysis
• Dissemination
– There need to be better ways than 10 page papers
• Learning
– Reflection on the methodology
– Developing Best Practices
– Teaching experimentation at scale
• Tools
– To streamline processes
– To record processes and outcomes
© 2011 NICTA 24

25. omf.mytestbed.net
wiki.mytestbed.net
max.ott@nicta.com.au
© 2011 NICTA 25