Towards Understanding
the Replication of
SE Experiments
Natalia Juristo
Universidad Politecnica de Madrid (Spain)
&
Univer...
Scope & Terminology
 This talk focuses on the replication of
experiments
 I will refer to the study whose results we
wan...
Replication Intuitive Definition
 Deliberate repetition of research
procedures in a second investigation for
the purpose ...
Content
 Replication & the experimental paradigm
 State of replication in ESE: practice & theory
 Shedding a bit of lig...
Role of Replication in
Experimental Paradigm
Searching for Regularities
 Science does not settle for anecdotes. A
scientific law or theory describes a regular
occurre...
One Result, Three Meanings
 Without reproduction of results it is impossible
to distinguish whether they
 occurred by ch...
State of Replication in ESE
Practice
Not Enough Replications
 Most SE experiments have not yet been
replicated
 Two reviews provide empirical data to
support...
Experiments in Leading Journals
& Conferences (1993-2002)
 5,453 articles published from 1993 to 2002
in major SE journal...
All Publications
 96 papers reporting replications
 133 replications of 72 baseline studies
 Any type of empirical stud...
Mostly Internal & Not Stand-Alone
Publications [Da Silva et al. 2013]
Internal – Original-Included reports
Internal – Repl...
First Paper Published on a
Replication [Da Silva et al. 2012]
 In SE, the first article that explicitly reported
a replic...
EMSE Special Issue on
Replication
 The large number of submissions was
admittedly more than we expected
 We received a t...
State of Replication in ESE
Theory
First Theoretical Publication on
Replication
 In 1999, a paper discussed a framework to
organize sets of related experime...
More Activity in the Last 10 Years
 Shull, Basili, Carver, Maldonado, Travassos, Mendonça & Fabbri Replicating
software e...
State of the Theory
 There is no agreement yet on terminology,
typology, purposes, operation and other
replication issues...
Example of Divergent Views
 Some researchers advise the use of different protocols
and materials to preserve independence...
Shedding some Light on
Replication
Role of Replication
The Two Roles of Replication
 Validation
 Learning
Learning Relevant Conditions
 As more replications of Thompson and
McConnell’s baseline experiment were run
 different c...
Which are the Important
Variables?
 “…In fact, the principle of Transversely
Excited Atmospheric (TEA) lasers,
scientists...
First Learn, Then Validate
 “In the early stages, failure to get the expected
results is not falsification but a step in ...
SE Problems with Identical
Replications
 SE has tried to repeat experiments identically,
but no exact replications have y...
In the Beginning Most is Unkown
 “Most aspects are unknown when we start
to study a phenomenon experimentally.
Even the t...
Start with Similar Replications
 “The less that is known about an area the more
power a very similar experiment has ... T...
Learning & Validation Process
1. Start with identical replications
 At the beginning of experimental research, equality, ...
Learning is Even More Important
 Replication is needed not merely to
validate one’s findings, but more
importantly, to es...
Shedding some Light on
Replication
Functions of Replication
Reminder of Experimental Setting
 Operationalization
 Treatments
 Response variable
 Protocol
 Experimental design
 ...
Verification Functions
 Control experimental errors
 Control protocol independence
 Understand operationalization limit...
Control Experimental Errors
 Verify that the results of the baseline experiment
are not a chance product of an error
 Al...
Control Protocol Independence
 Verify that the results of the baseline experiment
are not artifactual
 An artifactual re...
Understanding Operationalization
Limits
 Learn how sensitive results are to different
operationalizations
 Treatment ope...
Understand Population Limits
 Learn the extent to which results hold for
other subject types or other types of
experiment...
Changes & Replication Functions
Experimental
Configuration
Control
Experimental
Error
Control Protocol
Independence
Unders...
Some Answers
Question 1
What exactly is a replication
study?
Establishing Limits
Amount of Changes
Run
Same data
different
models or
statistical
methods
Identical
same site &
research...
Question 2
What level of similarity should an
experiment have to be
considered a replication rather
than a new experiment?
Unchanged elements of a
replication
 A replication must share the hypothesis
with the baseline experiment
 Same response...
Partials Replications
Exp. A (Baseline OR Replication)
Exp. B (Replication OR Baseline)
Exp. D
Replication C
RV3
RV2
RV1
T...
Question 3
What changes are acceptable?
Levels of Verification
 Similarity between the baseline
experiment and a replication serve
different verification purpose...
Can I Change Everything?
 It is better not to change everything at the
same time
 We can understand the source of differ...
Everything Different
Changes
Different
Identical
Experiment
Elements
One Change at a Time
Changes
Different
Iqual
Experiment
Elements
Increasing Changes
Changes
Different
Iqual
Experiment
Elements
Question 4
What is the level of similarity that
results must have to be
considered as reproduced?
Understanding the Natural
Variation
 Identical replications are useful for
understanding the range of variability of
resu...
Question 5
Should replications reuse
materials?
Accomodating Opposing Views
 The possible threat of errors being propagated
by experimenters exchanging materials does no...
Accomodating Opposing Views
 Replication functions accommodate
opposing views within a broader
framework
 Contrary stanc...
Summarizing
Main Ideas
 Replication in ESE
 Replication plays an essential role in the experimental paradigm
 Replication is not a ...
Towards Understanding
the Replication of
SE Experiments
Natalia Juristo
Universidad Politecnica de Madrid (Spain)
&
Univer...
Próxima SlideShare
Cargando en…5
×

Towards Understanding SE Experiments Replication (ESEM'13 Keynote)

404 visualizaciones

Publicado el

To consolidate a body of knowledge built upon evidence, experimental results have to be extensively verified. Experiments need replication at other times and under other conditions before they can produce an established piece of knowledge. Several replications need to be run to strengthen the evidence.

Most SE experiments have not been replicated. If an experiment is not replicated, there is no way to distinguish whether results were produced by chance (the observed event occurred accidentally), results are artifactual (the event occurred because of the experimental configuration but does not exist in reality) or results conform to a pattern existing in reality.

The immaturity of experimental SE knowledge has been an obstacle to replication. Context differences usually oblige SE experimenters to adapt experiments for replication. As key experimental conditions are yet unknown, slight changes in replications have led to differences in the results that prevent verification.

There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment?

The aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. We need to learn which types of replications are feasible in SE as well as the acceptable changes for each type and the level of verification provided.

Publicado en: Tecnología, Educación
0 comentarios
0 recomendaciones
Estadísticas
Notas
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Sin descargas
Visualizaciones
Visualizaciones totales
404
En SlideShare
0
De insertados
0
Número de insertados
1
Acciones
Compartido
0
Descargas
6
Comentarios
0
Recomendaciones
0
Insertados 0
No insertados

No hay notas en la diapositiva.

Towards Understanding SE Experiments Replication (ESEM'13 Keynote)

  1. 1. Towards Understanding the Replication of SE Experiments Natalia Juristo Universidad Politecnica de Madrid (Spain) & University of Oulu (Finland) ESEM Conference Baltimore (USA) October 11th, 2013
  2. 2. Scope & Terminology  This talk focuses on the replication of experiments  I will refer to the study whose results we want to check as the baseline experiment
  3. 3. Replication Intuitive Definition  Deliberate repetition of research procedures in a second investigation for the purpose of determining if earlier results can be reproduced
  4. 4. Content  Replication & the experimental paradigm  State of replication in ESE: practice & theory  Shedding a bit of light  Purposes for replicating  Replication functions  Some answers  Replication limits  Baseline and replication minimum degree of similarity  Admissible changes  Reproduction of results  Threats to reuse materials  Summary
  5. 5. Role of Replication in Experimental Paradigm
  6. 6. Searching for Regularities  Science does not settle for anecdotes. A scientific law or theory describes a regular occurrence in the world  Regularities existing in reality are identified by reproducing the same event in different replications  The result of one experiment is an isolated event
  7. 7. One Result, Three Meanings  Without reproduction of results it is impossible to distinguish whether they  occurred by chance  are artifactual  the event occurs only in the experiment, not in reality  really correspond to a regularity
  8. 8. State of Replication in ESE Practice
  9. 9. Not Enough Replications  Most SE experiments have not yet been replicated  Two reviews provide empirical data to support this point  Let us look at their results
  10. 10. Experiments in Leading Journals & Conferences (1993-2002)  5,453 articles published from 1993 to 2002 in major SE journals and conference proceedings  113 experiments  20 (17.7%) described as replications Sjøberg et al. “A survey of controlled experiments in SE” TSE, 2005
  11. 11. All Publications  96 papers reporting replications  133 replications of 72 baseline studies  Any type of empirical study  Quasi-Experiments 35 49%  Controlled Experiments 21 29%  Case Study 15 21%  Survey 1 1% da Silva et al. “Replication of Empirical Studies in Software Engineering Research: A Systematic Mapping Study” EMSE 2013
  12. 12. Mostly Internal & Not Stand-Alone Publications [Da Silva et al. 2013] Internal – Original-Included reports Internal – Replication-only reports 1994 1995 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 0 0 0 2 1 1 1 3 1 3 6 1 3 5 2 4 1 3 0 0 3 1 2 2 1 3 11 7 5 2 6 14 1996 0 0 NumberofReplications 2 4 6 8 10 12 14 16 Total 1 5 6 4 5 6 6 4 4 9 15 11 13 9 13 220 18 20 22 External 1 2 4 3 1 4 1 1 0 0 3 1 3 5 3 70
  13. 13. First Paper Published on a Replication [Da Silva et al. 2012]  In SE, the first article that explicitly reported a replication of an empirical study was published in 1994  Daly, Brooks, Miller, Roper & Wood Verification of Results in Sw Maintenance Through External Replication Intl Conf on Software Maintenance
  14. 14. EMSE Special Issue on Replication  The large number of submissions was admittedly more than we expected  We received a total of 16 submissions  Encouraging the publication of replications will foster researchers to replicate more studies
  15. 15. State of Replication in ESE Theory
  16. 16. First Theoretical Publication on Replication  In 1999, a paper discussed a framework to organize sets of related experiments (families) and the generation of knowledge from such sets  Basili, Shull & Lanubile. Building knowledge through families of experiments. TSE  Is a family exactly a set of replications?  …experiments can be viewed as part of common families of studies, rather than being isolated events…
  17. 17. More Activity in the Last 10 Years  Shull, Basili, Carver, Maldonado, Travassos, Mendonça & Fabbri Replicating software engineering experiments: Addressing the tacit knowledge problem. ISESE 2002  Vegas, Juristo, Moreno, Solari & Letelier Analysis of the influence of communication between researchers on experiment replication. ISESE 2006  Brooks, Roper, Wood, Daly & Miller Replication’s role in software engineering. Guide to Advanced Empirical SE. Springer 2008  Juristo & Vegas Using differences among replications of software engineering experiments to gain knowledge. ESEM 2009 [Juristo & Vegas The Role of Non- Exact Replications in SE EMSE Journal 2011]  Krein & Knutson A Case for replication: Synthesizing research methodologies in SE. RESER 2010  Gómez, Juristo & Vegas Replications types in experimental disciplines. ESEM 2010  Juristo, Vegas, Solari, Abrahao & Ramos A Process for Managing Interaction between Experimenters to Get Useful Similar Replications IST 2013
  18. 18. State of the Theory  There is no agreement yet on terminology, typology, purposes, operation and other replication issues  There is not even agreement on what a replication is!!  Different authors consider different types of changes to the baseline experiment as admissible
  19. 19. Example of Divergent Views  Some researchers advise the use of different protocols and materials to preserve independence and prevent error propagation in replications by using the same configuration  Kitchenham The role of replications in ESE - a word of warning EMSE 2008  Other researchers recommend the reuse of materials to assure that replications are similar enough for results to be comparable  Shull, Carver, Vegas & Juristo The role of replications in ESE EMSE 2008
  20. 20. Shedding some Light on Replication Role of Replication
  21. 21. The Two Roles of Replication  Validation  Learning
  22. 22. Learning Relevant Conditions  As more replications of Thompson and McConnell’s baseline experiment were run  different conditions influencing the results of this experiment were identified  After several hundred experiments had been run  experimenters managed to identify around 70 conditions influencing the behavior of this type of invertebrate
  23. 23. Which are the Important Variables?  “…In fact, the principle of Transversely Excited Atmospheric (TEA) lasers, scientists did not know that the inductance of the top was important” A physicist quoted in Changing Order: Replication and Induction in Scientific Practice Harry Collins 1992
  24. 24. First Learn, Then Validate  “In the early stages, failure to get the expected results is not falsification but a step in the discovery of some interfering factor. For immature experimental knowledge, the first step is … to find out which experimental conditions should be controlled” Validity and the Research Process Brinberg and McGrath 1985
  25. 25. SE Problems with Identical Replications  SE has tried to repeat experiments identically, but no exact replications have yet been achieved  The complexity of the software development setting prevents the many experimental conditions from being reproduced identically  Yet this is a regular rather than an exceptional situation
  26. 26. In the Beginning Most is Unkown  “Most aspects are unknown when we start to study a phenomenon experimentally. Even the tiniest change in a replication can lead to inexplicable differences in the results” Validity and the Research Process Brinberg and McGrath 1985
  27. 27. Start with Similar Replications  “The less that is known about an area the more power a very similar experiment has ... This is because, in the absence of a well worked out set of crucial variables, any change in the experiment configuration, however trivial in appearance, may well entail invisible but significant changes in conditions” Changing Order: Replication and Induction in Scientific Practice Harry Collins 1992
  28. 28. Learning & Validation Process 1. Start with identical replications  At the beginning of experimental research, equality, even if targeted, will not happen  There will be either invisible but significant changes in conditions or induced changes due to context adaptation or both  Failure to get the expected results should not be construed as falsification, but as a step towards the discovery of some new factor 2. Later on, both knowledge discovery and testing can be more systematic  Changes in the configuration will be made purposely to learn more variables and rule out artifactual results
  29. 29. Learning is Even More Important  Replication is needed not merely to validate one’s findings, but more importantly, to establish the increasing range of radically different conditions under which the findings hold, and the predictable exceptions The design of replicated studies. American Statistician Lindsay and Ehrenberg 1993
  30. 30. Shedding some Light on Replication Functions of Replication
  31. 31. Reminder of Experimental Setting  Operationalization  Treatments  Response variable  Protocol  Experimental design  Experimental objects  Guides  Measuring instruments  Data analysis techniques  Population  Objects  Subjects
  32. 32. Verification Functions  Control experimental errors  Control protocol independence  Understand operationalization limits  Understand population limits
  33. 33. Control Experimental Errors  Verify that the results of the baseline experiment are not a chance product of an error  All elements of the experiment must resemble the baseline experiment as closely as possible  Collateral benefit  Provide an understanding of the natural (random) variation of the observed results  critical for being able to decide whether or not results hold in dissimilar replications
  34. 34. Control Protocol Independence  Verify that the results of the baseline experiment are not artifactual  An artifactual result is due to the experimental configuration and cannot be guaranteed to exist in reality  The experimental protocol needs to be changed for this purpose  If an experiment is replicated several times using the same materials, the observed results may occur due to the materials  The same applies for all protocol elements
  35. 35. Understanding Operationalization Limits  Learn how sensitive results are to different operationalizations  Treatment operationalizations  treatment application procedures, treatment instructions, resources, treatment transmission …  Effect operationalizations  Metrics, measurement procedures …
  36. 36. Understand Population Limits  Learn the extent to which results hold for other subject types or other types of experimental objects  Learn to which specific population the experimental sample belongs and what the characteristics of such population are
  37. 37. Changes & Replication Functions Experimental Configuration Control Experimental Error Control Protocol Independence Understand Operationalization Limits Understand Population Limits Operationalization = = ≠ = Population = = = ≠ Protocol = ≠ = = Function of Replication LEGEND: = the element is equal to, or as similar as possible to, the baseline experiment ≠ the element varies with respect to the baseline experiment
  38. 38. Some Answers
  39. 39. Question 1 What exactly is a replication study?
  40. 40. Establishing Limits Amount of Changes Run Same data different models or statistical methods Identical same site & researchers + Protocol changes Operationa- lization changes Population changes Just the hypothesis kept RE-ANALYSIS REPETITION REPLICATION REPRODUCTION Not Run
  41. 41. Question 2 What level of similarity should an experiment have to be considered a replication rather than a new experiment?
  42. 42. Unchanged elements of a replication  A replication must share the hypothesis with the baseline experiment  Same response variable  although not same metric  Same treatments  although not same operationalization  at least two treatments in common
  43. 43. Partials Replications Exp. A (Baseline OR Replication) Exp. B (Replication OR Baseline) Exp. D Replication C RV3 RV2 RV1 T5 T4 T3 T2 T6 T1
  44. 44. Question 3 What changes are acceptable?
  45. 45. Levels of Verification  Similarity between the baseline experiment and a replication serve different verification purposes depending on the changes made  Replicating an experiment as closely as possible  Verifies results are not accidental  Varying the experimental protocol  Verifies results are not artifactual  Varying the population properties  Verifies types of populations for which the results hold  Varying the operationalization  Verifies range of operationalizations for which the results hold
  46. 46. Can I Change Everything?  It is better not to change everything at the same time  We can understand the source of differences in results better if only one change is made at a time  But a replication with a lot of changes is not rendered useless or doomed to failure  We will just need to wait until other replications are run
  47. 47. Everything Different Changes Different Identical Experiment Elements
  48. 48. One Change at a Time Changes Different Iqual Experiment Elements
  49. 49. Increasing Changes Changes Different Iqual Experiment Elements
  50. 50. Question 4 What is the level of similarity that results must have to be considered as reproduced?
  51. 51. Understanding the Natural Variation  Identical replications are useful for understanding the range of variability of results  This provides an estimate for other experimenters to use as a baseline when they replicate the experiment
  52. 52. Question 5 Should replications reuse materials?
  53. 53. Accomodating Opposing Views  The possible threat of errors being propagated by experimenters exchanging materials does not mean discarding replications sharing materials  Replications with identical materials and protocols (and possibly the same errors) are a necessary step for verifying that an exact replication run by others reproduces the same results  Other replications that alter the design and other protocol details should be performed in order to assure that the results are not induced by the protocol
  54. 54. Accomodating Opposing Views  Replication functions accommodate opposing views within a broader framework  Contrary stances are really tantamount to different types of replication conducted for different purposes  Different ways of running replications are useful for gradually advancing towards verified experimental results
  55. 55. Summarizing
  56. 56. Main Ideas  Replication in ESE  Replication plays an essential role in the experimental paradigm  Replication is not a regular practice in ESE today  More methodological research on the adoption and tailoring of replication in ESE is still necessary  Clarifying conceptions  Replication is necessary not merely to validate findings, but, more importantly, to discover the range of conditions under which the findings hold  Replications provide different knowledge depending on the changes to the baseline experiment  Knowledge gained from a replication needs to relate changes and findings
  57. 57. Towards Understanding the Replication of SE Experiments Natalia Juristo Universidad Politecnica de Madrid (Spain) & University of Oulu (Finland) ESEM Conference Baltimore (USA) October 11th, 2013

×