SE Myths on Replication
Natalia Juristo
Universidad Politecnica de Madrid
Spain
www.grise.upm.es/miembros/natalia/
Myths


In SE we have a mythical view about
replication in other sciences



Specially regarding replication in the hard...
Replication is Not a Myth





Replication does exist in other disciplines
To see replication and its absence in
practi...
Myth 1
Others are able to give ALL the information
to replicators
What Is All the information?


Myht




Information that other researchers (physicist
as paradigm) interchange for repl...
Inspiring quotes


“There is always something unspoken even in the middle
of the exact sciences”
“The Logic of Tacit Infe...
Inspiring quotes


“If I am trying to explain how to build the TEA laser I will not tell you that the
inductance of the t...
Inspiring quotes


“It is often hard to tell whether an inability to
replicate a result is due to a group’s failing or a
...
Myth 2
Independent replications means
0 interaction
Independence = No Interaction


Myth






Other researchers (physicist as paradigm) just
interchange materials and ne...
They Do Interact


“A paper can never be a foolproof recipe for a
replication of its results because this sort of
informa...
Inspiring quotes


“When I started in Science I was told that I
should be able to repeat an experiment by
reading the pap...
New ways of interchanging the
critical Info


PLoS ONE takes the comment model further
than anyone else with various func...
They Even Visit Each Other
“The story will be a familiar one to many biologist. After
publication, rival labs fell over th...
…Cold fusion (NYT May 1989)


Nathan Lewis (Caltech) “Pons would never
answer any of our questions. So we asked Los
Alamo...
Myth 3
Other sciences get equal results
Hard Sciences Get Same Results


Myth






If results are truth (not flaw) they get the same
results
Variability in P...
“Those of us […] know intuitively that there is something ‘softer’ and less cumulative
about our research than about those...
Myth 4
They know the relevant variables to control
&
Know how to measures them


Myth




Mature experimental sciences are able to
perfectly describe experiment variables, know
how to perfectly meas...
The Cold Fusion Case



After several trials of reproducing Pons&Fleischmann experiment and fail the
assertion was that ...
Myth 5
Learning vs. verifying
Maturity & Changes






Depending on the maturity of the experiment, the level of
similarity required can vary
When an...
Problems with Identical
Replications


SE has tried to make identical repetition of
experiments, but no exact replication...
Learning vs. Checking




A replication designed to verify results should be exact,
whereas a replication for the purpos...
Evidence Structure


Result




Constructs






A is more effective than B

A applied as follows: xxx
B applied…
E...
SE Myths on Replication
Natalia Juristo
Universidad Politecnica de Madrid
Spain
www.grise.upm.es/miembros/natalia/
Próxima SlideShare
Cargando en…5
×

Myths on Replication (LASER School Talk 2010)

319 visualizaciones

Publicado el

Publicado en: Tecnología
0 comentarios
0 recomendaciones
Estadísticas
Notas
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Sin descargas
Visualizaciones
Visualizaciones totales
319
En SlideShare
0
De insertados
0
Número de insertados
1
Acciones
Compartido
0
Descargas
5
Comentarios
0
Recomendaciones
0
Insertados 0
No insertados

No hay notas en la diapositiva.

Myths on Replication (LASER School Talk 2010)

  1. 1. SE Myths on Replication Natalia Juristo Universidad Politecnica de Madrid Spain www.grise.upm.es/miembros/natalia/
  2. 2. Myths  In SE we have a mythical view about replication in other sciences  Specially regarding replication in the hard sciences, Physics as paradigm
  3. 3. Replication is Not a Myth    Replication does exist in other disciplines To see replication and its absence in practice Nature Magazine tracked the fate of 19 papers in issue 6893 to check whether the results had been reproduced 2 years later In a large majority of cases they had. Papers on fields like Neurology, Paleontology, etc.
  4. 4. Myth 1 Others are able to give ALL the information to replicators
  5. 5. What Is All the information?  Myht   Information that other researchers (physicist as paradigm) interchange for replication is complete and perfect Reality  No, it is not. They face, as we do, two problems Tacit knowledge  Unknown relevant variables 
  6. 6. Inspiring quotes  “There is always something unspoken even in the middle of the exact sciences” “The Logic of Tacit Inference” Physicist-Chemist Michael Polanyi  “It’s very difficult to make a carbon copy. You can make a near one, but if it turns out that what is critical is the way he glued his transducers, and he forgets to tell you that the technicians always puts a copy of Physical Review on top of them for weight, well, it could make all the difference” A physicist quoted in “Changing Order” Harry Collins
  7. 7. Inspiring quotes  “If I am trying to explain how to build the TEA laser I will not tell you that the inductance of the top is important, because I assume you already know... …Finally, you do not know what you need to know and I do not know what I know... …In fact, the principle of TEA lasers, scientists did not know necessarily that the inductance of the top was important” A physicist quoted in “Tacit knowledge: you don't know how much you know” New Scientist Magazine N. 2762 Harry Collins 2010
  8. 8. Inspiring quotes  “It is often hard to tell whether an inability to replicate a result is due to a group’s failing or a flaw in the original paper. The reason is often the countless tiny details of experimental method that are omitted from the paper but can influence results” Gillian Murphy Cell Biologist University of Cambridge Quoted in “The Trouble with Replication” Nature 2006
  9. 9. Myth 2 Independent replications means 0 interaction
  10. 10. Independence = No Interaction  Myth    Other researchers (physicist as paradigm) just interchange materials and never interact that is why it is an independent replication Independent replication means do not interact with the previous experimenters at all except interchanging the protocol Reality  They interact, talk, discuss why my replication does not get the same results than yours, they visit each other to see how you run your replication,…
  11. 11. They Do Interact  “A paper can never be a foolproof recipe for a replication of its results because this sort of information can never be entirely captured in a scientific paper… Researchers compensate by exchanging tips by e-mail and at conferences. Because this social interactions are not recorded anywhere, it is hard to consult or build on them” “The Trouble with Replication” Jim Giles Nature 2006
  12. 12. Inspiring quotes  “When I started in Science I was told that I should be able to repeat an experiment by reading the paper, but that is almost never the case” Michael Ronemus Molecular and Cell Biologist Cold Spring Harbor Laboratory Editor of the Journal Cold Spring Harbor Protocols  Cold Spring Harbor Protocols, PLoS ONE and Nature Protocols include discussion facilities that allow researchers to talk to the original authors and others users of the protocol. These could prove ideal places to resolve problems that crop up during attempts to reproduce a previous result
  13. 13. New ways of interchanging the critical Info  PLoS ONE takes the comment model further than anyone else with various functions for promoting discussion  “The aim is to recreate the kind of discussion that takes place in front of conference posters” Surridge PLoS ONE Manager
  14. 14. They Even Visit Each Other “The story will be a familiar one to many biologist. After publication, rival labs fell over themselves to reproduce the results. Many contacted Verfaillie with requests for the cells and the reagents used to create them, or to ask for more details of the experimental protocol involved. Several high-profile groups sent researchers to Minnesota to learn how to extracted and culture the cells…the procedure takes up to six weeks to master…” “The Trouble with Replication” Jim Giles Nature 2006
  15. 15. …Cold fusion (NYT May 1989)  Nathan Lewis (Caltech) “Pons would never answer any of our questions. So we asked Los Alamos National Lab to put our questions to him instead, since they were in touch with him”  Edward Redish (Univ of Maryland) “We have invited Fleischmann to participate in the Baltimore sessions“  “Many speakers at the meeting reported failure in their effort to elicit information or comments from Pons”
  16. 16. Myth 3 Other sciences get equal results
  17. 17. Hard Sciences Get Same Results  Myth    If results are truth (not flaw) they get the same results Variability in Physics experiments results is very low Reality  Variation of experimental results is also very common in Physics
  18. 18. “Those of us […] know intuitively that there is something ‘softer’ and less cumulative about our research than about those of physical sciences. […] distinguished researchers have cited the pervasive presence of interactions or historical influences as reasons not to expect a cumulative […] science. Still others have cited the low quality of data […] as a barrier.” “What is surprising is that the research results in the physical sciences are not markedly more consistent […] data do suggest that results from replicated experiments do not always tend to be consistent” “The data […] are striking. When all studies are included in the quantitative reviews, the average Birge ratio is over 2.00, which is 100% larger than expected when studies yield consistent results. Moreover, 6 of the 13 reviews (46.2%) show statistically significant disagreement among studies.” L. Hedges How hard is hard science, how soft is soft science? The empirical cumulativeness of research American Psychologist 1987
  19. 19. Myth 4 They know the relevant variables to control & Know how to measures them
  20. 20.  Myth   Mature experimental sciences are able to perfectly describe experiment variables, know how to perfectly measure response variables, etc. Reality  Even in Physics they can discuss about interpretation of results or adequate ways of measuring response variables
  21. 21. The Cold Fusion Case   After several trials of reproducing Pons&Fleischmann experiment and fail the assertion was that cold fusion observations were based on experimental errors What type of errors can be done in a Physics experiment?  Using more sensitive equipment. Failing to install a stirring device in the test cell, temperature differences in the cell led to false estimates of its overall heat. This may have suggested that its cell was producing fusion energy   Regarding helium traces in P&F experiment: helium is a trace component of air, and the amount of helium in the cell corresponded to what normally enters from the atmosphere.    Results can be interpreted differently Regarding the burst of heat that was observed: Someone turned the current off for a while. When that happens hydrogen naturally bubbles out of the palladium cathode, and creates a hazard of fire or explosion. It is a simple chemical reaction that has nothing to do with fusion.   So measuring response variables are sometimes not direct or well-defined Again results interpretation is not clear The experiment was flawed because of the system used to measure heat. There were error on temperature measurement. Meyerhof of Stanford University said: “Some scientist put a thermometer at one place and not another” About Jones replication of P&F  had used relatively crude neutron-detecting equipment, and had measured only a very small excess of neutrons over what could be expected from natural sources without any fusion
  22. 22. Myth 5 Learning vs. verifying
  23. 23. Maturity & Changes    Depending on the maturity of the experiment, the level of similarity required can vary When an experimental study is young there are many aspects that are unknown. Any change in replication might cause a variation in the results of the experiment impossible to be interpreted In immature experimental studies begin replication as similar as possible can know the conditions to be controlled
  24. 24. Problems with Identical Replications  SE has tried to make identical repetition of experiments, but no exact replications have yet been achieved  The complexity of the software development setting prevents the many experimental conditions from being reproduced identically
  25. 25. Learning vs. Checking   A replication designed to verify results should be exact, whereas a replication for the purposes of knowledge discovery and learning should be different Both are necessary. At the beginning of experimental research, equality is not even an option    The critical variables to be controlled are now known! Later on, both knowledge discovery and testing can be more systematic In the early stages, failure to get the expected results should not be construed as falsification, but as a step towards the discovery of some new factor
  26. 26. Evidence Structure  Result   Constructs     A is more effective than B A applied as follows: xxx B applied… Effectiveness measured … Population    For inexperienced subjects For small programs For these types of faults: omission, …
  27. 27. SE Myths on Replication Natalia Juristo Universidad Politecnica de Madrid Spain www.grise.upm.es/miembros/natalia/

×