Revisiting publication at Nature Neuroscience in 2009 by Kriegeskorte, N. et al. on "Circular analysis in systems neuroscience: the dangers of double dipping"

1. 1 / 12
Journal Club – Music and Neuroscience Lab
Ana Luísa Pinho
3rd
of June, 2022

2. 2 / 12
Outline
●
“Show me the data.” = Show me the results.
●
Between raw data and results, we have the analysis.
●
Results reflect data indirectly, through the lens of the
analysis based on assumptions.
●
Ideally, no distortion caused by analysis and assumptions.

3. 3 / 12
The problem
●
Results shall reflect an aspect of the data filtered by the
assumption (left).
●
Results are predetermined by assumptions, the analysis
is circular.
●
Assumptions modify the results (center).
●
Assumptions tinge the results (right)

4. 4 / 12
Selection Criteria
●
Selection criteria is the most frequent reason for
distortion of results.
●
“Double Dipping”: using the same data for selection and
selective analysis
●
Data = true effects + noise, therefore selection is
affected by noise
●
Selection criteria is only valid when results are
statistically independent of the selection criteria under
the null hypothesis.

5. 5 / 12
Identifying the problem in the
literature: study overview
●
All fMRI studies from 2008 published in 5 journals were
examined.
●
Journals: Nature, Science, Nature Neuroscience, Neuron and
Journal of Neuroscience
●
Total: 134 papers
●
42% (57 papers) contained at least one non-independent
selective analysis

6. 6 / 12
Identifying the problem in the
literature: conclusions
●
Disclaimer: in many cases, the overall claim did not
depend directly on the distorted result.
●
Yet, the problem is frequent and compromises
transparency.
●
The problem is frequent because the desired solution
criterion is often related to the desired results in the
selective analysis.

7. 7 / 12
Identifying the problem in the
literature: take-home messages
●
Open-science and reproducibility
●
In neuroimaging:
– Whole-brain mapping that avoids selective bias
should be preferred.
– In-depth ROI analysis: use a different different dataset
for the selective analysis
– Use an ROI from a different study (e.g. meta-analysis)
to conduct the selective analysis

8. 8 / 12
Demonstration of the problem
●
Widely accepted neuroimaging methods applied
to random data:
– Example 1: Pattern-information analysis
– Example 2: Regional-activation analysis

9. 9 / 12
Example 2:
Regional Activation Analysis (1/2)

10. 10 / 12
Example 2:
Regional Activation Analysis (2/2)

11. 11 / 12
Example 1:
Pattern-Information Analysis

12. 12 / 12
Policy for noncircular analysis