1. What does it take to perform a
Systematic Review of animal studies?
Carlijn Hooijmans
C.Hooijmans@cdl.umcn.nl
Central Animal Laboratory, SYstematic Review Centre for Laboratory
animal Experimentation; SYRCLE
Radboud University Medical Centre
www.umcn.nl/SYRCLE
1

2. Systematic steps:
1. Phrase the research question
2. Define in- and exclusion criteria
3. Search systematically for ALL original papers
4. Select relevant papers
5. Assess study quality and validity
6. Extract data
7. Analyze results (when possible perform MA)
8. Interpret and present data
2

3. Focus of this presentation:
1. Phrase the research question
2. Define in- and exclusion criteria
3. Search systematically for ALL original papers
4. Select relevant papers
5. Assess study quality and validity
6. Extract data
7. Analyze results (when possible perform MA)
8. Interpret and present data
3

4. Focus of this presentation:
1. Phrase the research question
2. Define in- and exclusion criteria
3. Search systematically for ALL original papers
4. Select relevant papers
5. Assess study quality and validity
6. Extract data
7. Analyze results (when possible perform MA)
8. Interpret and present data
4

5. Goal of Searching systematically
• Detect the maximum amount of available information
Advantages of Searching systematically
• Prevent false or imprecise conclusions
• Prevent unnecessary experimentation
• Obtain new insights that may arise from aggregating
earlier work
5

6. How to search systematically?
• Formulate adequate and specific research question
 Define: - Disease of interest/ health problem
- Population
- Intervention/exposure
- Outcome measures
 Example: „What is the effect of [intervention/exposure] on [outcome
measures] in [population studied] for [disease of
interest/health problem]?
• Identify appropriate databases
• Create and run a comprehensive search strategy
6

7. How to search systematically?
• Transform research question into search strategy for PubMed
 Critical search components (SC).
• Identify relevant search terms for each SC.
o Identify Medical Subject Headings (MeSH) terms
"fatty acids, omega-3"[MeSH Terms]
o Identify free-text terms (synonims)
Fish oil [tiab], Omega-3 [tiab] , PUFAs [tiab], DHA [tiab], EPA [tiab], omega 3 fatty acids
[tiab], omega 3 fatty acid [tiab], polyunsaturated fatty acids [tiab] etc…..
o Combine MeSH terms and free-text terms
"fatty acids, omega-3"[MeSH Terms] OR Fish oil [tiab] OR
Omega-3 [tiab] OR, PUFAs [tiab] OR etc
• Repeat step 1-3 for every SC
• Combine SCs
7

8. Example: Systematic Review about supplementation of omega-3
fatty acids in animal models for Alzheimer’s Disease
Quick search in PubMed:
8

9. Example: Systematic Review about supplementation of omega-3
fatty acids in animal models for Alzheimer’s Disease
Comphrensive search strategy in PubMed:
Omega-3
Alzheimer‟s
Animal models
9

10. More information about searching
systematically and finding all animal studies?
10

11. Focus of this presentation:
1. Phrase the research question
2. Define in- and exclusion criteria
3. Search systematically for ALL original papers
4. Select relevant papers
5. Assess study quality and validity
6. Extract data
7. Analyze results (when possible perform MA)
8. Interpret and present data
11

12. Why should we critically appraise the included papers?
• Low methodological quality often causes bias in the study
results
 overestimation or underestimation
Macleod et al. Stroke, 2008
12

13. Why should we critically appraise the
included papers?
• The conclusions resulting from a SR/meta-analysis are
dependent of the quality of the original included studies
 Transparancy >>>> improves interpretation/reliability
 Invalid studies may produce misleading results
Garbage in = garbage out
13

14. How do we critically appraise the included
papers?
• 2 dimensions:
External validity
Internal validity
• External validity:
 Generalizability of the study results
• Internal validity:
 The extent to which the results of a study are correct for
the circumstances being studied (methodological quality)
 Threatened by bias (systematic errors)
14

15. How do we critically appraise the included
papers?
Type of bias Description Solution
Selection bias: Systematic differences between baseline Randomization
characteristics of the groups that are
compared
Performance Systematic differences between groups Allocation concealment
bias: in the care or in exposure to factors other
than the intervention of interest
Detection bias: Systematic differences between groups in Blinding
how outcomes are determined
Attrition bias: Systematic differences between groups in Reporting drop outs
“drop outs” from a study (reason and nr)
15

16. How do we critically appraise the included
papers?
0% 20% 40% 60% 80% 100%
1) Was it stated in the method section that the experiment was randomized?
2) In case the answer at 1) was yes: Was the method of randomization adequate?
2) In case the answer at 1) was no: Were the groups similar at baseline?
3) Was the allocation to the different groups during the randomization process concealed?
4) Were the caregivers blinded for the allocation of the animals to the specific groups
5) Was the outcome assessment blinded?
6) Methods for outcome assessment the same in both groups?
7) Is the timing of the intervention during the day similar in both groups?
8) Was the outcome assessment randomized across the groups?
9) Number of excluded animals specified per experimental group for each outcome measure?
10) Reason for exclusion mentioned for each excluded animal?
yes no unclear na
Yes=Low risk of bias. No= High risk of bias. ?=Unclear risk of bias. N.a.=not applicable
16

17. 17

18. Focus of this presentation:
1. Phrase the research question
2. Define in- and exclusion criteria
3. Search systematically for ALL original papers
4. Select relevant papers
5. Assess study quality and validity
6. Extract data
7. Analyze results (when possible perform MA)
8. Interpret and present data
18

19. Why conducting meta-analyses?
• To increase power
• To increase the precision of estimates of treatment
effects
• To obtain new information about safety and efficacy of
treatments that is not directly visible in the individual
studies
• To generate new hypothesis
19

20. How to conduct a meta-analyses?
Cell death Data from individual studies Adapted data for MA
Experimental group Control group
measure mean sd n mean sd n effect effect size (confidence interval)
Study A % 1.71 -0.05 6 1,9 0,07 5 significant -2.91 [-4.83, -0.98]
Study B ng/ml 104 -6.29 6 114 6,28 6 significant -1.51 [-2.86, -0.16]
Study C % 1 -0.17 12 1,3 0,35 12 significant -1.05 [-1.92, -0.19]
Study D nr of cells -241 14,7 8 -233 17,8 7 not significant -0.50 [-1.53, 0.54]
Study E nr of cells -190 15,1 6 -168 17,6 7 significant -1.24 [-2.47, -0.01]
Study F nr of cells -161 14,6 7 -181 10,3 7 not significant -0.19 [-1.24, 0.86]
Overall -1.02 [-1.61, -0.43]
-5 0 +5
20

21. Subgroup analyses (fictive dataset)
Overall -1.47 [2.33, -0.61]
21

22. Subgroup analyses (fictive dataset)
Overall -1.47 [2.33, -0.61]
Males -0.79 [-1.51, -0.06]
Females -3.64 [-4.78, -2.50]
22

23. Subgroup analyses (fictive dataset)
Overall -1.47 [2.33, -0.61]
Males -0.79 [-1.51, -0.06]
Females -3.64 [-4.78, -2.50]
Hippocampus -1.92 [-2.65, -1.19]
Cortex 0.37 [-0.24, 0.98]
23

24. Subgroup analyses (fictive dataset)
Overall -1.47 [2.33, -0.61]
Males -0.79 [-1.51, -0.06]
Females -3.64 [-4.78, -2.50]
Hippocampus -1.92 [-2.65, -1.19]
Cortex 0.37 [-0.24, 0.98]
24

25. So we discussed:
1. Phrase the research question
2. Define in- and exclusion criteria
3. Search systematically for ALL original papers
4. Select relevant papers
5. Assess study quality and validity
6. Extract data
7. Analyze results (when possible perform MA)
8. Interpret and present data
25

26. Narrative review versus systematic review
Feature Narrative review Systematic review
Research Often unclear or broad Specified and specific
question
Sources and Not usually specified Comprehensive and explicit
search search strategy.
More than 1 database.
Study Not usually specified Explicit selection criteria
selection
Risk of bias Not usually present or only Critical appraisal on the basis
assessment implicit of explicit criteria
Data Often a qualitative summary Often also a quantitative
synthesis summary (meta-analysis)
26

27. 27