2. Whatis meta-analysis? Science is a cumulative process . Therefore, it is not surprising that one can often find dozens and sometimes hundreds of studies addressing the same basic question. Researches trying to aggregate and synthesize the literature on a particular topic are increasinglyconducting meta-analyses
3. Why do we need meta-analyses? Literature expansion in research Allows researchers the ability to statistically combine countless studies to increase power Allows us to measure “how much” of a relationship exists, rather than just whether a relationship exists Allows us to account for the variation in results between similar studies based on procedural characteristics of individual studies
4. What is meta-analysis? A standardized secondary analysis of primary data results from different studies that share same hypothesis A quantitative aggregation of findings during a research synthesis Calculating a standardized effect size for multiple studies
5. Effect Size in MA Effect size makes meta-analysis possible it is the “dependent variable” it standardizes findings across studies such that they can be directly compared Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio) as long as: It is comparable across studies It represents the magnitude and direction of the relationship of interest It is independent of sample size Different meta-analyses may use different effect size indices
6. Examples of Different Types of Effect Sizes: Standardized Mean Difference (continuous outcome) group contrast research treatment groups naturally occurring groups Odds-Ratio (dichotomous outcome) group contrast research treatment groups naturally occurring groups Correlation Coefficient association between variables research
7. Statistical significance Turns out a lot of researchers do not know what precisely p < .05 actually means Cohen (1994) Article: The earth is round (p<.05) What it means: "Given that H0 is true, what is the probability of these (or more extreme) data?” Trouble is most people want to know "Given these data, what is the probability that H0 is true?"
8. Always a difference With most analyses we commonly define the null hypothesis as ‘no relationship’ between our predictor and outcome (i.e. the ‘nil’ hypothesis) With sample data, differences between groups always exist (at some level of precision), correlations are always non-zero. Obtaining statistical significance can be seen as just a matter of sample size Furthermore, the importance and magnitude of an effect are not accurately reflected because of the role of sample size in probability value attained
9. What should we be doing? We want to make sure we have looked hard enough for the difference – power analysis Figure out how big the thing we are looking for is – effect size Effect size refers to the magnitude of the impact of some variable on another
10. Examples of Different Types of Effect Sizes: Standardized Mean Difference (continuous outcome) group contrast research treatment groups naturally occurring groups Odds-Ratio (dichotomous outcome) group contrast research treatment groups naturally occurring groups Correlation Coefficient association between variables research
11. The Standardized Mean Difference Represents a standardized group comparison on a continuous outcome measure. Uses the pooled standard deviation (some situations use control group standard deviation). Commonly called “Cohen’s d” or occasionally “Hedges’ g”.
12. The Correlation Coefficient Represents the strength of association between two continuous measures. Generally reported directly as “r” (the Pearson product moment coefficient).
13. Odds-Ratios The Odds-Ratio is based on a 2 by 2 contingency table, such as the one below. The Odds-Ratio is the odds of success in the treatment group relative to the odds of success in the control group.
14. Converting results into a common metric Can convert p-values t, F, etc. into the standardized effect size metric being used in the meta-analysis (e.g., d, r)
15. Interpreting Effect Size Results Cohen’s “Rules-of-Thumb” standardized mean difference effect size small = 0.20 medium = 0.50 large = 0.80 correlation coefficient small = 0.10 medium = 0.25 large = 0.40
17. Cohen’s d Now compare to the one-sample t-statistic So This shows how the test statistic (and its observed p-value) is in part determined by the effect size, but is confounded with sample size This means small effects may be statistically significant in many studies (esp. social sciences)
18. Example Average number of times MGEC students curse in the presence of others out of total frustration over the course of a day Currently taking R course vs. not Data:
19. Example Find the pooled variance and sd Equal groups so just average the two variances such that and sp2 = 6.25
20. Odds ratios Especially good for 2X2 tables Take a ratio of two outcomes Although neither gets the majority, we could say which they were more likely to vote for respectively Odds Clinton among Dems= 564/636 = .887 Odds McCain among Reps= 450/550 = .818 .887/.818 (the odds ratio) means they’d be 1.08 times as likely to vote Clinton among democrats than McCain among republicans However, the 95% CI for the odds ratio is: .92 to 1.28
21. Voting Method Voting method was commonly employed for aggregation of studies before the conception of meta-analysis Procedure: Studies with a dependent variable and a specific independent variable are examined Studies are dichotomized as either statistically significant or not statistically significant Classification with higher tally is considered to be the “true” relationship between variables
22. Voting Method Researcher A is conducting a study on the effects of RtI on a group of 1st graders’ fluency rate. In A’s study, which has a sample size of n=180, 110 children are given RtI and 70 children are given traditional instruction. After 12 weeks of instruction, children are dichotomized as either “pass” or “fail” on a reading measure. The improvement rate for the RtI group is .45 vs. .43 for the control group.
23. Voting Method Researcher B conducts the same study at a different site In B’s study, which has a sample size of n=230, 90 children receive RtI and 140 receive traditional instruction Again the improvement rate for the RtI group is .67 vs. .64 for the control group. That’s 2-0 for the experimental group!
24. Aggregation of Raw Data Suppose another researcher aggregates the data from the same studies by summing the raw data instead of employing the voting method Add the number of subjects in both studies that received treatment and control: n=200 received RtI and n=210 received traditional instruction When dichotomized into “pass” or “fail”, the improvement rate for the treatment group is now .55 vs. 0.57 for the control group! This is known as Simpson’s Paradox
25. Voting Method Flaws: Bias in favor of large-sample studies Why is this a problem? No weighting of sample size Tells us nothing about strength of relationship Does not control for variation between studies
26.
27. What do the preponderance of your studies report as an effect size, if any?
30. Calculating Effect Sizes So, if you were to calculate the standardized mean difference in the fluency rate of the following two groups in an RtI study, what would you get as the effect size? Group 1 (experimental): M1 = 80, SD1 =10, n1= 250 Group 2 (control): M2 = 65, SD2 = 20, n2=230 Effect size = ? What if you had three groups?
31. Calculating Effect Sizes What if the means and SDs aren’t reported and you only have a t-value? What if you have the F-value for two means? Formula for d-index when the F-value of two means is reported: dferror = (n1+n2-2)
32. Calculating Effect Sizes r-index The correlation coefficient tells you about the strength of the relationship between two variables Most appropriate metric for expressing an effect size when interested in the relationship strength of two continuous variables Most common in correlational studies Usually reported when appropriate EX: relationship between years of schooling and yearly salary
33. Calculating Effect Sizes What if you only have a t-value? Formula for r-index when only t-value is reported:
37. Calculating Effect Sizes Odds-Ratio (OR) Applicable when both variables are dichotomous The relationship between two sets of odds EX: Suppose a study measures the effects of RtI on whether students in two groups (e.g., experimental/control ) “pass” or “fail” a math test.
38. Calculating Effect Sizes Of n=80 in RtI, the ratio of passing is 15 to 1. Of n =65 in control, the ratio of passing is 1.6 to 1. Calculate the odds ratio: OR = ad/bc= ?
39. Combining Effect Sizes Once individual study effect sizes have been calculated, the next step involves combining them to provide an average effect size. You must weight the individual effect sizes. What do you base this weight on?
40. Combining Effect Sizes Suppose you have 7 d-indexes and group ns that compares the effect of homework vs. no homework on a measure of academic achievement:
45. Combining Effect Sizes Step Three: Divide the sum of these products by the sum of the weights. Formula: EX: d. = 62.56/544.27 = +.115 (average ES)
46. Combining Effect Sizes Step Four: Computing Confidence Intervals Formula: EX: Thus, we expect 95% of estimates of this effect to fall between .031 and .199. Do we reject the null?
47. Combining Effect Sizes Suppose that you have 6 r-indexes and ns that show the relationship between the amount students spend on homework and their score on an achievement test. Step One: Transform the r-indexes into a z-scores because as r gets larger the distribution gets more skewed. Formula:
52. Combining Effect Sizes Step Four: Divide the sum of these products by the sum of the weights. Formula: EX: z. = 5462.97/26,372 = +.207 (average ES)
53.
54.
55. As sample size increases, the width of the confidence interval should decrease.
56.
57.
58. If most studies are similar to each other and show a similar result (low heterogeneity), this increases confidence that the effect being measured is real.
59. If results from different studies are vastly different from each other, this suggests that each study is measuring something slightly different from the other studies.
73. Tuberculosis The data set taken from van Houwelingen, Arends, and Stijnen (2002) consists of randomized controlled trials of a vaccine, Bacillus Calmette-Guerin (BCG), for the preventionof tuberculosis (TB). The data presented consist of the sample size and the number of cases of tuberculosis. Furthermore some covariates are available that might explain the heterogeneity among studies: geographic latitude of the place where the study was done, year of publication, and method of treatment allocation (random, alternate, orsystematic).
77. Dentifrices The data set is taken from Abrams and Sanso (1998) and concerns a previously published meta-analysis which was conducted of all randomized controlled trials comparing sodiummonofluorophosphate (SMFP) tosodiumfluoride (NaF) dentifrices(toothpastes) in the prevention of caries; see Johnson (1993). The outcome in each trial was the change from baseline in the decayed missing (due to caries) filled surface (DMFS) dental index at three years follow-up.
81. Validity Thestudies were usually conducted in multisection courses in which the sections had different instructors but all sections used a common final examination. The index of validity was a correlation coefficient (a partial correlation coefficient, controlling for a measure of student ability) between the section mean instructor ratings and the sectionmean examination score.
