Accountability has to do with showing our programs make a difference We need to sow this through the data we produce and the comparisons we make, not just people saying it helps
Research used in everyday speech to cover a broad spectrum of meaning Multiple, misleading meanings Giving and recording known facts and the opinions of others; looking at existing data
Formal definition What it really means Opinion gathering versus research Ex. Stories of how students feel TRiO has helped them Vs. TRiO students versus others--change in grade between junior and senior year
World filled with unanswered questions, unresolved problems Observe things that cause us to wonder, to speculate, to ask questions Inquisitive mind is beginning of research What’s the cause of that? What does it mean? With questions like these, research begins 2. A clear unambiguous statement of the problem is critical 3. Carefully planned attack, search and discover mission explicitly planned in advance 4. Whole is composed of sum of it’s parts Main Problem: How do I get from Town A to Town B Subproblems: 1. What is the most direct route? 2. How far do I travel on the freeway? 3.What is the number of the exit I take? Subproblems keep main problem from becoming cumbersome and unwieldy 5. Each subproblem is a hypothesis After the hypothesis comes the data Reach to turn on lamp and it doesn’t turn on Hypotheses burnt out bulb (put in new bulb) not plugged in (plugged in) wire from lamp to wall is defective (lift the cord, lights briefly) forgot to pay electric bill
Equivalent to axioms in geometry - self-evident truths Example: do students more quickly acquire the ambience of a foreign language (native accent , comprehension of the expressive characteristics of grammatical, idiomatical accentual, etc)by learning only one foreign language at a time or by learning two languages at a time What assumptions underlie such a problem Teacher competent to teach the language and mastered linguistic ambience Student capable of hearing subtleties of accent and alert to other unique characteristics of language Languages selected would have distinguishable ambience characteristics that could be recognized, learned and practiced 8. A. Questioning b. Formally stated as a problem c. Data gathered d. Data point to tentative solution - hypothesis is made e. Quest for more data f. Data processed and interpreted g. Conclusion is reached h. Hypothesis supported or not supported i. Cycle complete
F1 = quantitative - objective F2 = qualitative - subjective GPA - What is your GPA vs do you get good or bad grades?
Rigorous or strong research
Random assignment assumes that attributes of both groups are equal; need to double check Tries to remove any subjective influences
A randomized controlled trial of Even Start - a federal program designed to improve the literacy of disadvantaged families--found that the program had no effect on improving the school readiness of participating children at the 18th month follow-up. Specifically, there were no significant differences between young children in the program and those in the control group on standard measures of school readiness. If a pre-post design rather than a randomized design had been used in this study, the study would have concluded erroneously that the program was effective in increasing school readiness. This is because both the children in the program and those in the control group showed improvement in school readiness during the course of the study (both groups of children improved substantially in their national percentile ranking on the tests). A pre-post study would have attributed the participants’ improvement to the program whereas in fact it was the result of other factors, as evidenced by the equal improvement for children in the control group. Example: A randomized controlled trial of the Summer Training and Education Program -- a Labor Department pilot program that provided summer remediation and work experience for disadvantaged teenagers--found that program’s short term impact on participant’s reading ability was positive. Specifically, while the reading ability of the control group members eroded by a full grade level during the first summer of the program, the reading ability of participants in the program eroded by only half a grade level. If a pre-post design rather than a randomized design had been used in this study, the study would have concluded erroneously that the program was harmful. That is, the study would have found a decline in participants’ reading ability and attributed it to the program. In fact, however, the participants’ decline in reading ability was a result of other factors--such as the natural erosion of reading ability during the summer vacation months--as evidenced by the even greater decline for members of the control group.
Example from medicine: Over the past 30 years, more than two-dozen comparison group studies have found hormone replacement therapy for postmenopausal women to be effective in reducing the women’s risk of coronary heart disease, by about 35-50 percent. But when hormone therapy was finally evaluated in two large scale randomized controlled trials--it was actually found to do the opposite In most cases well-matched comparison groups ( prior test scores, demographics, time period in which they are studied and methods used to collect outcome data) seem to yield correct overall results about whether an intervention is effective, ineffective, or harmful However, their estimates of the size of the interventions impact are still often inaccurate In medicine, subjectively give most aggressive therapies to best patients Group needs to be large enough to be comparable in all other relevant features
Large enough samples to get randomization
Gives you a comparison group
Next I’ll talk about why prior experience and why two stages
