Conference Abstract: Did you know that there are measurement experts who spend their lives researching the validity of different question styles and formats? How ’bout we spend 30 minutes reviewing some of the most relevant findings in survey methods research!? During this presentation, you will learn about fascinating topics such as the validity of true/false questions, when to use “none of the above” style questions, and which question styles will help you assess the information you are actually trying to asses (read= minimize bias and error)! Participants will be asked to vote on sample question strength, and recraft “before’s” into highly valid “after’s.”
1. Wendy Turell, DrPH, CHCP
Senior Director, Outcomes and Analytics
PlatformQ Health Education
2. • Measurement about how people think or act gathered via Q&A process
• Can be used in experimental, quasi-experimental, and non experimental research
• Initially developed in the 1920s, now have theories and principles that guide survey
science
• Survey Methods = commonly use in CEHP
• Surveys:
• Can be administered via self administered questionnaires or interviews
• Paper/pencil, verbally, online
• Are most often quantitative (but can be qualitative)
• Are a standardized measurement instrument
• Replicable
• Can measure:
• Attitudes and opinions
• Behaviors
• Motivations
• Emotions
• Satisfaction
3. Measurement Validity
How accurately the instrument measures the construct you are trying to measure
Examples:
A Ruler = High measurement validity
Measurement reliability
How consistently your instrument measures what you are trying to measure.
Does your testing instrument generate the same result in similar circumstances?
4. Many errors “bias” research results
Examples: sampling error, nonresponse error, self-presentation/social desirability bias,
acquiescence bias, etc.
Today: Focus on measurement error
The difference between the estimate produced and the true value because respondents
gave inaccurate answers to survey questions.
Occurs when respondents are unable / unwilling to provide accurate answers
May be due several factors……..including poor question design
Does the question adequately measure the idea or concept of interest?
If not = specification error (low construct validity)
5. Pilot your questions
Psychometric review
validate your questions against an external construct (e.g., correlate with a measure that
is known to measure the construct of interest very well)
examine internal consistency of constructs
To begin with………..write solid questions by using question structures that have
been shown to elicit more valid & reliable responses!
6. TIP 1:
Question:
Measurement studies have shown that 5 option MCQ formats are more reliable and
valid than dichotomous answer “true/false” style questions:
a. True
b. False
True/false; Yes/no; Agree/disagree
Probability of guessing correctly = .5 (dichotomous) versus .2 (5 choice MCQ)
“Acquiescence” = tendency to agree as a response style
When guessing, examinees tend to lean towards “true” > “false”
Grosse, Martin E.; Wright, Benjamin D. (1985) Validity and Reliability of True-False Tests. Educational and Psychological
Measurement, 45(1): 1-13
Norcini JJ, Swanson DB, Grosso LJ, Webster GD. (1985). Reliability, validity and efficiency of multiple choice question and patient
management problem item formats in assessment of clinical competence. Med Educ. 19(3):238-47.
7. TIP 2:
Examples:
All of the Above, None of the Above, A and E, All except D
More likely to lead to measurement error
More challenging to think of good distractor answers
All of the Above:
When studied, responses skew more towards “all of the above” response (typically artificially
elevating scores)
Loss of discriminatory power in reviewing other response options
Cuing Effect of "All of the above" on the Reliability and Validity of Multiple-Choice Test Items P. H. Harasym,E. J. Leong, C. Violato, R.
Brant,and F. L. Lorscheider. Evaluation & the Health Professions, March 1998; vol. 21, 1: pp. 120-133.
8. TIP 3:
Novel targets in clear cell RCC include all of the following EXCEPT:
a. PD-1
b. PD-L1
c. MET
d. WNT pathway
Which of the following prognostic features is associated with WORST overall survival?
a. Low Ki-67 proliferative index
b. Morphology of small cells resembling follicular leukemia
c. Presence of 3 or more complex karyotype abnormalities
d. Translocation involving chromosomal 11
“not”, “except”, “least likely” etc.
Flip cognitive reasoning
> Taxing = > measurement error
9. TIP 4:
Questions that contain multiple concepts
Do you recommend changes in diet and exercise for all of your overweight patients?
In the past month, how frequently have you self-monitored your blood sugar and blood
pressure?
How satisfied are you with the accuracy and delivery of the content?
Do you desire more live and online CME activities in oncology?
Rewrite:
Do you recommend changes in diet for all of your overweight patients?
Do you recommend changes in exercise for all of your overweight patients?
10. TIP 5:
I can rarely differentiate a diagnosis of patients with schizophrenia from those
with schizoaffective disorder.
how to define “rarely”
My health has improved since participating in the educational session.
how to define “health”
I have visited my doctor in the past 6 months.
how to define “doctor” – do other healthcare providers count?
how to define “visit” – do phone/email consults count? What if you just saw the nurse?
11. Which question format do you prefer?
Question 1: Basal insulin should provide about ______ of daily insulin requirement.
a. 10%
b. 25%
c. 50%
d. 75%
Question 2: In the evaluation and diagnosis of a person with mild cognitive impairment, which of
the following biomarkers are considered most likely to indicate that MCI is due to AD:
a. A positive CSF Aβ biomarker only
b. A positive Aβ biomarker and a positive tau, FDG, or sMRI biomarker
c. Positive tau and FDG biomarkers
d. A positive FDG biomarker only
12. Which question format do you prefer?
Question 1: Some ER/LA opiates have eliminated risk of abuse.
a. True
b. False
Question 2: Patients with an unprovoked VTE are at risk for a recurrence for
what amount of time?
a. 3 months
b. 6 months
c. 12 months
d. Indefinitely
13. Based on what you learned in this session, rewrite this question:
Which is NOT your favorite color crayon?
Yellow
Blue
Green
Brick Red
14. Based on what you learned in this session, rewrite this question:
The following outcomes level is the most difficult for me to evaluate and report:
Level 3: Knowledge
Level 4: Competence
Level 5: Performance
Level 6: Patient Health
Level 7: Community Health
15. Thank you & good luck!
Contact: wturell@platformq.com
16.
17. Wednesday April 8th, 2015
1:30 PM ET
5 Tips to Make You a Survey Measurement Rock Star
Did you know that there are measurement experts who spend their lives
researching the validity of different question styles and formats? How ’bout we
spend 30 minutes reviewing some of the most relevant findings in survey methods
research!? During this presentation, you will learn about fascinating topics such as
the validity of true/false questions, when to use “none of the above” style questions,
and which question styles will help you assess the information you are actually
trying to asses (read= minimize bias and error)! Participants will be asked to vote
on sample question strength, and recraft “before’s” into highly valid “after’s.”
Notas del editor
When respondents knew A & C were correct, they’d more likely select E without thoughtfully considering B & D (cuing effect) – Harasym et al.
There are various aspects of survey methodology that can help improve validity- including sampling, question ordering, how to structure scales and ranking questions and more.
For this presentation, we will focus on ways to improve multiple choice questions- which are the most often used evaluation tool of choice in CME.