Kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen

Instrumentasi:
Pembentukan Instrumen,
Kesahan dan
Kebolehpercayaan
Instrumen
Powerpoint Templates Mengilham harapan, mencipta masa depan
Inspiring Futrures, Inspiring Possibilities

Main Steps in Quantitative
Research:
1. Theory
2. Hypothesis
3. Research design
4. Plan/Develop/Create measures of concepts
5. Select research site(s)
6. Select research subjects/respondents
7. Administer research instruments/ collect data
8. Process data
9. Analyse data
10. Write up findings and conclusions
Powerpoint Templates

Examples of Quantitative Research
Methods:
Experiments
Social surveys
Cross-sectional
Comparative (cross-national)
Longitudinal
Content Analysis
Statistical Analysis
Official Statistics
Demography
Epidemiology
Field stimulations
Structured Interviews and Observation

Terms “survey” and “questionnaire” are
often used interchangeably
“Survey” refers to a descriptive
research technique or methodology
(also includes interviews, and focus
groups)
“Questionnaire” refers to a paper and
pencil instrument through which
information is obtained
“Survey”/Tinjauan

• Useful for surveying groups of any size
• Typically designed to determine
opinions, attitudes or present practices
• Limitation: rely on self report
information v. observed behavior
• Challenges to validity become a
concern

Validity: the extent to which the test measures
what it significances/meaning/senses to
measure
Internal Validity: Relates to instrument
(questionnaire) quality. Are the appropriate
questions asked clearly and logically?
External validity: can the results be
generalized?
Measurement error: difference between the
characteristics of your sample and the
characteristics of the population
Non-response bias: answers of respondents
do not mirror the characteristics of those
who refused participation.
Validity

Create a questionnaire that is clear
and logical
Content Relevance And Content
Representativeness
Content Relevance: are the questions relevant to
the purpose of the questionnaire? Unimportant
content should be eliminated.
Content Representativeness: Are the questions an
adequate representation of the universe of
possible questions?
Solicit the input of experts in scale development
and the content of the questionnaire for an
objective evaluation

QUESTIONNAIRE PLANNING
• Determine the Objective
– Is the development of a new questionnaire
• Necessary?
• Feasible ?
(practicable/viable/achievable/possible..etc)
– What do you want to ask and of whom?
– Consider how the data will be analyzed. What
information is needed?
• Select mode of administration
– Web Based?
– Paper and pencil?
– Combination?

QUESTIONNAIRE PLANNING
• Determine the Sampling Methodology
– To whom will you send the questionnaire? Who can supply
the information?
– Will everyone be surveyed?
• ALL current students? ALL alumni?
• If yes, this is a census of the finite population
– The selection of a representative sample of the population
may be more effective based on time and cost of
administration
• Decision depends on the size of the finite population

CONSTRUCTING the QUESTIONNAIRE
• Consider what objective each question measures
• Three to five questions per objective is recommended
• How many questions?
– Find a balance: long enough to obtain the necessary
information but short enough so respondents won’t lose
interest.
– Questionnaires that are too long won’t be completed
– Goal: 25-30 minutes MAX

DESIGNING THE
QUESTIONNAIRE
• Question Formats: Open Ended
Questions
– Allow respondents to express feelings and expand on ideas.
– Question should be phrased to avoid a one word response.
• “What aspects of the Sociology program best prepared
you for your current position?”
– Open ended questions at the beginning of the questionnaire
may be discouraging to respondent – include one or two at
the end
– Analysis of responses can be challenging

QUESTIONNAIRE
DEVELOPMENT
• Question Formats: Closed Questions
– Ranking
• Forces the respondent to rank order responses based on a
value judgment.
• Rank the following items with regard to how you spend time
at Bucknell from 1 (spend the most time) to 5 (spend the
least time).
_____ Studying
_____ Socializing
_____ Volunteering
_____ Working at a part time job
_____ Exercising

QUESTIONNAIRE
DEVELOPMENT
– Checklist
• A number of possible answers are provided and the
respondent is asked to choose one or choose all that apply.
• How did you finance your Bucknell University education?
Choose all that apply.
_____ Parental contribution
_____ Student Loan
_____ Work Study
_____ Off campus employment
_____ Other (please specify): _____________

QUESTIONNAIRE
DEVELOPMENT
– Scaled items
• Very common
• Indicate strength of agreement or disagreement with a
statement.
• Numbers are assigned to each response on the continuum
• The English curriculum at Bucknell University adequately
prepared me for my current position.
Strongly
1 2 3 / No
4 5
Disagree
Disagree
Neutral Opinion
Agree
Strongly
Agree

QUESTIONNAIRE
DEVELOPMENT
– Suggestions for writing scaled items
• Use positively worded or negatively worded items
consistently
• Use an odd number of responses along the continuum to
allow for a neutral response
• Anchors may be different based on focus of the item
(Strongly Agree – Strongly Disagree; Frequently – Never;
Very Good – Very Poor)
• Keep items short

QUESTIONNAIRE
DEVELOPMENT
• Question Formats: Categorical
– Typical demographic questions:
• Yes/No/?, Male/Female, F/S/J/S/G
– Answers fall into certain categories – respondents pick the one
that applies to them
– Analyzed using frequency, proportion, percentages
– 2 considerations:
• Questions on a continuum should stay on a continuum if they
will be directly analyzed
– Example: Age – allow respondent to write in their age
rather than check off a box for their age range
• Categories must be collectively exhaustive & mutually
exclusive (next slide)

QUESTIONNAIRE
DEVELOPMENT
• Problematic:
• Number of hours spent
studying per week:
– 18 – 21
– 21 – 24
– 24 – 27
• Categories are not
mutually exclusive
• Corrected:
• Number of hours spent
studying per week:
– 18 – 21
– 22 – 25
– 26 – 29

IMPORTANT
CONSIDERATIONS
• APPEARANCE AND DESIGN
– Have clear directions that are age appropriate & easy to
follow
– Format, size, reproduction should facilitate completion
• No less than 12 pt
• Times, Arial, Courier
• PILOT STUDY
– Give the questionnaire to a few people to try it out
– Have them evaluate readability and clarity of questions
– Estimate time required to complete the questionnaire
– Suggestions for revisions

SENDING IT OUT
• INITIAL CONTACT / COVER LETTER
– Short & to the point
– Identify person by name
– Convince respondent that participation is
important
– Ensure confidentiality
– Deadline for return of questionnaire
– SASE for return (mailed questionnaires only)

FINAL THOUGHTS
• FOLLOW UP PROCEDURES
– Send to all participants to preserve confidentiality (“If you
haven’t already returned the questionnaire…..”)
– May include a second copy if available
– Usually done in 2 week intervals
• RESPONSE RATE FOR MAILED QUESTIONNAIRES
– Typically low - 30% is considered good
– Higher response rate with web-based questionnaires
– Good follow-up increases response rate

SUMMARY OF STEPS
1. Determine the Objective (consider analyses)
2. Determine the Mode of Administration
3. Determine the Sampling Methodology
4. Construct the Questionnaire
5. Institutional Approval
6. Conduct the Pilot Study
7. Write the Initial Communication
8. Send the Questionnaire
9. Follow up
10. Analyze the Results

Research design and methods
Develop your own
research design
and methods of
your study

DEVELOPING QUALITATIVE
INSTRUMENT

Main Steps in Qualitative
Research:
1. General research question
2. Select relevant site(s) and subjects
3. Collection of relevant data
4. Interpretation of data
5. Conceptual and theoretical work
6. Tighter specification of the research question
7. Collection of further data
8. Conceptual and theoretical work
9. Write up findings

Examples of Qualitative Research
In-depth Interviews
Focus Groups
Ethnography/Field Research
Historical-Comparative Research
Discourse Analysis
Narrative Analysis
Media Analysis

RELIABILITY & VALIDITY OF
INSTRUMENT - QUANTITATIVE
INSTRUMENT

CCrriitteerriiaa ooff MMeeaassuurreemmeenntt QQuuaalliittyy
• HHooww ddoo wwee jjuuddggee tthhee rreellaattiivvee ssuucccceessss ((oorr
ffaaiilluurree)) iinn mmeeaassuurriinngg vvaarriioouuss ccoonncceeppttss??
– RReelliiaabbiilliittyy –– ccoonnssiisstteennccyy ooff mmeeaassuurreemmeenntt
– VVaalliiddiittyy –– ccoonnffiiddeennccee iinn mmeeaassuurreess aanndd ddeessiiggnn

RReelliiaabbiilliittyy aanndd VVaalliiddiittyy
• RReelliiaabbiilliittyy ffooccuusseess oonn mmeeaassuurreemmeenntt
• VVaalliiddiittyy aallssoo eexxtteennddss ttoo::
• PPrreecciissiioonn iinn tthhee ddeessiiggnn ooff tthhee ssttuuddyy –– aabbiilliittyy ttoo
iissoollaattee ccaauussaall aaggeennttss wwhhiillee ccoonnttrroolllliinngg ootthheerr
ffaaccttoorrss
– ((IInntteerrnnaall VVaalliiddiittyy))
• AAbbiilliittyy ttoo ggeenneerraalliizzeedd ffrroomm tthhee uunniiqquuee aanndd
iiddiioossyynnccrraattiicc sseettttiinnggss,, pprroocceedduurreess aanndd
ppaarrttiicciippaannttss ttoo ootthheerr ppooppuullaattiioonnss aanndd ccoonnddiittiioonnss
– ((EExxtteerrnnaall VVaalliiddiittyy))

RReelliiaabbiilliittyy
• CCoonnssiisstteennccyy ooff MMeeaassuurreemmeenntt
– RReepprroodduucciibbiilliittyy oovveerr ttiimmee
– CCoonnssiisstteennccyy bbeettwweeeenn ddiiffffeerreenntt
ccooddeerrss//oobbsseerrvveerrss
– CCoonnssiisstteennccyy aammoonngg mmuullttiippllee iinnddiiccaattoorrss
• EEssttiimmaatteess ooff RReelliiaabbiilliittyy
– SSttaattiissttiiccaall ccooeeffffiicciieennttss tthhaatt tteellll uussee hhooww
ccoonnssiisstteennttllyy wwee mmeeaassuurreedd ssoommeetthhiinngg

MMeeaassuurreemmeenntt VVaalliiddiittyy
• AArree wwee rreeaallllyy mmeeaassuurriinngg ccoonncceepptt wwee ddeeffiinneedd??
– IIss iitt aa vvaalliidd wwaayy ttoo mmeeaassuurree tthhee ccoonncceepptt??
• MMaannyy ddiiffffeerreenntt aapppprrooaacchheess ttoo vvaalliiddaattiioonn
– JJuuddggmmeennttaall aass wweellll aass eemmppiirriiccaall aassppeeccttss

KKeeyy ttoo RReelliiaabbiilliittyy aanndd VVaalliiddiittyy
• CCoonncceepptt eexxpplliiccaattiioonn
– TThhoorroouugghh mmeeaanniinngg aannaallyyssiiss
• CCoonncceeppttuuaall ddeeffiinniittiioonn::
– DDeeffiinniinngg wwhhaatt aa ccoonncceepptt mmeeaannss
• OOppeerraattiioonnaall ddeeffiinniittiioonn::
– SSppeelllliinngg oouutt hhooww wwee aarree ggooiinngg ttoo mmeeaassuurree
ccoonncceepptt

FFoouurr AAssppeeccttss ooff RReelliiaabbiilliittyy::
• 11.. SSttaabbiilliittyy
• 22.. RReepprroodduucciibbiilliittyy
• 33.. HHoommooggeenneeiittyy
• 44.. AAccccuurraaccyy

11.. SSttaabbiilliittyy
• CCoonnssiisstteennccyy aaccrroossss ttiimmee
– rreeppeeaattiinngg aa mmeeaassuurree aatt aa llaatteerr ttiimmee ttoo
eexxaammiinnee tthhee ccoonnssiisstteennccyy
– CCoommppaarree ttiimmee 11 aanndd ttiimmee 22

22.. RReepprroodduucciibbiilliittyy
• CCoonnssiisstteennccyy bbeettwweeeenn oobbsseerrvveerrss
• EEqquuiivvaalleenntt aapppplliiccaattiioonn ooff mmeeaassuurriinngg
ddeevviiccee
– DDoo oobbsseerrvveerrss rreeaacchh tthhee ssaammee ccoonncclluussiioonn??
– IIff wwee ddoonn’’tt ggeett tthhee ssaammee rreessuullttss,, wwhhaatt aarree wwee
mmeeaassuurriinngg??
• LLaacckk ooff rreelliiaabbiilliittyy ccaann ccoommpprroommiissee vvaalliiddiittyy

33.. HHoommooggeenneeiittyy
• CCoonnssiisstteennccyy bbeettwweeeenn ddiiffffeerreenntt mmeeaassuurreess
ooff tthhee ssaammee ccoonncceepptt
– DDiiffffeerreenntt iitteemmss uusseedd ttoo ttaapp aa ggiivveenn ccoonncceepptt
sshhooww ssiimmiillaarr rreessuullttss –– eexx.. ooppeenn--eennddeedd aanndd
cclloosseedd--eennddeedd qquueessttiioonnss

44.. AAccccuurraaccyy
• LLaacckk ooff mmiissttaakkeess iinn mmeeaassuurreemmeenntt
• IInnccrreeaasseedd bbyy cclleeaarr,, ddeeffiinneedd pprroocceedduurreess
– RReedduuccee ccoommpplliiccaattiioonnss tthhaatt lleeaadd ttoo eerrrroorrss
• OObbsseerrvveerrss mmuusstt hhaavvee ssuuffffiicciieenntt::
– TTrraaiinniinngg
– MMoottiivvaattiioonn
– CCoonncceennttrraattiioonn

IInnccrreeaassiinngg RReelliiaabbiilliittyy
• GGeenneerraall::
– TTrraaiinniinngg ccooddeerrss//iinntteerrvviieewweerrss//llaabb ppeerrssoonnnneell
– MMoorree ccaarreeffuull ccoonncceepptt eexxpplliiccaattiioonn
((ddeeffiinniittiioonnss))
– SSppeecciiffiiccaattiioonn ooff pprroocceedduurreess//rruulleess
– RReedduuccee ssuubbjjeeccttiivviittyy ((rroooomm ffoorr
iinntteerrpprreettaattiioonn))
• SSuurrvveeyy mmeeaassuurreemmeenntt::
– IInnccrreeaassee tthhee nnuummbbeerr ooff iitteemmss iinn ssccaallee
– WWeeeeddiinngg oouutt bbaadd iitteemmss ffrroomm ““iitteemm ppooooll””
• CCoonntteenntt aannaallyyssiiss ccooddiinngg::
– IImmpprroovvee ddeeffiinniittiioonn ooff ccoonntteenntt ccaatteeggoorriieess
– EElliimmiinnaattee bbaadd Powerpoint ccooddeerrss
Templates

IInnddiiccaattoorrss ooff RReelliiaabbiilliittyy
• TTeesstt--rreetteesstt
– MMaakkee mmeeaassuurreemmeennttss mmoorree tthhaann oonnccee aanndd sseeee iiff tthheeyy
yyiieelldd tthhee ssaammee rreessuulltt
• SSpplliitt--hhaallff
– IIff yyoouu hhaavvee mmuullttiippllee mmeeaassuurreess ooff aa ccoonncceepptt,, sspplliitt
iitteemmss iinnttoo ttwwoo ssccaalleess,, wwhhiicchh sshhoouulldd tthheenn bbee ccoorrrreellaatteedd
• CCrroonnbbaacchh’’ss AAllpphhaa oorr MMeeaann IItteemm--ttoottaall CCoorrrreellaattiioonn

RReelliiaabbiilliittyy aanndd VVaalliiddiittyy
• RReelliiaabbiilliittyy iiss aa nneecceessssaarryy ccoonnddiittiioonn ffoorr
vvaalliiddiittyy
– IIff iitt iiss nnoott rreelliiaabbllee iitt ccaannnnoott bbee vvaalliidd
• RReelliiaabbiilliittyy iiss NNOOTT aa ssuuffffiicciieenntt ccoonnddiittiioonn ffoorr
vvaalliiddiittyy
– IIff iitt iiss rreelliiaabbllee iitt mmaayy nnoott nneecceessssaarriillyy bbee vvaalliidd
• EExxaammppllee::
– BBaatthhrroooomm ssccaallee,, oolldd sspprriinnggss

NNoott RReelliiaabbllee oorr VVaalliidd

RReelliiaabbllee bbuutt nnoott VVaalliidd

RReelliiaabbllee aanndd VVaalliidd

TTyyppeess ooff VVaalliiddiittyy
• 11.. FFaaccee vvaalliiddiittyy
• 22.. CCoonntteenntt vvaalliiddiittyy
• 33.. PPrraaggmmaattiicc ((ccrriitteerriioonn)) vvaalliiddiittyy
– AA.. CCoonnccuurrrreenntt vvaalliiddiittyy
– BB.. PPrreeddiiccttiivvee vvaalliiddiittyy
• 44.. CCoonnssttrruucctt vvaalliiddiittyy
– AA.. TTeessttiinngg ooff hhyyppootthheesseess
– BB.. CCoonnvveerrggeenntt vvaalliiddiittyy
– CC.. DDiissccrriimmiinnaanntt vvaalliiddiittyy

FFaaccee VVaalliiddiittyy
• SSuubbjjeeccttiivvee jjuuddggmmeenntt ooff eexxppeerrttss aabboouutt::
– ““wwhhaatt’’ss tthheerree””
– DDoo tthhee mmeeaassuurreess mmaakkee sseennssee??
• CCoommppaarree eeaacchh iitteemm ttoo ccoonncceeppttuuaall ddeeffiinniittiioonn
– DDoo iitt rreepprreesseenntt tthhee ccoonncceepptt iinn qquueessttiioonn??
• IIff nnoott,, iitt sshhoouulldd bbee ddrrooppppeedd
– IIss tthhee mmeeaassuurree vvaalliidd ““oonn iittss ffaaccee””

CCoonntteenntt VVaalliiddiittyy
• SSuubbjjeeccttiivvee jjuuddggmmeenntt ooff eexxppeerrttss aabboouutt::
– ““wwhhaatt iiss nnoott tthheerree””
• SSttaarrtt wwiitthh ccoonncceeppttuuaall ddeeffiinniittiioonn ooff eeaacchh
ddiimmeennssiioonn::
– IIss iitt rreepprreesseenntteedd bbyy iinnddiiccaattoorrss aatt tthhee
ooppeerraattiioonnaall lleevveell??
– AArree ssoommee oovveerr oorr uunnddeerrrreepprreesseenntteedd??
• IIff ccuurrrreenntt iinnddiiccaattoorrss aarree iinnssuuffffiicciieenntt::
– ddeevveelloopp aanndd aadddd mmoorree iinnddiiccaattoorrss
• EExxaammppllee----CCiivviicc PPaarrttiicciippaattiioonn qquueessttiioonnss::
– DDiidd yyoouu vvoottee iinn tthhee llaasstt eelleeccttiioonn??
– DDoo yyoouu bbeelloonngg ttoo aannyy cciivviicc ggrroouuppss??
– HHaavvee yyoouu eevveerr aatttteennddeedd aa cciittyy ccoouunncciill
mmeeeettiinngg??
– WWhhaatt aabboouutt ““pprrootteesstt ppaarrttiicciippaattiioonn”” oorr ““oonnlliinnee
oorrggaanniizziinngg””??

PPrraaggmmaattiicc VVaalliiddiittyy
• EEmmppiirriiccaall eevviiddeennccee uusseedd ttoo tteesstt vvaalliiddiittyy
– CCoommppaarree mmeeaassuurree ttoo ootthheerr iinnddiiccaattoorrss
• 11.. CCoonnccuurrrreenntt vvaalliiddiittyy
– DDooeess aa mmeeaassuurree pprreeddiicctt ssiimmuullttaanneeoouuss
ccrriitteerriioonn??
• VVaalliiddaattiinngg nneeww mmeeaassuurree bbyy ccoommppaarriinngg ttoo eexxiissttiinngg mmeeaassuurree
• EE..gg..,, DDooeess nneeww iinntteelllliiggeennccee tteesstt ccoorrrreellaattee wwiitthh eessttaabblliisshheedd
tteesstt
• 22.. PPrreeddiiccttiivvee vvaalliiddiittyy
– DDooeess aa mmeeaassuurree pprreeddiicctt ffuuttuurree ccrriitteerriioonn??
• EE..gg..,, SSAATT ssccoorreess:: DDoo tthheeyy pprreeddiicctt ccoolllleeggee GGPPAA??

CCoonnssttrruucctt VVaalliiddiittyy
• EEnnccoommppaasssseess ootthheerr eelleemmeennttss ooff vvaalliiddiittyy
• DDoo mmeeaassuurreemmeennttss::
– AA.. RReepprreesseenntt aallll ddiimmeennssiioonnss ooff tthhee ccoonncceepptt
– BB.. DDiissttiinngguuiisshh ccoonncceepptt ffrroomm ootthheerr ssiimmiillaarr
ccoonncceeppttss
• TTiieedd ttoo mmeeaanniinngg aannaallyyssiiss ooff tthhee ccoonncceepptt
– SSppeecciiffiieess tthhee ddiimmeennssiioonnss aanndd iinnddiiccaattoorrss ttoo bbee
tteesstteedd
• AAsssseessssiinngg ccoonnssttrruucctt vvaalliiddiittyy
– AA.. TTeessttiinngg hhyyppootthheesseess
– BB.. CCoonnvveerrggeenntt vvaalliiddiittyy
– CC.. DDiissccrriimmiinnaanntt vvaalliiddiittyy

AA.. TTeessttiinngg HHyyppootthheesseess
• WWhheenn mmeeaassuurreemmeennttss aarree ppuutt iinnttoo pprraaccttiiccee::
– AArree hhyyppootthheesseess tthhaatt aarree tthheeoorreettiiccaallllyy
ddeerriivveedd,, ssuuppppoorrtteedd bbyy oobbsseerrvvaattiioonnss??
• IIff nnoott,, tthheerree iiss aa pprroobblleemm wwiitthh::
– AA.. TThheeoorryy
– BB.. RReesseeaarrcchh ddeessiiggnn ((iinntteerrnnaall vvaalliiddiittyy))
– CC.. MMeeaassuurreemmeenntt ((ccoonnssttrruucctt vvaalliiddiittyy??))
• IInn sseeeekkiinngg ttoo eexxaammiinnee ccoonnssttrruucctt vvaalliiddiittyy::
– EExxaammiinnee tthheeoorreettiiccaall lliinnkkaaggeess ooff tthhee
ccoonncceepptt ttoo ootthheerrss
• MMuusstt iiddeennttiiffyy aanntteecceeddeenntt aanndd ccoonnsseeqquueenncceess
– WWhhaatt lleeaaddss ttoo tthhee ccoonncceepptt??
– WWhhaatt aarree tthhee eeffffeeccttss ooff tthhee ccoonncceepptt??

BB.. CCoonnvveerrggeenntt VVaalliiddiittyy
• MMeeaassuurriinngg aa ccoonncceepptt wwiitthh ddiiffffeerreenntt mmeetthhooddss
– IIff ddiiffffeerreenntt mmeetthhooddss yyiieelldd tthhee ssaammee rreessuullttss::
• tthhaann ccoonnvveerrggeenntt vvaalliiddiittyy iiss ssuuppppoorrtteedd
– EE..gg..,, SSuurrvveeyy iitteemmss mmeeaassuurriinngg PPaarrttiicciippaattiioonn::
• VVoottiinngg
• DDoonnaattiinngg ttoo mmoonneeyy ttoo ccaannddiiddaatteess
• SSiiggnniinngg ppeettiittiioonnss
• WWrriittiinngg lleetttteerrss ttoo tthhee eeddiittoorr
• CCiivviicc ggrroouupp mmeemmbbeerrsshhiippss
• VVoolluunntteeeerr aaccttiivviittiieess

C. Discriminant ((DDiivveerrggeenntt)) VVaalliiddiittyy
• MMeeaassuurriinngg aa ccoonncceepptt ttoo ddiissccrriimmiinnaattee tthhaatt
ccoonncceepptt ffrroomm ootthheerr cclloosseellyy rreellaatteedd ccoonncceeppttss
– EE..gg..,, MMeeaassuurriinngg MMaatteerrnnaalliissmm aanndd PPaatteerrnnaalliissmm
aass ddiissttiinncctt ccoonncceeppttss

DDiimmeennssiioonnss ooff VVaalliiddiittyy ffoorr
RReesseeaarrcchh DDeessiiggnn
• IInntteerrnnaall
– VVaalliiddiittyy ooff rreesseeaarrcchh ddeessiiggnn
• VVaalliiddiittyy ooff ssaammpplliinngg,, mmeeaassuurreemmeenntt,,
pprroocceedduurreess
• EExxtteerrnnaall
– GGiivveenn tthhee rreesseeaarrcchh ddeessiiggnn,, hhooww vvaalliidd aarree
• IInnffeerreenncceess mmaaddee ffrroomm tthhee ccoonncclluussiioonnss
• IImmpplliiccaattiioonnss ffoorr rreeaall wwoorrlldd

IInntteerrnnaall aanndd EExxtteerrnnaall VVaalliiddiittyy iinn
EExxppeerriimmeennttaall DDeessiiggnn
• IInntteerrnnaall vvaalliiddiittyy::
– DDiidd tthhee eexxppeerriimmeennttaall ttrreeaattmmeenntt mmaakkee aa
ddiiffffeerreennccee??
• OOrr iiss tthheerree aann iinntteerrnnaall ddeessiiggnn ffllaaww tthhaatt iinnvvaalliiddaatteess tthhee rreessuullttss??
• EExxtteerrnnaall vvaalliiddiittyy::
– AArree tthhee rreessuullttss ggeenneerraalliizzaabbllee??
• GGeenneerraalliizzaabbllee ttoo::
– WWhhaatt ppooppuullaattiioonnss??
– WWhhaatt ssiittuuaattiioonnss??
• WWiitthhoouutt iinntteerrnnaall vvaalliiddiittyy,, tthheerree iiss nnoo eexxtteerrnnaall vvaalliiddiittyy

RELIABILITY & VALIDITY
OF
INSTRUMENT -
QUALITATIVE
INSTRUMENT

Core Qualitative Methods
• Semi- or Un-structured, Open-Ended:
– In-depth Interviews
(in the field, face-to-face)
– Participant Observation
(field/site visits)
– Archival Research
(document review and analysis)

Guiding Principles
• Qualitative research designs consider
ways to foster:
– Reflexivity (an ongoing process of reflecting on
the researcher’s subjective experience, ways to
broaden and enhance this source of knowing, &
examining how it informs research)
– Iteration (a spiraling process: sequential and
repetitive steps in examining preliminary findings
for the purposes of guiding additional data
collection and analysis)
– Intersubjectivity (a process of reaching a
shared/ “objective” agreement about how to assign
meaning to a social experience - with insiders and
outsiders)

The Iterative Process of Qualitative
Research: A Model
DDaattaa CCoolllleeccttiioonn
RReefflleeccttiioonn
AAnnaallyyssiiss

Qualitative Research Techniques
• Instrumentation:
– Key Informants (question development and
piloting of instrument)
– Unstructured to Semi-structured
– Probing
– Data Processing and Analytic Tools

Qualitative Research Techniques/
Considerations
• Sampling
– Single v. Multiple Cases (not an individual)
– Expert and Key Informants (identification and
recruitment of sample)
– Roles of the Researcher (identification and
recruitment of sample)

• Data Collection
– Participants as Data Collectors
– Field Notes (personal reflections, observations,
emerging concepts/theories)
– Debriefing (a participant, a participating
researcher, a non-participating researcher)

• Analysis
– Key Informant Feedback
– Codebooks (specifies definitions and
relationships of concepts and terms)
– Memos (emerging patterns, concepts;
documentation of analytic pathways)
– Case Analysis Meeting (a meeting of a
research team for the purposes of reflecting on
analytic process, tools, and findings)
– Matrices or Diagrams (to identify and examine
time sequencing, the structure of relationships,
conditions of Powerpoint cross case Templates
events)

Three Qualitative Methods:
Ethnography
• Focus: study human behaviour in the
cultural context in which it is embedded
• Ethnography is the work of describing a
culture – the way of life of a cultural
group
– Associated with Cultural Anthropology
• Example: David Counts and Dorothy
Ayer Counts: “An Ethnography of RVing
Seniors”

Grounded Theory
• Focus: develop a theory to explain underlying
social processes of a cultural group
• Useful in areas where little is known or when a
new perspective is needed
– Used for exploratory, descriptive studies
• Because the theory emerges from the data, it
is said to be grounded in the data
– Foundation in Symbolic Interactionism
• Example: Kerry Daly and the social
construction of fatherhood

Phenomenology
• Focus: reveal the meaning of the lived experience from
the perspective of participants
• Describe the essences of lived experience
– Essences: elements related to the true meaning of
something that gives common understanding to the
phenomenon under study
– Conveyed with descriptive language
– Drawn from Philosophy; used across disciplines
• Example: J.E. Solchany: A phenomenological study of
women’s preadoptive experiences

Data Collection Methods
in Qualitative Studies
• Three data collection strategies
introduced:
1. Participant observation
2. In-depth interviews
3. Focus group interviews
• Qualitative researchers may combine
more than one method

Data Collection Methods:
1. Participant Observation
• Intensive, usually long term, examination
of a social group, an organization, etc.
• Researcher becomes a participant in the
lives of group members
– Observes their behaviour and learns meaning
systems (which are tied to language)
• Most closely associated with Ethnography,
as developed in Classical Anthropology
• Now done in a variety of disciplines

1. Participant Observation (cont’d)
• Many classic participant observations studies
– E.g., Asylums (Erving Goffman), Tally’s
Corner (Elliot Liebow), Street Corner
Society (William F. Whyte), etc.
• Today most ethnographers take an overt role
– I.e., their identity as a researcher is known
to the people being studied
• Covert participation (i.e., identity concealed
from participants) is fraught with ethical issues
– e.g., Humphrey’s “Tearoom Trade”

Steps Involved in
Participant Observation Research
A. Gaining entry into the group
B. Developing and maintaining rapport
C. Developing a method for taking field
notes
D. Integrating data collection and data
analysis

Steps in Participant Observation:
A. Gaining Entry into the Group
• Take into consideration the type of group
– formal organizations require formal entry;
involves letter writing, permission requests,
etc.
– Informal groups – different strategy needed
• Access may be gained through a gatekeeper
(an individual with special status)
• Want to involve key informants (those who are
most knowledgeable about the group)

B. Developing/Maintaining Rapport
• Researcher must work hard to develop
and maintain good relationships in the
field
– E.g., be sure not to become associated
with one faction in a group or organization
• Researcher could be blamed for
problems that arise in the setting

C. Strategies for Taking Field Notes
Field notes – integral to participant observation
• Include descriptions and interpretations of individuals,
interactions, and events
– Distinguish descriptions from interpretations
• Record time and location of observations, as well as
key information (weather, events happening and their
significance)
• Keep theoretical memos – which are the tentative
interpretations emerging and being assessed through
further data collection

Field Notes (cont’d)
• May not be possible or advisable to
take notes while in the field
– Important that they be done as soon after
field observation as possible
• Note-taking is time-consuming because
it is integral to guiding the data
collection and continuing the analysis
– E.g., field notes for When Prophecy Failed
were well over 1,000 typed pages

D. Integrating Data Collection and
• Organizing field noteAsn inatloy dsiifsferent types of files
facilitates data analysis
• Master field file – complete journal of field notes;
number pages and include entry dates
• Background, history file – subfile organizing
background material
• Key character files – subfiles on key players in
the group or organization
• Analytic files – subfiles for different types of
observations or relationships

2. In-depth Interviews
• Some studies cannot employ the participant
observation method
– E.g., Desroches’s study of bank robbers
• In-depth interviews allow participants to describe
their experiences and the meaning of events
taking place in their lives
– Verbatim quotes capture the language and
meaning expressed by participants
• Interviews are flexible and allow for probing
– Interview method is quite diverse, adaptive

2. In-depth Interviews (cont’d)
• Spradley – three key elements for the interview
method to be successful
1. Explicit purpose – researcher and informant are
aware that the discussion has a purpose
2. Ethnographic explanations – researcher tries
out explanations on the participants to see if
they make sense
– Encourage the informants to use colloquial
language, and teach the researcher its
meaning

2. In-depth Interviews (cont’d)
3. Ethnographic questions include:
i. Descriptive questions – ask participants to
describe their experiences (e.g., their ideas,
circumstances, viewpoints, dilemmas, etc)
ii. Structural questions – ask participants how they
organize their world (e.g., activities)
iii. Contrast questions – ask participants what is
meant by specific terminology
• Prus & Grills stress the value of the interview method
in a multi-method approach

3. Focus Group Interviews
• Interview format, but in a group setting
– 6-12 participants with common experience
• Dates back to the 1940s – used to assess
effectiveness of morale-boosting radio shows
– 1970s onward – used by market researchers
– 1980s onward – used by academics
• Transcript of discussion is the data
– Plus accompanying notes
– Use content analysis or grounded theory
approach to analyze the data

3. Focus Group Interviews (cont’d)
• Strengths:
– Open-ended question
• Spontaneously deal with issues as they arise
– Cost-effective method of collecting data
– Less time-consuming
• Weaknesses:
– One or two participants may dominate
– Not done in a natural setting, so little “observation” to
help understand the experience of the participants

Implications for Handling Threats
to Validity and Reliability
• In quantitative research, threats to
validity are addressed by prior design
features (such as randomization and
controls)

Implications for Handling Threats
to Validity and Reliability
• In qualitative research, such prior elimination of
threats to validity is less possible because:
– qualitative research is more inductive, and
– it focuses primarily on understanding particulars
rather than generalizing to universals.
• Qualitative researchers view “threats” as an
opportunity for learning
- e.g. researcher effects and bias are part of the story
that is told; they Powerpoint are not “Templates
controlled for”

Validity
• In general, validity concerns the degree
to which an account is accurate or
truthful
• In qualitative research, validity concerns
the degree to which a finding is judged to
have been interpreted in a correct way

Assessing the Validity of
Qualitative Research
Can another research read your field (and other types
of) notes (i.e., the explication of your logic) and come
to the same understandings of a given phenomenon?
• Concern about validity (as well as reliability) is the
primary reason thick description is an essential
component of the qualitative research enterprise
• Resources:
– Handout: Different Types of Notes
– Example: ACY Site Visit Toolkit

Major Types of Validity in
• Descriptive Validity
• Interpretive Validity
• Theoretical Validity
• External Validity (i.e., generalizability)

Descriptive Validity
• Concerned with the factual accuracy of an
account (that is, making sure one is not
making up or distorting the things one hears
and sees)
• All subsequent types of validity are
dependent on the existence of this
fundamental aspect of validity

Descriptive Validity
Behavior must be attended to, and with
some exactness, because it is through
the flow of behavior – or, more
precisely, social action – that cultural
forms find articulation.
Geertz 1973:17

Interpretive Validity
• Interpretive accounts are grounded in
the language of the people studied and
rely, as much as possible, on their own
words and concepts
• At issue, then, is the accuracy of the
concepts as applied to the perspective
of the individuals included in the
account

Interpretive Validity: Design
Consideration
• While the relevant consensus about the terms used in
description rests in the research community, the
relevant consensus for the terms used in
interpretation rests, to a substantial extent, in the
community studied
• An important design element, for increasing
interpretive validity, therefore, is to employee, at
some level/to some degree, a participatory research
approach (e.g., through member checks, peer to peer
research model, etc.)

Theoretical Validity
• Theoretical understanding goes beyond concrete
description and interpretation; it’s value is derived
based on its ability to explain succinctly the most
amount of data
• A theory articulates/formulates a model of
relationships as they are postulated to exist between
salient variables or concepts
• Theoretical validity is thus concerned, not only with
the validity of the concepts, but also their postulated
relationships to one another, and thus its “goodness
of fit” as an explanation

Major Threats to Validity
• Type I error: believing a principle to be
true when it is not (i.e., mistakenly
rejecting the null hypothesis)
• Type II error: rejecting a principle
when, in fact, it is true
• Type III error: asking the wrong
question

Triangulation: An Important
Theoretical Validity Check
• Case example: Parable of the blind men
and the elephant

Triangulation: An Important
Theoretical Validity Check
The most fertile search for validity comes from a
combined series of different measures, each with its
own idiosyncratic weaknesses, each pointed to a
single hypothesis. When a hypothesis can survive
the confrontation of a series of complementary
methods of testing, it contains a degree of validity
unattainable by one tested within the more
constricted framework of a single method.
Webb et al. 1966:174

External Validity in Qualitative
Research
• There is broad agreement that generalizability (in the
sense of producing laws that apply universally) is not
a useful standard or goal for qualitative research
• This is not to say, however, that studies conducted to
examine a particular phenomenon in a unique setting
cannot contribute to the development of a body of
knowledge accumulating about that particular
phenomenon of interest
• Consensus appears to be emerging that for
qualitative researchers generalizability is best thought
of as a matter of the “fit” between the situation
studied and others to which one might be interested
in applying the concepts and conclusions of that
study.

Enhancing External Validity
• Thick descriptions are crucial.
• Such descriptions of both the site in which the studies
are conducted and of the site to which one wishes to
generalize (or apply one’s findings) are critical in
allowing one to search for the similarities and
differences between the situations.
• Analysis of these similarities and differences makes it
possible to make a reasoned judgment about the
extent to which we can use the findings from one
study as a working hypothesis about what might
occur in another situation.

Multi-site Studies: Another Way to
Enhance Generalizability
• A finding emerging repeatedly in the study of
numerous sites would appear to be more likely to be
a good working hypothesis about some as yet
unstudied site than a finding emerging from just one
or two sites.
• A finding emerging from the study of several very
heterogeneous sites would be more robust and, thus,
more likely to be useful in understanding various
other sites than one emerging from the study of
several very similar sites.
• Heterogeneity may be obtained by creating a
sampling frame that maximizes the variation inherent
in the sample, specifically in terms of potentially
theoretically important Powerpoint dimensions
Templates

Reliability
• Reliability concerns the ability of different
researchers to make the same
observations of a given phenomenon if
and when the observation is conducted
using the same method(s) and
procedure(s)

Enhancing the Reliability of
• Researchers can enhance the reliability of their
qualitative research by:
– Standardizing data collection techniques and
protocols
– Again, documenting, documenting, documenting
(e.g., time day and place observations made)
– Inter-rater reliability (a consideration during the
analysis phase of the research process)

Kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen

Similar a Kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen (20)

Último

Último (20)

Kesahan & kebolehpercayaan pembentukan instrumen, kesahan dan kebolehpercayaan instrumen

Notas del editor