Qualitative vs Quantitative research methods - the right tools for the right job. New Media Research Methods.

1. New Media Research Methods
Part 1 – How research
methods relate to the
research question
Part 2- Qualitative and
Quantitative
Part 3 – Data collection
Part 4 – Presentation and
analysis

2. New Media Research Methods- Part 2
Focus on the qualitative and quantitative methods:
The right tools for the right job.
Gosia Kwiatkowska gosia@uel.ac.uk

3. Recap from last week

4. New Media Research process
Analysis
Design
Implementation

5. Qualitative Quantitative
• Qualitative - deep
understanding of the
experience. Not
everything can be
reduced to numbers.
• E.G.
Observation, ethnogr
aphy
• Quantitative –
objective,
measurable. Helps to
establish patterns
and relationships.
• E.G. Survey

6. Which one to use?
http://www.youtube.com/watch?v=ddx9PshVWXI

7. Qualitative Method?
– User perspectives
– Questions: how and why?
– Captures perceptions, judgements, meanings,
processes and reasons
– Open ended questions, checklist of topics
– Hypothesis and follow up questions are generated
during data collection and analysis
– Subjective – not easy to objectively verify
– Interviews allow to probe
– Provides deeper insight
– Broader understanding and explanations, views

8. Qualitative Method?
• Strengths
– Participatory
– Rich, detailed data
– Considers users perspectives and the context for their
behaviours
• Weaknesses
– Hard work
– Time consuming
– Smaller sample of users
– Not easily verifiable
– Not easy to group your responses/categorised

9. Quantitative Methods
– Designed to ensure objectivity, reliability and ability to
generalise
– Test predetermined hypothesis – explanations
– Who, how much, how many?
– Closed questions
– Short answers
– Averages, percentages, ranges, means, frequencies
– Can generate accurate and precise data
– Can test statistical relationship between variable
– Can prove whether or not a particular problem exists
– Can identify specific characteristics of a population

10. Quantitative Methods
• Strengths
– Robust
– Objective
– Verifiable
• Weaknesses
– Out of context – human behaviour, real world
settings are not considered
– Any variables left out of data collection are not
used in analysis

11. Quantitative or Qualitative or Both?
• Quantitative methods can highlight an issues
that could then be studied in depth using
qualitative methods.
• Qualitative methods might be used at the
beginning of a study to help the researcher to
decide what closed questions could be used in
the bigger quantitative survey or
questionnaire.

12. New Media Research
Interviews
• Purpose
• Characteristics
• Advantages
• Limitations
• Process/Stages/Questions

13. Definitions / purposes
“Qualitative research [such as interviewing]
attempts to understand meanings that
people give to their deeds, experiences, or to
other social phenomena”
Silverman D (1997)
‘The purpose of interviewing is to find out what
is in and on someone else’s mind. We find
out from them those things which we cannot
directly observe’
May T (1997)

14. Advantages / limitations
• Your views …

15. Advantages
• Rich data - excellent for complex subjects
• Meanings / understandings / perceptions explored
• Powerful when territory unfamiliar
• Unforeseen issues / experiences elicited
• Clarification / Follow up Q&As possible
• Captive subjects
• Certainty over who is responding (unlike surveys)
• Non-verbal communication / Observational opportunities

16. Disadvantages (1)
Subjects may:
• Conform to expectations – (social desirability)
• try to be rational
Interviewer may:
• Be inconsistent
• (Unwittingly) bias respondents’ answers
• Mis-interpret answers

17. Disadvantages (2)
• Difficult to get quantitative data
• Data analysis difficult / complicated
• Unrepresentative - reliance on key figures
• Time consuming / costly
• Stressful for interviewer (& interviewee?)
• Low validity / reliability

18. Reliability and validity
• Validity: "By validity, I mean truth: interpreted as the
extent to which an account accurately represents the
social phenomena to which it refers. " (Hammersley,
1992). Pg. 57. (e.g. does the data-gathering measure
what you want it to measure?)
• Reliability: the degree of consistency with which
instances are assigned to the same category by
different observers or by the same observer on
different occasions". (Hammersley, 1990). Pg. 67 (e.g.
does the data-gathering produce the same results if
repeated?) .

19. Validating interviews
• Triangulation
• Purposive sampling - focus on specific population
• Choose deviant case
• Member check (refer findings back to subjects)
• Co-researcher re-coding
• Researcher ‘reflexivity’
• ‘Fair dealing’ – ensure all viewpoints mentioned

20. But don’t forget …
One case may be enough(!):
‘What is wrong with samples of one? Why should
researchers have to apologise for them? Should
Piaget apologise for studying his own children, a
physicist for splitting only one atom?’
Mintzberg H (1973) The nature of managerial work NY:
Harper & Row

21. Preparing the interview (1)
Getting ethical approval:
“The (UH) Ethics Committee will … carefully evaluate the
following aspects of your application:
• the validity of the research
• the welfare of the research subjects
• the dignity of the research subjects
• the ability of the researcher/team/supervisor to conduct the
research”
[see: http://perseus.herts.ac.uk/uhinfo/library/j34130_13.doc]

22. Preparing the interview (2)
• Aims/objectives of the interview
• Where these fit with overall research study
• Interview style:
– Structured unstructured
– Formal informal
– Open closed
• Choose physical setting (if possible)
• Atmosphere / environment

23. Preparing the interview (3)
Individual questions/areas/themes
Fact
• objective information (e.g., age, gender, education, behaviour,
experience)
Opinion / Preference / attitude / feelings
• evaluative (e.g., satisfaction, agreement, likedislike)
Intended Behaviour
• motivation or intention (e.g., likeliness, willingness)

24. Recording the interview
To tape or not to tape?? Consider:
• Effect on interviewee
• Listening / transcribing time
• Reliability of machine / recording
But also…
• Scribbling whilst listening
• Making sense of notes

25. Conducting the interview
• Thank interviewee
• Set the scene (why / how / where of study)
• Give idea of question areas
• Give ground rules (can refuse to answer/can
terminate interview, anonymity etc.)
• Start with demographic and ID questions
• (usually) start with general question and then
funnel

26. Types of question (1)
Open
• Opening stages in line of questioning (funnel)
• Invites opinion, general knowledge.
• Can cover areas where interviewer’s own knowledge
lacking
• No presumption about response
Closed
• Elicit hard facts
• Control pace/direction of interview

27. Types of question (2)
Probing
• Extracts more depth
• Maintains line of enquiry
Leading
• Confirm interviewee’s answer
• Help interviewee, by rephrasing answer
• Bring a line of questioning to an end (summarising)

28. New Media Research
Questionnaires/Surveys
• Purpose
• Characteristics
• Advantages
• Limitations
• Process/Stages/Questions

29. Purpose of questionnaires
To survey large number of people, to
describe/explain characteristics/opinion of a
population, usually through a representative
sample.
Questionnaires measure generalities / the
extent to which groups of people behave or
think in certain ways.

30. Characteristics of questionnaires
• Cross-sectional
• Mainly quantitative
• ‘Snapshot’ in time
• Qs and As standardised

31. Advantages / limitations
Your views …

32. Advantages
• Reaches large / dispersed populations
• Can generalise results
• Provides quantitative, authoritative (?) data
• Appears easy– work done by respondent
• Relatively cheap and quick (per unit)
• Removes personal influence
• Respondent works in own time
• Provides structure for report
• Replicable

33. Limitations (1):
Questionnaire construction
• Low response rate (5-25%?)
• Respondents may differ from non-respondents
• No way to adapt add/remove questions
• Little opportunity for respondents to explain
• Can over-simplify issues
• Respondents ‘shoehorned’

34. Limitations (2):
Respondent issues
Respondents may:
• Take middle position in polar responses
• Lack recall / rely on recent experience
• Want to please researcher by being:
– socially responsible
– compliant

35. Process/stages
• Formulate study aims
• Identify objectives that address aims
• Decide what information is required, and from whom
• Decide sampling frame (elements making up population)
• Research for similar questionnaires
• Formulate appropriate questions
• Decide distribution method
– Postal; Telephone; Clip board/street; Web/Internet; Email;
f2f
• Pilot (see next slide) and get feedback
• Amend
• Distribute / administer

36. The pilot
• Use more “open ended” questions than in final
product (often interviews are undertaken first)
• For “Pilot” responses look for:
– Variation in type of answer
– Redundancy – areas of no/limited response
– Evidence of ambiguity etc.
– Acquiesce
• Amend questionnaire as appropriate

37. Question areas
Fact
• objective information (e.g., age, gender, education,
behaviour)
Opinion / Preference / attitude
• evaluative (e.g., satisfaction, agreement, likedislike)
Intended Behaviour
• motivation or intention (e.g., likeliness, willingness)

38. Question types (1)
• Open questions
– For further information (‘tell us more ‘)
– For new information (‘what are your view on …’)
• Multiple choice checklist
– One answer only
– All that apply (inc. Guttman scale)
• Binary (yes/no)
– Good for filtering
• Rank order

39. Question types (2)
• Likert scale
– Agreement (strongly agree, agree, …)
– Frequency (always, frequently, …)
– Importance (very important, important …)
– Quality (excellent, good, average …)
– Likelihood (definitely, probably …)
• Semantic differential
Reactions to stimulus words / concepts in terms of ratings on
‘bipolar’ scale with contrasting adjectives at each end:
– Excellent ………………terrible
– Helpful …………………unhelpful

40. Closed v open questions (1)
closed …
Advantages
• Quick & easy for respondents
• Less articulate for disadvantaged
• Fewer irrelevant answers
• Easy to code and analyse
Disadvantages
• Responses suggest ideas (e.g., shoehorning)
• Frustrates respondents if categories not exhaustive
• Misinterpretation goes unnoticed
• Complex issues forced into simple categories

41. Closed v open questions (2)
Open …
Advantages
• Permits detail, clarification
• Unanticipated answers
• Reveals the logic behind response
Disadvantages
• Generalisation or comparison difficult
• Coding and statistical analysis difficult
• Irrelevant answers possible
• Bias towards educated
• Time consuming for respondents & researcher

42. Sampling

43. Good practice
• Explain purpose
• Catch interest early
• Question sequence logical / helpful - broad to
specific
• Closed questions need "complete" set of response
alternatives
• Use appropriate language
• Include clear instructions
• Use clear tick boxes
• Keep short

44. Increasing response rate
• Name recipient if possible
• SAE
• Prize (?)
• Offer copy of the report
• Good design/construction
• Engender involvement/interest
• Chase non-respondents
• Target ignored/committed groups

45. Issues to consider
Question wording
Question order
Question types

46. Question wording
What’s wrong with these questions:
Do you like using the Internet and playing online
games?
Do you agree with most people that online
gaming is becoming more popular?

47. Question order
What’s wrong with these questions:
1) Do you have any children?
2) How old are you?
3) Are you married?
4) How long have you been studying at the
University?

48. Question types (1)
Leading questions:
• Why do you think the PC’s are worse than Apple computers?
Social pressure:
• Would you prefer to have more money spent on improving the
education or more tax cuts?
Presuming questions
• How often do you search the Internet?

49. Question types (2)
Ambiguous/imprecise questions
• How much time to you spend playing online games?
Double-barreled questions
• How long have you been using computers and the Internet?
Too much knowledge dependent questions
• Do you agree with the … theory?

50. Question types (3)
Memory dependent
• How many times have you played a computer game in the last
month?
Wish List/hypothetical questions
• Would you like better computer games to be designed?

51. Match tools to your questions
• For questions on factual
clarification
• Use polls and surveys
• For questions on opinion
• Use surveys, interviews, focus
groups
• For questions on experience
• Interviews, focus groups,
observations, user testing, and
ethnographies
• For questions on concepts
• surveys, interviews,
ethnographies, and user testing
• For questions on emotions
• Surveys, interviews, focus groups,
and observations

52. You can mix tools
• You need to map out each step you will take in
your research so that things follow in a logical
order…
• This is your method!

53. Spend the rest of the time thinking through
your proposed question
• Independent Variable
– What is tested, measured or manipulated
• Dependent Variable
– What is observed – the outcome

54. References
• Curasi CF (2001) A critical exploration of face-to-face interviewing vs.
computer-mediated interviewing International Journal of Market
Research 43(4)pp361-375
• Hammersley, M. (1992). What’s wrong with Ethnography:
Methodological Exploration, London: Routledge.
• May T (1997) Social research: issues, methods, progress Buckingham:
OUP
• Rugg, D. (1941) Experiments in wording question. II Public Opinion
Quarterly, 5:91-92.
• Silverman D (1997) Qualitative research: theory, method,
practiceLondon: Sage
• Silverman D (2004) Doing Qualitative Research: A Practical Handbook
London: Sage