Presentation for the HEA-funded workshop ‘Teaching Research Methods in Business and Management’. Drawing on a mixture of practice and evidence, this one-day event provided an opportunity for those interested in the teaching of research methods in Business and Management – including qualitative, quantitative and mixed methods – to share experiences, insights, and good practice, and to discuss challenges and explore potential solutions. This presentation forms part of a blog post reporting on the event which can be accessed via: http://bit.ly/1fcTwna For further details of HEA Social Sciences work relating to teaching research methods in the Social Sciences please see http://bit.ly/15go0mh

1. Quality issues in mixed methods research
(with an emphasis on teaching) Alan Bryman
Talk to Higher Education Academy event – Teaching Research Methods in Business and
Management, 10 January 2014
©Alan Bryman

2. Is quality appraisal important for students?

Growing focus on quality criteria





Important for students to be familiar with aspects of quality
that are especially valued





Need for consistent standards in assessments of research
Interest among qualitative researchers has increased
Use of systematic reviews
Relevance for mixed methods research?
May increase skill levels
Provides vantage point for critical assessment of research
May improve literature reviews for dissertations and theses
In the case of mixed methods research, important to realise that it is
more that just 2 separate components
But might creativity and innovation be stifled?

3. Creswell’s skill-based mixed methods
seminar







Learning how to identify a mixed methods study
Learning appropriate search terms for locating mixed
methods studies
Writing a rationale for mixing quantitative and qualitative
research
Writing research questions for mixed methods study
Summarizing types of data collected in mixed methods
research
Transforming quantitative data into qualitative data and
vice versa
Combining quantitative and qualitative data to arrive at
conclusions

4. Creswell’s skill-based mixed methods
seminar (contd.)






Identifying procedures used in software for analysis of
quantitative and qualitative data
Drawing visual model of mixed methods research design
Identifying distinguishing features of mixed methods
designs
Comparing strengths and weaknesses of different mixed
methods designs
Evaluating a mixed methods study in terms of
criteria
Applying steps in design of a mixed methods study to a
research proposal
J. Creswell et al. (2003) ‘Teaching mixed methods research: practices, dilemmas and challenges’, in A. Tashakkori
and C. Teddlie (eds), Handbook of Mixed Methods in Social & Behavioral Research, Thousand Oaks, CA: Sage.

5. Do we expect too much of quality criteria?
Two prominent roles:
1. Appraisal role.





Consideration of adequacy of research from the point of view
of assessing research quality (e.g. for inclusion/exclusion in a
systematic review) or as part of the refereeing function for
research proposals and journal articles.
With qualitative research, has led to long lists partly associated
with lack of agreement on criteria (cf. Cassell & Symon, 2011)
Difficult to argue for the removal of criteria.
In the case of qualitative research, has led to long lists…
C. Cassell & G. Symon (2011) ‘Assessing “good” qualitative research in the work psychology field: a narrative
analysis’, Journal of Occupational and Organizational Psychology, 84: 633-50.

6. Encyclopaedic approach to quality criteria for qualitative research?
1.
How credible are the findings?
2.
Has knowledge/understanding been extended by the research?
3.
How well does the evaluation address its original aims and purposes?
4.
Scope for drawing wider influences—how well is this explained?
5.
How clear is the basis of the evaluative appraisal?
6.
How defensible is the research design?
7.
How well defended is the sample design/target selection of cases/documents?
8.
Sample composition/case inclusion—how well is the eventual coverage described?
9.
How well was the data collection carried out?
10.
How well has the approach to, and formulation of, the analysis been conveyed?
11.
Contexts of data sources—how well are they retained and portrayed?
12.
How well has diversity of perspective and content been explored?
13.
How well has detail, depth and complexity (richness?) of the data been conveyed?
14.
How clear are the links between data, interpretation and conclusions—i.e. how well can the route to any
conclusions be seen?
15.
How clear and coherent is the reporting?
16.
How clear are the assumptions/theoretical perspectives/values that have shaped the form and output of the
evaluation?
17.
What evidence is there of attention to ethical issues?
18.
How adequately has the research process been documented?
Spencer, L., et al. (2003), Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence www.civilservice.gov.uk/wpcontent/uploads/2011/09/a_quality_framework_tcm6-38740.pdf

7. Tracy’s 8 ‘must have’ criteria
1.
Worthy topic—relevant, interesting, significant, etc.
2.
Rich rigour—rich data supplied in abundance and appropriately
3.
Sincerity—the researcher is reflexive about values and biases and is transparent in
approach
4.
Credibility—implements practices such as thick descriptions, triangulation, and
respondent validation
5.
Resonance—has an affecting impact on readers
6.
Significant contribution—makes an impact in terms of such outcomes as theory,
practice, and morality
7.
Ethical—considers and engages in ethical practices
8.
Meaningful coherence—addresses what it claims to address, uses appropriate
methods, and links research questions, literature, findings and interpretations.
S. J. Tracy (2010). ‘Qualitative Quality: Eight “Big Tent” Criteria for Excellent Qualitative Research’, Qualitative
Inquiry, 16: 837–51.

8. Do we expect too much of quality criteria?
Teaching role
2.


Cohorts of students introduced to quantitative research
quality criteria (internal, external, construct, conclusion, and
ecological validity). Advantage of small number of issues around
which criteria are grouped.
From a teaching point of view, long lists may be unhelpful.




Lack of differentiation and therefore guidance concerning
importance/significance.
Potentially off-putting.
Unrealisable.
Therefore, for teaching purposes, shorter lists may be more
helpful to students (and maybe lecturers).

9. Trends in discussions of mixed methods
criteria

Discussions of mixed methods research criteria may be
treading a similar path to discussions of qualitative
research criteria

10. Mixed methods criteria
The comprehensive approach (O’Cathain, 2010) – 6 domains + 2
Domain 1: Planning quality – 4
Domain 2: Design quality – 4
Domain 3: Data quality – 5
Domain 4: Interpretive rigour – 8
Domain 5: Inference transferability – 4
Domain 6: Reporting quality – 3
28 + 2 = 30 quality criteria
A. O’Cathain (2010). ‘Assessing the quality of mixed methods research: toward a comprehensive framework’. In
A. Tashakkori and C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (2nd
edition, pp. 531-55). Los Angeles: Sage.

11. Some recurring themes: Alan’s ‘must haves’






Need for quantitative and qualitative components to be
appropriately implemented
Need for transparency
Use of mixed methods to be linked to research questions
Need to be explicit about the mixed methods design and its
appropriateness to research questions
Importance of a rationale for the use of mixed methods
research
Importance of integration
To appear in a forthcoming article which has been accepted for publication

12. Some recurring themes: Alan’s ‘must haves’






Need for quantitative and qualitative components to be
appropriately implemented
Need for transparency
Use of mixed methods to be linked to research questions
Need to be explicit about the mixed methods design and its
appropriateness to research questions
Importance of a rationale for the use of mixed methods
research
Importance of integration
To appear in a forthcoming article which has been accepted for publication

13. Recent review of health services research

‘Key methodological components that cut across
qualitative and quantitative methodologies were often
missing from mixed methods publications. Descriptions of
sample selection and sampling procedures, the study
context, and data-gathering procedures are essential
aspects of interpreting study findings, and mixed methods
studies should not be exempt from these basic research
requirements. Many mixed methods studies did not
include the level of detail that would likely be required for
a qualitative or quantitative paper to be accepted in these
high-ranking journals’.
J.P. Wisdon et al. (2012) ‘Methodological reporting in qualitative, quantitative and mixed methods health
services research articles’, Health Services Research, 47(2): 721-45.

14. Some recurring themes: Alan’s ‘must haves’






Need for quantitative and qualitative components to be
appropriately implemented
Need for transparency
Use of mixed methods to be linked to research questions
Need to be explicit about the mixed methods design and its
appropriateness to research questions
Importance of a rationale for the use of mixed methods
research
Importance of integration
To appear in a forthcoming article which has been accepted for publication

15. Two common approaches to mixed methods
research questions
1.
2.
Separate quantitative and qualitative research questions.
Overarching mixed research question; then expanded
or broken down into quantitative and qualitative subquestions.
From: Teddlie, C. and Tashakkori, A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative
Approaches in the Social and Behavioral Sciences, Los Angeles: Sage, 2009, p. 133.

16. Some recurring themes: Alan’s ‘must haves’






Need for quantitative and qualitative components to be
appropriately implemented
Need for transparency
Use of mixed methods to be linked to research questions
Need to be explicit about the mixed methods design and its
appropriateness to research questions
Importance of a rationale for the use of mixed methods
research
Importance of integration
To appear in a forthcoming article which has been accepted for publication

17. Types of mixed methods research design

18. Some recurring themes: Alan’s ‘must haves’






Need for quantitative and qualitative components to be
appropriately implemented
Need for transparency
Use of mixed methods to be linked to research questions
Need to be explicit about the mixed methods design and its
appropriateness to research questions
Importance of a rationale for the use of mixed methods
research
Importance of integration
To appear in a forthcoming article which has been accepted for publication

19. Structure of the article: Parmelee et al.




Introduction
Young people, voting, cynicism, and advertising
Theoretical and mixed methods perspectives
Method



Results





Focus groups
Content analysis
Short introduction
Media-based frames and the failure to engage college-age voters
Negative ads, audience-based frames, and cynical voters
Students’ recommendations for how to build more engaging political ads
Discussion


Summary and implications
Limitations and future research
J.H. Parmelee, et al. (2007) ‘ “What about people our age?”: Applying qualitative and quantitative methods to
uncover how political ads alienate college students’, Journal of Mixed methods research, 1(2): 183-99

20. May be more useful for research quality
appraisal too


Lessons from qualitative research
Pratt (2008) – reviewers of qualitative management research articles
for North American journals believe qualitative standards are:





difficult to understand
lack consensus/consistent standards
lack clarity
also, articles often evaluated against inappropriate (usually quantitative
research) criteria
Savall et al. (2008) – reviews of qualitative management research
articles for a European journal:


Quality criteria proposed by literature not identified in a content analysis
of reviews (9 criteria)
Quality criteria used by reviewers but rarely mentioned in the literature
(6 criteria)
M.G. Pratt (2008) ‘Fitting oval pegs into round holes: tensions in evaluating and publishing qualitative research in top-tier North
American journals’, Organizational Research Methods, 11: 481-509.
H. Savall. et al. (2008) ‘The emergence of implicit criteria actually used by reviewers of qualitative research articles: case of a
European journal’, Organizational Research Methods, 11: 510-40.

21. May be more useful for research quality
appraisal too



Focusing on a core set of criteria rather than a
proliferation may mitigate some of these practices.
For mixed methods research, concentrating on small
number of core issues may help to focus attention on
quality at a relatively early stage of its evolution.
Importance of examples of mixed methods research

22. Key points

Need to distinguish between



quality criteria as a screening or evaluation tool; and
the possible pedagogical role of quality criteria
In the context of mixed methods research, I’ve tried to
suggest that a small number of criteria that capture
widely-held positions may be more helpful in a teaching
context.