This presentation was developed for participants on a PgDip programme - the remit was to offer insight into focus groups as a potential method of data collection for their research project.

1. Using focus groups for evaluating
learner experiences
Tünde Varga-Atkins - eLearning Unit, CLL
Luciane Vieira de Mello Rigden – School of Life Sciences
University of Liverpool
6th June 2013
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

2. Outline
1. Why focus group?
2. What is a focus group?
3. Focus groups in context of other methods
4. Viewpoints: focus groups as a
– Commissioner (stakeholder)
– Observer
– Facilitators
5. Focus group analysis and reporting
6. Alternatives / enhancements:
– Nominal Group Technique
– Visual and any other methods
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

3. Your experience
• Have you had experiences of focus group,
either as:
– Commissioner?
– Facilitator?
– Observer?
– Participant?
What is your research question?
Appropriate/Not to explore in a
focus group?
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

4. 1. Why focus groups?
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

5. Rationale for focus groups
“When people gather to talk about
something, their contributions and
understandings will be enriched by the group
dynamic.”
(Cousins 2009)
“To display and discuss differences in a group. ...”
(Lewis 2003)
Focus groups are for sharing and comparing
… and remembering
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

6. 2. What is a focus group?
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

7. Focus group process
Neutral facilitator
Size: 6-8 (4-12)
participants
Incentive
Sample selection!
Ethics: informed consent and reporting
Audio recorder & other resources
or stimulus material
1-2 hours
an open group discussion with a focus
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

8. What is a focus group?
Study
skills
International student experienceFocus:
Question(s):
English
language Peer
support
Can you tell me about the support that you have received on campus?
A focused but open discussion with a group of 6-8 participants, moderated by a
neutral facilitator usually lasting 1-2 hours. The facilitator usually has a topic (or
question) guide which may or may not be covered in sequence.
Introduction Closure
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

9. The evaluation/research cycle*
1. Set out
purpose
2. Agree what
questions
to ask
3. Agree on
method
(focus group)
5. Compiling
draft report
6. Draft report
sent to students
for confirmation
10. Action &
feedback to
students
7. Production of
final report
4. Conduct of
focus groups
8. Circulation of
report to staff
9. Presentation
and discussion
of report
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.
*of not just focus groups
but any participant
consultation!

10. 3. Focus groups
in context of other methods
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

11. Some possible combinations
Exploring depth
survey
survey
focus group(s)
focus group(s)
Generate ideas
focus group(s)
a
b
c
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

12. Methods: from individual to group
Surveys
Focus groups
Interviews
Researcher present
individual
group
Researcher
not present
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

13. Methods: from individual to group
Surveys
Focus groups
Interviews
Researcher present
individual
group
Researcher
not present
How many hours do you
spend studying?*
What makes you study (more)?
Can you tell me a bit more about what you do
when you study?
Idea
development
aided by
others?
Participant
motivation
Bias by
others?
Shared
experience?
* BUT: in a focus group setting, participants can have a discussion about what constitutes study, which may make them
adjust their own understanding of what study is.Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

14. Surveys Focus groups Interviews
Bias? Useful when you
don’t want others to
bias your participant
Potential of ‘group
bias’ – participants say
what they think you or
other participants
want to hear
Useful when you don’t
want others to bias
your participant
Shared experience? Participant relies on
own experience –
may be more difficult
to recall.
Useful to talk about
shared experiences.
‘You bounce off ideas
from one another’.
Useful when you
want to explore
individual
experiences.
Idea development
aided by others?
Relies on participant
able to formulate
their ideas without
prompts.
FGs are useful when
the development of
ideas and
understanding is aided
by other group
members.
FGs are useful when
the development of
ideas and
understanding is aided
by other group
members.
Motivation/Enjoyment ?
Non-response and
quality of response
as a problem.
Participants generally
enjoy the process.*
They learn something
new.
Participants generally
enjoy the process.
They learn something
new.
BUT: feasibility and practicality may be the largest factors
(of getting volunteers, organising them together etc.)
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

15. Sample questions from a UoL
curriculum evaluation focus group
• 1.What aspects of the X best facilitated your
learning?
• 2. What aspects of the X inhibited your learning ?
• 3. What aspects of the X caused the most
stress/anxiety and did this stress/anxiety worsen
or lessen as the X progressed?
• 4. Optional specific questions.
• 5. Can you suggest one (two) different ways in
which the X could be improved?
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

16. 4. Viewpoints
Focus groups as a:
commissioner
observer
facilitator
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

17. As a commissioner of a focus group
• Experiences of Dr Mello (Lu):
– Asking the right questions
• After a questionnaire: realisation of non-properly covered aspects
guided question formulation
• Questions were discussed with the facilitators (but I would do it more
extensively next time….)
– Coverage of questions
• Some really short answers from students
• Group dynamic
– What is important?
• Have clear what you want
• Take the first one as an experience
– Expectations
• If you have an open mind, you will surprise yourself!

18. As an observer of a focus group
• Experiences of Dr Mello (Lu):
– What is going on?
• Got to know the questions just before meeting the students
– Benefits
• Better appreciation & understanding of the whole process
• Confidence to plan another focus group
• How I learned from the students!!!
– Challenges
• Hard to just observe!
• Have a good chat with the facilitator in advance

19. As a facilitator of a focus group
• Design
• Introduction, scene setting
• Questioning techniques
• Managing group dynamics
• Managing schedule / topics / coverage

20. Slides supporting focus group
process (facilitation)

21. The schedule of a Focus Group
1. Introduction = setting scene and establishing trust:
– purpose
– ground rules (conformity / convergence)
– ethics
– reporting
1. Introduction
2. The ‘Focus’ = questions
3. Closure
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

22. 2. The ‘Focus’ = aka questions
Low vs high moderation
2 questions 6 questions: exploring international student
experience on campus
6 questions
(Cousins 2009, p63)
Note: Participants do not normally see the questions – the discussion is good if it seems to flow naturally.
These will not
necessarily be
covered in
order.

23. Anatomy of a good focus group question
with ready probes
(Cousins 2009, p63)
Preparation and good question (and prompts, probes) design is very important!
English
language
Peer
support
Probing for
range
Probing for
depth
Study
skills
Can you say
something
about further
support..?
Open question

24. 5. Analysis and reporting
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

25. Analysis
Transcript
Report
Analytical process
Various decisions and challenges!
• Emphasis – key points
• Focus
• Agreement
• Majority vs single voices
Research question
dependent

26. 6. Alternative methods
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

27. It doesn’t have to be just words!
Tasks e.g. projection techniques
Source: http://bit.ly/JZRz1T
Visual techniquesStimulus
Source http://bit.ly/Mc0kSu
Bringing as many stimulus
material related to the
topic as possible to help
participants recall e.g. bring
assignment samples for a discussion
on feedback.
Asking participants to take
photos, draw diagrams or
drawings to elicit their
experiences e.g. ask them to
draw a timeline of their time at
university
Asking participants to
undertake an activity, e.g.
using projective techniques
or write down their top
five priority in the given
topic etc.
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

28. The Nominal Group Technique
• Good decision-making technique with no need for
transcription!
• Brings in a quantitative element through the ranking
stages.
• More useful for evaluation and less useful for ‘full’
research.
– See references for more details.
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

29. The Nominal Group Technique:
an alternative focus group method
Surveys
Focus groups
Interviews
individual
group
Nominal Group Technique
The Nominal Group Technique is a useful technique that is structured in a way that
although it is a group session, it does focus on individual experiences and opinions rather
than group consensus. Participants at various points in the session are asked to give their
individual opinions which are ranked at the end.

30. Summary
• Focus groups are useful when:
– exploring a shared experience.
– participants (and the research!) benefits from the development of
understanding and ideas in a group environment.
– group bias or conformity is not likely/important.
• Focus groups:
– need to be facilitated by a neutral facilitator.
– produce quality of the output (if focused and run well).
– can be enriched by various techniques (visuals, stimulus) or tasks.
• Analysis involves various decisions on what/how to report.
• A useful alternative is the Nominal Group Technique.
– Just writing up a combined ‘Nominal Focus Group’ as a useful method.
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

31. References
• Focus groups
– Cousin, G. (2009). Focus Group Research. IN:
Researching learning in higher education : an
introduction to contemporary methods and
approaches. London: Routledge, pp.51-69.
– Ritchie, J., & Lewis, J. (2003). Qualitative research
practice : a guide for social science students and
researchers. London; Thousand Oaks, Calif.: Sage
Publications.
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

32. References
• Nominal Group Technique
– Practical guide: Varga-Atkins, T., with contributions from Bunyan, N;
McIsaac, J; Fewtrell J. (2011) The Nominal Group Technique: a practical
guide for facilitators. Written for the ELESIG Small Grants Scheme.
Liverpool: University of Liverpool. October. Version 1.0. Available at
http://slidesha.re/s5KPUr
– Project report: Varga-Atkins, T. with contributions from Bunyan, N;
McIsaac, J and Fewtrell, R. (2011) Using the nominal group technique
with clickers to research student experiences of e-learning. Project
Report written for the ELESIG Small Grants Scheme. Liverpool:
University of Liverpool. Available from slideshare.net at
http://slidesha.re/sc8gwT
Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.