This document discusses best practices for conducting oral exams, including balancing assessment of knowledge, skills, and abilities. It recommends preparing standardized questions and marking schemes to reduce examiner bias while still allowing flexibility. Questions should be open-ended, focused on clinical scenarios, and probe higher-order thinking. Examiners must be aware of different discourse styles and frame questions and expectations appropriately. Ongoing examiner training and calibration helps ensure fair and reliable oral exams.

1. Massey University | massey.ac.nz | 0800 MASSEY
Oral exams
Liz Norman
Massey University

2. insight reliability
Oral exams: strengths and limitations

3. insight reliability
Oral exams: strengths and limitations
• depth of knowledge
• analytical and reasoning
capabilities
• cognitive processes
• decision making abilities
• oral communication

4. insight reliability
Oral exams: strengths and limitations
• depth of knowledge
• analytical and reasoning
capabilities
• cognitive processes
• decision making abilities
• oral communication
• few cases/topics sampled
• lack of examiner
agreement
• assessing candidates on
different cases/topics
• assessing different
numbers of cases/topics

5. insight reliability
Oral exams: strengths and limitations
• Ability to tailor
questions and probing
increases insight
• Long cases allow you
to explore depth
• Standardisation of
questions and probing
increases reliability
• More short cases
increases reliability

6. insight reliability
Oral exams: strengths and limitations
1. Finding the right balance
2. Reducing examiner-examiner unreliability

7. Reducing examiner-examiner unreliability
• Examiner training
– Use of exemplars of different standard
– Practice in assessment
• Advance preparation of questions and prompts
– Well-structured questions aimed at assessing appropriate skills
– Clear and straightforward to answer
• Advance preparation of marking schemes
– Common picture of performance levels: Familiarisation, discussion, and
agreement between examiners on the marking schemes
– Well structured marking schemes which are easy to apply and well-
understood by examiners

8. Reducing examiner-examiner unreliability
• Allowing only preplanned questions and prompts
– Planning for contingencies
• Putting candidates at ease as much as possible
• Reducing examiner load as much as possible
• Monitor examiner agreement
• Consensus decision making

9. Finding the right balance: blueprinting
• Don’t aim to assess factual knowledge in oral exams – written
exams are better for this
• Aim to assess:
– Analytical, problem solving, reasoning skills
– Cognitive processes
– Decision-making abilities
– Judgement
– Oral communication
– Professionalism, values, ethics (if these are part of the subject guidelines)

10. Finding the right balance: questions
• Make your questions short, straightforward and clear
• Centre on clinical scenarios
• Get straight to the point.
– Remove unnecessary information
– Consider even saying what the point is:
– “This question is going to be about (disease x) and I want to explore
your approach to management.”
• Keep Qs short - aim for new Q every 5 mins

11. Finding the right balance: questions
• Make Qs open-ended
– Require interpretation, problem solving, reasoning, judgement, or
decisions
– Do not have one right answer
• Do not ask for personal experience or opinion
– Eg: “How is this usually managed in your practice?”, “What is your
opinion on?”
– It’s “unmarkable” content
– Can throw candidates. Differential effect on those not “testwise”

12. Broad types of oral Qs
Recall: Questions that ask candidates only to recall facts,
principles, or other information.
– List the most common causes of (x).
– Define (x).
– What is the most common disease associated with (x)?
– Explain two mechanisms by which (x) results in (y).
– Describe the differences between Type 1 and Type 2 (x).
McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.

13. Broad types of oral Qs
Recall: Questions that ask candidates only to recall facts,
principles, or other information.
Interpretive: Questions that require interpretation of
information, and using principles to deal with new situations.
– Describe the changes on this radiograph.
– Interpret the results of this biochemistry panel.
– Describe the abnormalities you see (in this image).
McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.

14. Broad types of oral Qs
Recall: Questions that ask candidates only to recall facts, principles,
or other information.
Interpretive: Questions that require interpretation of information,
and using principles to deal with new situations.
Problem solving: Questions that require finding a solution.
– What is the list of differentials for these findings?
– Outline you management of this case.
– What would be the best next step in this case?
– Can you account for the clinical findings in this case?
McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.

15. Finding the right balance: prompts, probes
• Plan them in advance but they can never be entirely
standardised
• Only plan a few – need to move on to next case/topic

16. Types of prompts and probes
Exploring the depth of knowledge and higher order abilities
– Can you explain your reasoning?
– What are the implications of each of these options?
– What is the justification for your decision?
– What evidence supports your judgement in this case?
– Explain how you arrived at that conclusion
– Because?
– Why is that?
– What would you do if your approach does not work?

17. Types of prompts and probes
Seeking more detail from the candidate
– Any other differentials?
– Can you tell me which specific drug you would use?
– Which particular suture pattern would be most appropriate?
– Which particular signs are leading you to conclude that?
– Can you elaborate on that?

18. Types of prompts and probes
Clarifying the question being asked or redirecting the candidate
– What I am getting at here is…
– Can I get you to focus on (x) in this case?
– Can you tell me all the options rather than what you would do?
– Just coming back to something you mentioned earlier about (x), can you
tell me…
Asking about things that the candidate has missed out
– Can you tell me any other implications? What about implications for the
whole farm?
– One thing you didn’t mention was (x). Can you tell me about that?

19. Discourse
Three different types of discourse in oral exams (Roberts, 2000)
• Personal experience
– talk that deals with personal experience and feelings and often
involves narratives and stories
• Professional
– talk of professionals to clients and colleagues to colleagues
• Institutional
– principles: abstract, analytic, depersonalised.
Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375

20. Discourse
• Important to frame your question in the same discourse that
you expect an answer
• Usually you expect an answer in institutional discourse.
– Candidates may not know this, especially those inexperienced in this
type of exam, from ethnic minorities or overseas trained.
– Causes candidate uncertainty and can result in differential bias in
exam scores
• But in oral exams its easy to slip into one or other of the other
discourses
Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375

21. The following question combines elements of both professional and personal
experience.
Question: If someone took out a personal complaint against you, how would you react?
Answer: That's where my personal stress management plan comes in . . . .
Although the theme of the question was professional and personal, the candidate
responded in an institutional way. Instead of describing how she would feel
(personal experience discourse) or what she did in her practice in relation to a
specific case (professional discourse), she produced an abstract and analytical
statement (institutional discourse) that satisfied the examiners as a response they
could mark. Head nodding and mutual smiles provided confirmation.
Quote from Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231),

22. Ancillary materials
• Need to have clear function
• Use them purposefully, not as decoration
• Ensure they clearly show what they are intended to show,
when displayed in the way candidates will see them
– Check with someone who did not see the case

23. Marking schemes
• Rate on only a few criteria eg
– Factual recall
– Analysis and interpretation of data
– Problem solving
– Verbal presentation
• Plus overall rating
• Separately grade each main question/topic
• Provide descriptors for 3-4 levels of performance eg: clear fail,
marginal fail, bare pass, good pass

24. Conducting oral exams
• Create the right climate
• Introductions and non-threatening questions (eg weather)
• Convey a sense of genuine interest
• Adapt body language to candidate’s needs
– Some need eye contact, some find it off-putting
– Candidate interpretations of “mhm” and “uhuh”:
• Encouragement to go on
• Stop talking so I can ask the next Q
• Well done you have passed
• Avoid humour

25. Not to much structure, not too little
• Plan all the questions and possible prompts including how
these should be worded
• But on the day don’t read as if from a script
• Adapt your language and body language to the candidate
• Change the order or leave out probes if necessary
• Use clarification and redirection only if necessary
• Give all candidates the same information
• Metacommunicate

26. Dealing with issues
• When the candidate can’t answer or says very little
• Ensure candidate has had enough time to respond (but don’t
give them endless time)
• Do not give hints
• Ask the candidate if they would like the question clarified. If
they do, be careful not to
– give hints or additional information
– change the discourse or requirements of the answer
• Often best just to move on to next question

27. Dealing with issues
When candidates go off track or take too long
• Use clarifying or redirecting questions
– What I am getting at here is…
– Can I get you to focus on (x) in this case?
– Can you tell me all the options rather than what you would do?
– Just coming back to something you mentioned earlier about (x), can
you tell me…

28. Dealing with issues
When candidates are exceedingly nervous
• Trying to help often makes things worse
• Poor performance may be the cause: move on to the next
question

29. Scoring candidates
• Record grade for each question as soon as it is finished
• If you are thinking of giving quite a high grade think “What
would the candidate have to have done to earn an even higher
grade?” – helps examiners use the whole scale
• At end of exam record your holistic judgement as well as
sum/average of scores
• Do not confer with your co-examiners until you have recorded
your own judgement
• Discuss your judgements and reach a consensus decision