The document discusses analyzing student confidence data from diagnostic tests to better understand their knowledge levels and identify alternative conceptions. It describes how diagnostic questions were designed to probe specific alternative conceptions in NMR, and analyzing student responses helped categorize common errors. The relationship between diagnostic test scores and exam scores was also examined, identifying groups of students that did better or worse from the diagnostic to the exam.

1. Confidence: Analysis of the confidence data indicates a range of behaviours
from students having low confidence in correct answers indica8ng
uncertainty in knowledge or ability, through to students with high
confidence incorrect answers, exemplifying the Dunning-Kruger Effect. The
advantage of a diagnos8c test may be to encourage these students to
confront their level of knowledge.
In the 2nd year diagnos8c test, ques8on 5 was answered correctly by 26
students and was based on VSEPR. The frequency of points on the
confidence scale is similar and in the moderately confident range (3 – 5). It
differs for low and high confidence depending on whether the ques8on was
answered correctly or incorrectly. Ques8on 10 was answered correctly by
the majority of students, but has a very different profile to Ques8on 5.
Changing how we write exam ques6ons: Construc8ng correct structures is
viewed as an essen8al process that ‘expert’ chemists do to understand a
molecule. It is also viewed as one that is more cogni8vely demanding and
may involve several steps: iden8fy bonding and non-bonding electrons,
create a 2-D representa8on, translate that into a 3-D representa8on2,
determine whether some subs8tuents will be ‘different’ (e.g. axial and
equatorial subs8tuents), iden8fy any symmetry the molecule may have. By
separa8ng the molecular structure iden8fica8on step from the mNMR
thinking step in examina8on ques8ons, students are less likely to be
overwhelmed with extraneous informa8on while s8ll working through the
demands of structure determina8on. This also allows ease of applica8on of
an ‘error carried’ approach to marking.
Old Style: Predict the theore8cal low temperature 19F NMR spectrum of PF5.
[6 marks]
Revised Style: (a) Use VSEPR theory to draw the structure of PF5. [1 mark]
(b) : Predict the theore8cal low temperature 19F NMR spectrum of of PF5.
[5 marks}
Diagnosing Alterna8ve Concep8ons in NMR
Eilesh Tolly-Brewster, Sam Goodwin and Katherine Haxton*
k.j.haxton@keele.ac.uk @kjhaxton
School of Chemical & Physical Sciences, Keele University
Introduc6on: diagnos8c tests can provide insight into prior knowledge and alterna8ve concep8ons held by students before teaching. This can then be useful in
direc8ng students towards addi8onal learning resources, and tailoring taught sessions to the level of the students. Mul8ple choice ques8ons (MCQs) can be
carefully designed to probe common errors or alterna8ve concep8ons, and in some cases the ability of the student to apply their prior knowledge to unfamiliar
scenarios through recall or extrapola8on1. Our 1st year diagnos8c test covers introductory topics in spectroscopy including basic NMR and is intended to
evaluate the prior learning of the students, as well as probing some known alterna8ve concep8ons that they may hold from pre-university teaching. Our 2nd
year diagnos8c test covers VSEPR (mainly hypervalent p-block compounds) and NMR (I=1/2 nuclei), aimed at students ajer 2nd year organic NMR teaching but
before inorganic mul8nuclear NMR teaching. The 2nd year test includes ques8ons that require the students to apply their knowledge of NMR to unfamiliar
scenarios, and is ideally placed to iden8fy alterna8ve concep8ons arising from 1st year teaching.
Methodology: Alterna8ve concep8ons and common errors in understanding
and applying NMR theories have been iden8fied by:
- analysis of wriken exam answers
- focus groups and interviews
- analysis of free text answers to preliminary diagnos8c test
MCQs have been created to test specific alterna8ve concep8ons. Each
distractor is mapped onto a specific alterna8ve concep8on or common error.
Ques8ons range from straighlorward tests of knowledge (familiar, recall),
through to ques8ons that test conceptual understanding1 (unfamiliar,
extrapola8on).
MCQs are coupled with free-text space to explain answers/show working,
and a 7-point confidence scale from ‘not confident’ through to ‘highly
confident’. Analysis of free-text comments allows for greater understanding
of any issues, and whether the ques8ons are themselves confusing.
Common Errors: Analysis of 2nd year mul8nuclear exam answers to
categorise common errors was undertaken and 12 scripts analysed, reasons
for lost marks recorded by category of error. ‘Other’ refers to an error that
was specific to the design of the ques8on so would unlikely to be seen again.
‘Omission’ refers to not answering part of the ques8on. All answers analysed
had mul8ple errors.
The majority of errors came from incorrect molecular structures, highligh8ng
that molecular shape remains a barrier to student learning at higher levels of
studies2. Some integra8on and splinng errors were directly linked to
incorrect molecular structure, others were incorrect for the postulated
structure.
2nd Year Diagnos6c Test (DT): The rela8onship between DT results and exam
ques8on marks was probed (Graph 2). Four groups of students were
iden8fied based on the magnitude and direc8on of change in marks. Groups
C and D indicate students showing improvement from DT to exam, with D
(increase >37%) showing par8cularly good achievement in the exam. B
represents students with limited change in mark (±7%), and A represents
students who’s marks decline from DT to exam which may be for a variety of
reasons not necessarily related to the content. The range in marks is such
that the DT is of limited prognos8c value, however plans for future years
involve direc8ng students with low DT scores to addi8onal learning
resources.
References:
1. Holme, Luxford and Brandriet, J. Chem. Educ., 2015, 92, 1477- 1483
2. Tiekmeyer et al, J. Chem. Educ. 2017, 94, 282
Online Material: want to read more or try some
diagnos8c test ques8ons?
Please visit: bit.ly/Diagnos6cNMR
0
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 100
DT Score
Exam Ques6on Score
A
B
C
D
How Much People Think They Know How Much People Know
Dunning-Kruger Effect
High Confidence,
Wrong Answer
Low Confidence,
Correct Answer
Increasing Confidence,
Correct Answer Learning?
0
10
20
30
1 2 3 4 5 6 7
% Students
Q5 (26/75 correct) Q10 (61/75 correct)
0
10
20
30
40
1 2 3 4 5 6 7
% students
Q5 Incorrect Q5 correct
0 20 40 60 80
VSEPR
Splinng
Integra8on
Omission (part of ques8on)
Other (ques8on design issue)
% of errors in answers analysed