Assessment validation for the national vocational qualification in amenity horticulture level 1

1. Singing from the Same
Hymn Sheet
A VALIDITY STUDY ON ASSESSMENT OF THE BARBADOS NVQ-B
LEVEL I IN AMENITY HORTICULTURE
Roxanne Waithe
The University of the West Indies, Cave Hill Campus
4th October 2016

2. Background to the study
oBarbados National Vocational Qualifications (NVQ-Bs) were
established in 2003 but up until 2008 only the NVQ-B in Amenity
Horticulture was administered
oNVQ-Bs for six occupations – organizations were not using them.
Competency-based qualifications……Why?
oNVQ-B assessments – measurement of competency for the NVQ-B
in Amenity Horticulture

3. Presentation Outline
Purpose of the study
Research questions
Review of the Literature
Methodology
Results
Implications and conclusions

4. Purpose of the study
1.Examine how the NVQ-B was being used to
measure employee competence in Amenity
Horticulture
2.Establish the validity of assessment in the
NVQ-B Amenity Horticulture Level I

5. Research questions - Phase 1
Examine how the NVQ-B is used to measure employee competence
based on experiences and perceptions of key users
a) How are occupational standards for the NVQ-B used in compliant
organizations?
b) How does the assessment process for the NVQ-B establish competence
in relation to occupational standards?
c) What are some of the challenges associated with administering the
NVQ-B?
d) What is the perceived influence of NVQ-Bs on the accomplishment of
competency?

6. Research questions – Phase 2
1) Establish the validity of assessment in the NVQ-B amenity horticulture
level 1
1. How does NVQ assessment reflect the evidence requirements outlined
in the national occupational standards for Amenity Horticulture?
1) How congruent are assessors’ interpretations of evidence for the NVQ in
Amenity Horticulture?
2) To what extent does the evidence collected during assessments for the
NVQ-B Level I in Amenity Horticulture establish the competencies
outlined in the occupational standards?

7. Review of the Literature
LITERATURE REVIEW I
Making NVQ
assessment
decisions
•Perceptions on the
Proficiency of NVQs
Evolution
of NVQs
Structure
& nature
of NVQs
LITERATURE REVIEW II
Approaches
to
Assessment
Validation
Modern
Validity
Theory
Defining
validity
Underlying
assumptions &
threats to
validity
Are NVQs
Valid?
Validity
theory in
practice

8. Seminal Sources - Literature Survey I
 Swailes & Roodhouse (2003) – Key features of NVQ
qualifications
 Hager & Butler (1996) - Comparison of assessment in academic
and vocational qualifications
 Qualifications and Curriculum Authority (2006) – Comparability
study of NVQ assessment in practice
 Roe et al. (2006) Perceptions and use of NVQs
Competence: the ability to apply knowledge, understanding, practical,
and thinking skills to achieve affective performance to the standards
required in employment (QCA, 2006)

9. Seminal Sources - Literature Survey II
 Greatorex & Shannon(2003) – Evaluation of validity of NVQs
 Messick (1989, 1995, 1996, 1998) - Validity theory
 Kane (1992, 2001, 2004, 2006, 2009, 2013) – Validity process and
validity evidence
 Johnson & Blinkhorn (1992) Model for validating NVQ
assessment
Validity: the extent to which an assessment decision about a candidate,
based on performance by the candidate, is justified.
(TVET Australia, 2010)

10. Philosophical assumptions
Interpretive
NVQ-B users
Multiple sources
of evidence
• Interpretivist philosophy – facilitate
experiences and perceptions of all key NVQ-
B users (employers, administrators,
candidates, assessors)
• Interpretative approach to assessment
validation – multiple data sources at various
assessment stages
• Interpretive inquiry processes serve to
expand validation practice to include
diverse evidences for the generation of
multiple-perspective validity arguments.
(DeLuca, 2011)

11. Case Study Methodology
1. Case study research was compatible with my philosophical
viewpoint – interpretivist orientation toward data collection and
analysis
2. Case study is ideal when the research questions take the form of
‘how’ and ‘why’. (Yin, 1984)
3. Case study is an appropriate vehicle for interpretive investigations
because of its ability to accommodate different research
techniques, both qualitative and quantitative (Yin, 2003)

12. Case Study Methodology
CASE: NVQ-B AMENITY HORTICULTURE
LEVEL I
Phase I:
Exploratory study
(How does assessment
work?)
Phase II:
Assessment
Validation
Assessment
system
(processes)
Assessment
content
Assessment
decisions

13. Data collection – Phase I
Participants
• NVQ candidates (6)
• NVQ assessors (6)
• TVET technical
officers (3)
• Employers (25)
Methods
• Interviews
• Document analysis
• Surveys
Data
• NVQ system
(Operational context)
• NVQ assessment
• NVQ perceptions

14. Key findings from the research – Phase I
1. The NVQ-B qualification is patterned after the traditional British NVQ model. NVQ-B
compliant organizations use occupational standards for the purposes of training, developing
job descriptions, and conducting employee interviews.
2. NVQ-B Assessment Process – involves NVQ assessors gathering evidence of candidates’
competence through observation, questioning, photography or witness testimony
3. NVQ-B Challenges – were mostly assessment related including timely record-keeping for
evidence of competence, and practical difficulties associated with assessing people in the
workplace
4. NVQ-B Perceptions -
Improves
employee
performance
Improves self-
efficacy
Too much
paperwork
What NVQs?

15. Data collection – Phase II
Participants
• NVQ assessor trainers (2)
• NVQ assessors (3)
• TVET technical officers (3)
• NVQ candidate trainers (3)
• NVQ Coordinators (4)
• External verifier (1)
NVQ Assessment
Validation Framework
• Interpretive argument
• Validity argument
• (Data collection:
interviews, document
analysis, two focus groups)
Data
• Assessment process
• Assessment content
• Assessment
decisions

16. Assessment Validation for the NVQ-B in
Amenity Horticulture Level I
The validation process is the assembly of an extensive argument for the claims
that are made about an assessment. (Kane, 2006)
Purpose of the assessment validation was to produce
validity evidence for assessment including
representativeness of content, coherence between
occupational standards and assessment tasks, and the level
of congruence of assessment decisions under different
conditions

17. Assessment Validation Framework
Present interpretive
arguments about the
NVQ-B Amenity
Horticulture Level I (Use
research questions)
Collect validity evidence
to support or refute
claims: document
review, interviews, focus
groups
Systematic evaluation of
evidence concerning
assessment process,
assessment content,
assessment decisions
Build a validity
argument based
on evaluation of
evidence (Messick, 1989; Kane, 2006, 2009)

18. Interpretive Argument 1
Assessment of the NVQ in Amenity Horticulture provides a
reasonable indication of competence in the occupational standards
Supporting claims
The content coverage of the NVQ assessment adequately represents
the occupational standards for amenity horticulture
The NVQ-B assessment system uses a rigorous methodology for
providing evidence of competence (knowledge, understanding and
skills) in amenity horticulture

19. Interpretive Argument 2
Assessments for the NVQ in Amenity Horticulture Level I provide
meaningful data on candidates’ readiness to perform in this field
Supporting claims
Rules for assessment judgement are consistently and accurately applied
Candidates deemed competent in Amenity Horticulture Level I are likely
to be given the same decision regardless of assessor or assessment
location
A decision of ‘competent’ in Amenity Horticulture is a reliable indicator of
performance in the occupational area

20. Key findings from the research – Phase II
Validity Evidence for assessment content
Generally good coherence between NVQ assessment content and evidence requirements outlined in
national occupational standards for amenity horticulture.
NVQ coordinators from the four registered training organizations concur that assessments
adequately cover the content standards that they are designed to measure.
Threats to validity evidence
 Measurement of some constructs related to health and safety that are critical to occupational
competence are not sufficiently covered in practical assessment tasks
The evidence guide for assessors do not provide rules for decision-making or assessment exemplars
that would allow comparison of content across parallel assessment events (nursery/golf
course/cricket pitch/farm)

21. Key findings from the research – Phase II
Validity Evidence for assessment process
The NVQ-B system conducts assessments for amenity horticulture using a systematic method
for collecting evidence of competence and making a judgement that the candidate can perform
work in accordance with the occupational standards.
The assessment process is administered in four stages by TVET Council and the RTOs described
to some extent in the NVQ-B user guide.
Threats to validity evidence
 Blueprint for NVQ-B assessment not fully documented
The rating scale of 1 – 5 which was introduced to assess skills and competence is somewhat
ambiguous and could lead to errors in judgement

22. Key findings from the research – Phase II
Validity Evidence for assessment decisions
Assessors have been trained to judge candidates’ performance against occupational standards.
High level of inter-rater agreement amongst assessors was recorded for a mandatory unit of
assessment.
Threats to validity evidence
 Rating scale for performance criteria does not provide enough information
Assessors are experts in either one or two areas of amenity horticulture

23. Validity evaluation for NVQ Amenity
Horticulture Level I
INTERPRETIVE ARGUMENT CLAIMS VALIDITY EVALUATION
1. Content coverage of the NVQ adequately
represents the occupational standards for amenity
horticulture
Evidence from competency mapping and participant
interviews support the claim for content
2. The NVQ-B assessment system uses a rigorous
methodology for providing evidence of competence
in amenity horticulture
Evidence from document review and participant
interviews support the claim for rigor
3. Rules for assessment judgement are consistently
and accurately applied
Evidence from document review and participant
interviews do not support the claim for consistent
assessment judgement
4. Candidates deemed competent are likely to be
given the same decision regardless of assessor or
location
Research evidence does not support claim for equivalent
assessor decisions
5. A decision of ‘competent’ is a reliable indication
of performance in the occupational area
Evidence supports claim for ‘competent’ decisions where
assessors assess in their area of specialization

24. Validity Arguments for NVQ-B in Amenity
Horticulture
VALIDITY ARGUMENT I
Assessment of the NVQ-B in Amenity Horticulture Level I provides a
reasonable indication of competence in specific areas of Amenity
Horticulture outlined in the occupational standards
VALIDITY ARGUMENT II
Assessments for the NVQ-B in Amenity Horticulture Level I provide
meaningful data on candidates’ readiness to perform in their specific
area of amenity horticulture.

25. Research implications
How did the results compare with what was presented in the literature review?
1. Compared to other more commonly known qualifications, the NVQ-B is not easy to
interpret. Cox (2007) review of employer and employee perceptions; Qualifications
and Curriculum Authority (2010) comparability study of NVQ assessment in practice
2. The validity of assessors’ judgements for the NVQ-B in Amenity Horticulture is
definitely influenced by assessor experience and training. Johnson (2008) judging
vocational portfolios; Wolf (1995) competence based assessment
3. Assessment in the NVQ-B Amenity Horticulture tends to focus on adherence to the
assessment process more than on standardizing assessment decisions. Greatorex
and Shannon (2003, 2008) standardizing assessors’ judgements

26. Singing from the same hymn sheet
• All NVQ-B users refer to the occupational standards
for Amenity Horticulture Level I as the primary
resource to guide assessment
• Interpretation and use of the occupational standards
depends on two factors:
1. Role in the NVQ assessment process
2. Context in which the assessment takes place

27. Thank you.