1. Quality Aspects of Computer-based Skills Assessment
Selection Criteria for Software Tools
Context of CBA
Needs Technological + Computer Item Internet Budgets
Work in this area is focused on computer-supported Methodological Innovation
Technology
CAT
Types Multimedia
,,,
methodologies for assessing the outputs of learning and
tools for computer-based testing. Contextual Parameters
The Joint Research Centre of the European Commission Rationale Measurement Assessment User User ICT
Design Groups Profiles Resources
is carrying out a research project on “Quality Criteria for
Test
Open Source Assessment Platforms”. Purpose Subjective
measurement
Stakes
Publishers
Languages IT
infrastructure
Objectives Objective Approach Test Special Networking
measurement Developers Needs resources
Functions Instruments Test etc.
Takers
Background Subject
Area
Phases and
Timing
Test
Administrators
• “Lisbon Strategy” (2000, 2006): Recommendations on key
Software Characteristics needed
competences for lifelong learning and the use of a common
reference tool to observe and promote progress CBA Requirements
• New European surveys to be carried out on foreign
languages and learning-to-learn Assessment Item Language Hardware Network Software
Features Types Version(s) Requirements Requirements Requirements
• Increasing importance of computer-based assessment (CBA)
in international studies (e.g. PISA, TIMSS, PIRLS) Authoring / User (group) Test Results Reporting Access Interoperability etc.
Item Creation Management Delivery Analysis Restrictions
Potentials
Software Selection Process and Quality Indicators
CBA has a great deal of advantages compared with Features available
traditional pencil and paper testing, namely:
• types of items Software
Quality
• types of assessments (e.g. adaptive testing)
Efficiency Portability Reliability Maintainability Usability Functionality
• speed of test execution and administration ISO 9126
• distributive testing administration and preparation, etc. Time
Behaviour
Adaptability Maturity Analysabilty Understandability Suitability
Resource Fault
Installability Changeabilty Learnability Acurateness
Behaviour Tolerance
Conformance Recoverability Stability Operability Compliance
Challenges
Replaceability Testability Security
Product-
Interoperability
related
• High stakes assessments can imply issues of security Market Regularity of Warranty Support Licence / Price
• The infrastructure required (computers, logistics, Share Updates
Consistency of Price
telecommunications, etc.) Documentation
with comparable
Products
• Validity (confounding variables) Training
(incl. Time and Price)
Price consistent with
estimated price range
• Hardware performance Provider-
related
Totals Cost
of Ownership
• Quality of Internet-based testing
• Training (additional skills are required for test takers, test General
Qualifications
Financial
Stability
Reputation Quality
Assurance
authors and test administrators)
• Organisational issues (some countries are highly On-Going Freshmeat Volume of Number of Repository
decentralised, other are highly centralised with regards to Open Source-
related
Effort User Rating available
Documentation
Downloads Checkouts
decision making)
• Interoperability among testing platforms and sharing of Degree of Degree of Volume of Number of Number of Support
Peer Review Stakeholders Mailing Lists Contributors releases Community
entities involvement
• Costs associated are said to first investments but then they
are expected to be lower with time Selected Product
CB Test Delivery and Assessment
Standards Guidelines
ITC: International Buros Center for
Guidelines on APA Standards AERA: Standards for ETS Standards for Testing
Computer-Based and (www.apa.org/ Educational and Quality and Fairness (www.unl.edu)
Internet-Based science/testing.html) Psychological Testing (www.ets.org)
Delivered Tests
(www.intestcom.org/) ISO/IEC etc.
Association of Test 24763
Publishers (ATP): Toetsgids
EFPA Review Model (www.toetswizzer.nl)
Guidelines for CBT
(www.efpa.be)
ADL Scorm ISO/IEC
ISO/IEC
IMS QTI 23988
19796
Contact
© European Communities, 2004
Friedrich Scheuermann / Ângela Guimarães Pereira
European Commission • DG Joint Research Centre • IPSC
Knowledge Assessment Methodologies (QSI/KAM)
Centre for Research on Lifelong Learning (CRELL)
Web: http://kam.jrc.it • http://crell.jrc.ec.europa.eu
E-mail:friedrich.scheuermann@jrc.it / angela.pereira@jrc.it