A framework and approaches to develop an in-house CAT with freeware and open sources

Symposium 1
A framework and approaches to develop an in-
house CAT with freeware and open sources.
Tetsuo Kimura (Niigata Seiryo University)
Kyung (Chris) T. Han (Graduate Management
Admission Council)
Michal Kosinski (University of Cambridge)
Kojiro Shojima (The National Center for University
Entrance Examinations in Japan

CAT is greedy! CAT likes big pool!

The outline of the symposium
Framework to develop a CAT
(Thompson & Weiss, 2011)
Exametrika
Introduction of freeware and
R package ltm
open sources for CAT
SimulCAT
development
R package catR
Approaches to develop an in-
house CAT with freeware and Moodle UCAT
open sources Concerto

The framework to develop a CAT
Framework Proposed by Thompson & Weiss (2011)
Step Stage Primary work
1 Feasibility, applicability, Monte Carlo simulation;
And planning studies business case evaluation
2 Develop item bank content Item writing and review
or utilize existing bank
3 Pretest and calibrate item Pretesting; item analysis
bank
4 Determine specifications for Post-hoc or hybrid
final CAT simulations
5 Publish live CAT Publishing and distribution;
software development

The three stages of CAT development

Pretesting & Item Analysis: Exametrika
Construction of Item Bank R package ltm

Simulating CAT with Existing Item SimulCAT
Bank: Determine specifications R package catR

Implementing CAT: Publishing a Moodle UCAT
CAT on a software Concerto

Pretesting & Item Analysis:
Construction of Item Bank
Pretesting
Item bank
Item analysis: calibration,
elimination of misfit &
equating
Calibrated items

More pretests with new Anchored items
items and anchored items

Simulating CAT with Existing Item Bank:
Determine specifications
Simulating CAT Item bank

Examining:
Item selection rules, Calibrated items
Item exposure,
Stopping rules, etc.

Determine
CAT specifications

Implementing CAT:
Publishing a CAT on a software
Specify CAT Algorithm Item bank
On a CAT Software

Implementing CAT Calibrated items

Examine CAT Results

Framework to develop a CAT
(Thompson & Weiss, 2011)
Exametrika
Introduction of freewares and
R package ltm
open sources for CAT
SimulCAT
development
R package catR
Approaches to develop an in-
house CAT with freewares and Moodle UCAT
open sources Concerto

The three stages of CAT development

Construction of Item Bank
R package ltm



R package: ltm
• ltm: Latent Trait Models under IRT
– Dimitris Rizopoulos
• This R package provides a flexible framework for IRT
analyses for dichotomous and polytomous data under a
Marginal Maximum Likelihood approach. The fitting
algorithms provide valid inferences under Missing At
Random missing data mechanisms.
http://rwiki.sciviews.org/doku.php?id=packages:cran:ltm
• ltm: An R Package for Latent Variable Modeling and Item
Response Theory Analyses. 2006, Journal of Statistical
Software, 17(5), 1-25. http://www.jstatsoft.org/v17/i05/

ltm: Available Features
• Descriptives:
– samples proportions, missing values information, biserial
correlation of items with total score, pairwise associations
between items, Cronbach’s α, unidimensionality check
using modified parallel analysis, nonparametric correlation
coefficient, plotting.
• Dichotomous data:
– Rasch Model, Two Parameter Logistic Model, Birnbaum’s
Three Parameter Model, and Latent Trait Model up to two
latent variables (allowing also for nonlinear terms between
the latent traits).

• Test Equating:
– Alternate Form Equating (where common and unique items
are analyzed simultaneously) and Across Sample Equating
(where different sets of unique items are analyzed
separately based on previously calibrated anchor items).
• Plotting:
– Item Characteristic Curves, Item Information Curves, Test
Information Functions, Standard Error of Measurement,
Standardized Loadings Scatterplot (for the two-factor latent
trait model), Item Operation Characteristic Curves (for
ordinal polytomous data), Item Person Maps.

• Polytomous data:
– Graded Response Model and Generalized Partial Credit
Model.
• Goodness-of-Fit:
– Bootstrap Pearson χ2 for Rasch and Generalized Partial Credit
models, fit on the two- and three-way margins for all models,
likelihood ratio tests between nested models (including AIC
and BIC criteria values), and item- and person-fit statistics.
• Factor Scoring:
– Empirical Bayes (i.e., posterior modes), Expected a Posteriori
(i.e., posterior means), Multiple Imputed Empirical Bayes,
and Component Scores for dichotomous data.

Construction of Item Bank R package ltm



R package: catR
• catR : Latent Trait Models under IRT
– David Magis & Gilles Raîche
• This R package catR was developed to perform adaptive
testing with as much flexibility as possible, in an
attempt to provide a developmental and testing
platform to the interested user.
• Random Generation of Response Patterns under
Computerized Adaptive Testing with the R Package catR.
Journal of Statistical Software, 48(8), 1-31.
http://www.jstatsoft.org/v48/i08/.

catR: Available Features
• The item bank can be provided by the user previously
calibrated according to the 4PL model or any simpler
logistic model, or randomly generated from parent
distributions of item parameters.

• The package proposes
– several methods to select the early test items, several
methods for next item selection
– different estimators of ability (maximum likelihood, Bayes
modal, expected a posteriori, weighted likelihood),
– three stopping rules (based on the test length, the
precision of ability estimates or the classification of the
examinee).

• The output can be graphically displayed.

Moodle UCAT
UCAT: Rasch-based CAT program written in
BASIC (Linacre, 1987)
http://www.rasch.org/memo69.pdf

Moodle UCAT: converted into PHP so that
CATs can be administered on a major open
source LMS, Moodle
(Kimura, Ohnishi & Nagaoka, 2012)

Moodle UCAT beta ver.
Development Status
CAT setting window
•Ending conditions
•Logit to unit conversion Unit = Logit×10 + 100
•Logit bias
CAT administration window
•Set item difficulty individually or category by category
•Set student’s ability individually or as a whole
Administer CAT and provide result individually
Retrieve CAT processes and results
Under Development for Ver.1 to be released in late August 2012
Recalibration of item difficulty & estimate ability
26

CAT Algorithm: Initial Ability Estimation
UCAT Moodle UCAT
Lower Limit (LL) =
AVG(D)－ (0.5+0.5*RND) Assign each student’s initial
Upper Limit (UL) = LL＋1 ability in the CAT
administration window
based on other test results
B0 = AVG(D)－0.5*RND or intelligently one by one,
or as a whole.
AVG(D): average item difficulty
RND: random value between 0 & 1
B0 : initial ability

27

CAT Algorithm: Ability (B) Estimation
UCAT / Moodle UCAT
m
Rm Pmi
i 1
Bm 1 Bm m
Pmi (1 Pmi )
i 1 Rmthe number of successes
:

e ( Bm Di)
pmi
1 e ( Bm Di)
Pmi :probability of success of a
student of ability Bm on the i-th
dministered item of difficulty Di
28

CAT Algorithm: Standard Error (SE)
Estimation
UCAT / Moodle UCAT

1
SEm 1 m
Pmi (1 Pmi )
i 1

CAT Algorithm: Item Selection
UCAT / Moodle UCAT
Next item will be selected randomly between LL and UL
m
Rm 1 Pmi
i 1
Ability estimate when the
LL Bm m next answer will be wrong
pmi (1 Pmi )
i 1

1 Ability estimate when the
UL LL m
next answer will be correct
pmi (1 Pmi )
i 1

Rm 1 :score when he next (m-th) answer will be wrong

If no item found between LL & UL , use the closest. 30

CAT Algorithm: Ending Condition
UCAT / Moodle UCAT

Prescribed number of item
Prescribed SE
Both number of item and SE
All item

CAT Algorithm: Item Selection (logit bias)
Moodle UCAT
LL and UL can be adjusted by adding logit value to the Logit bias
box in the CAT setting window

Biased _ LL LL Bias
Biased _ UL UL Bias

Positve logit value decrease the
chance of answer correct

Negative logit value increase
the chance of answer correct

32

Questions & Answers
• Tetsuo Kimura (Niigata Seiryo University)
tetsuo.kmr<AT>gmail.com
• Kyung (Chris) T. Han (Graduate Management
Admission Council)
khan<AT>gmac.com
• Michal Kosinski (University of Cambridge)
mk583<AT>cam.ac.uk
• Kojiro Shojima (The National Center for University
Entrance Examinations in Japan)
shojima<AT>rd.dnc.ac.jp

A framework and approaches to develop an in-house CAT with freeware and open sources

Recomendados

Recomendados

Más contenido relacionado

Similar a A framework and approaches to develop an in-house CAT with freeware and open sources

Similar a A framework and approaches to develop an in-house CAT with freeware and open sources (20)

Último

Último (20)

A framework and approaches to develop an in-house CAT with freeware and open sources