Presentation from the PISA for Schools webinar on 19 November 2020. International Linking Study - Connecting the PISA-based Test for Schools (PBTS) to the PISA scale

1. International Linking Study
- Connecting the PISA-based Test for Schools (PBTS)
to the PISA scale -

2. Why did we conduct an international linking study?
PISA score
• measured with PISA cognitive items (competency-based items) via Computer-based Testing (CBT).
• estimated using item parameters (Item Response Theory model parameters - difficulty and discrimination)
of the cognitive items that are calibrated with the representative samples of the world’s 15-years-old students.
PISA-based Test for Schools (PBTS) score
• measured with items that are developed under the same analytical framework of the PISA test.
• was estimated using own item parameters based on Paper-and-Pencil Testing (PPT);
PBTS have transited from PPT to CBT in the 2019 test administration.
The international linking study was conducted in order to
• Obtain new parameters of the PBTS items that are fully aligned with PISA scale (CBT).
• Confirm PBTS items measure the same abilities that PISA items do.
PBTS is the only school level assessment in which scores can be compared with the PISA scale.

3. How did we conduct the international linking study?
Booklet assembly
• Divided all OECD-test items into 6 clusters in each domain. Each cluster consisted of 6-10 items.
• Inserted 11-13 PISA items into the clusters and composed booklets
Each booklet had around 20 items for each domain (60% PISA items and 40% PBTS items).
• All students took both PISA items and PBTS items.
Test administration
• Data were collected from over 95,000 students in 4 countries (Different languages / continent).
• Each country had the sufficient number of samples for the item parameter estimation.
Analysis
• Confirmed identity between constructs that we measure in PISA and in PBTS.
• Estimated new item parameters of PBTS items on the PISA scale
based on the item parameters of the PISA items.
• Validated the PBTS items and its item parameters for each country
Estimated country-specific item parameters for some items.

4. What did we obtain in the international linking study?
Response data of students for PISA items and PBTS items.
• X-axis; Item IRT difficulty parameter on PISA scale (=PBTS scale).
• Y-axis; Average score of the samples in the international linking study.
• Dots represent PISA and PBTS items.
• Colour represents discrimination parameter (warmer colours mean less discriminative items).
PISA items PISA and PBTS items
Put PBTS items
on PISA scale

5. What did we obtain in the international linking study?
• Successfully estimated new item parameters of PBTS items on PISA scale
(Confirmed with data from 4 countries; Brazil, Spain, Russian Federation, and United States).
• Consequently, score of PBTS and PISA can be put on the same scale.

6. The International Linking Study
• succeeded in linking PBTS to PISA scale;
• scores of the PBTS can be fully aligned with that of the PISA.
• provides a solid basis for ongoing monitoring of item traits
to assure quality of PBTS and its psychometric properties.
Thank you!
Conclusions