This talk describes the design, development and some validation work of the revised Oxford Online Placement Exam.

1. THE TESOL-ILTA JOINT SESSION The Design and Development of the Web-Based Online Oxford Placement Exam James E. Purpura, Teachers College, Columbia University [email_address] TESOL, Boston March 2010 (Thanks to Yoko Ameriks, Simon Beeston, Jee Wha Dakin, Kristin DiGennaro, & Kirby Grabowski) (For support thanks to: Oxford University Press, TESOL International & The International Language Testing Association (ILTA—www.iltaonline.com)

4. Initial Design Decisions • The revised OOPT would have 5 sections: a language knowledge section (i.e., grammatical & pragmatic knowledge) & 4 language use (i.e., skills) sections. • The decision to measure language knowledge separately stemmed from research showing that grammatical knowledge (i.e., form & meaning) is a critical resource (Purpura, 1999, 2006) & a strong predictor of the ability to communicate in meaningful & pragmatically appropriate ways (Ameriks, 2009; Chang, 2002, 2004; Grabowski, 2009; Kim, 2009; Liao, 2009). • The language knowledge section would consist of 4 tasks: (1) MC grammar (F&M); (2) MC Meaning (Literal & Intended); (3) Cloze (F&M); & (4) MC Pragmatic Meaning.

5. Initial Design Decisions • The grammar section was developed first since it would be used to determine what CEFR-level-specific items the examinees would receive in the rest of the adaptive test. This is a computer-adaptive test.

6. How Does the Computer Adaptive System Work? 1. Candidate views test demo info, then registers & selects level description e.g. B1 4. Listening items (15 items) selected on the basis of level at end of the Task 3. 2. Candidate starts test with B1 level item from Task 1 (MC F&M). Person ability is calculated & the next item selected & so on, until 10 items are administered—at the point the person may be identified at B1 or other level Item level adaptive across the CEFR levels 3. Task 2 (Meaning) testlet is selected from database based on the examinee’s level at end of Task 1. Task 2 then administered. The same procedures occurs for Task 3 (F&M). Pre-constructed CEFR level tests The score is derived from 2 levels combined into one “block” score showing the examinee’s relative position each. This is a probabilistic model based in IRT. 5. Score generated: etc 48 47 46 45 etc. Listening Average Grammar

7. How the tests work - adaptively B1--1st item CORRECT Easier 7th item CORRECT Easier 5th item CORRECT Harder 6th item INCORRECT Harder 4th item INCORRECT Easier 9th item CORRECT Harder 3rd item CORRECT Harder 2nd item CORRECT Test focuses on student’s ability & adapts to provide items targeted for that person’s ability level – final test is shorter but still accurate. Harder 8th item INCORRECT

8. Initial Design Decisions • The listening section was developed last. Listening ability is defined in terms of understanding a range of literal, intended & implied meanings. This section consists of 3 tasks.

9. The Language Knowledge Section Assessing grammatical & pragmatic knowledge

10. The Language Knowledge Section Purpose: to measure the grammatical & pragmatic knowledge resources that examinees need to communicate at six levels of CEFR proficiency . Information needed • How the structures & associated meanings in the OUP course books related to the scaled descriptors of grammatical knowledge in the CEFR. — Could we get this from the CEFR itself? • How grammatical knowledge was defined & operationalized in the 6 CEFR levels — Could we get this from the CEFR itself?

13. The Oxford Curricula (aligned with CEFR) Grammar covered at A1/A2 level • countable & uncountable nouns • adjectives & adverbs of frequency • how much/many & other quantifiers • present & past simple/present perfect • present continuous, going to future • can/can’t , like + ing, would like to • some/any; there is/are/was/were • subject & object pronouns • possessive adjectives/pronouns • demonstrative adjectives/pronouns • prepositions of time & place • comparative/superlative adj. • Saxon genitive • telling the time Vocabulary & Topic Areas • classroom expressions • personal information • daily routines, days of the week • family, holidays, • shopping, food & drink, Instances of Language Use • greetings (formal/informal) • dinner invitations • making small talk • past experiences, • making reservations

14. Grammar from the CEFR B1 > B2 Intermediate [Independent user-Vantage] • Uses accurately & appropriately vocabulary adequate for understanding and responding to a range of familiar & some unfamiliar topics & can identify appropriate lexical items . • Can handle with reasonable accuracy simple & some more complex grammatical forms . Can produce longer relevant utterances in spoken & written discourse using a variety of cohesive devices . • Can read & understand different kinds of texts. Can identify overall meaning & specific detail attitudes & opinions & can understand text structure. • Can understand standard spoken language on both familiar & unfamiliar topics relating to personal social academic life or work. • Can communicate in most situations ask & respond to questions appropriately & participate in interactions without resorting to repair strategies.

15. SLA Research—Developmental Orders

16. SLA Research—Developmental Sequences

17. SLA Research—Developmental Sequences

18. We decided to use Purpura’s (2004) model of language ability as a basis for test development at 6 CEFR levels & as a basis for validation. Models of Communicative Language Ability as an Organizing Framework Models of language ability: Canale & Swain, 1980; Bachman, 1990; Bachman & Palmer, 1996; Chapelle, 1998; Purpura, 2004, etc.).

19. Forms Meanings Grammatical Knowledge Based on Jaszczolt (2002) & Wedgwood (2005) The model assumes that grammatical forms are the primary resource for conveying a range of literal & intended meanings. What is Language Knowledge? Knowledge of forms at the sentence & discourse levels Logical representation of semantic meanings conveyed by grammatical forms Use in Context Pragmatic Meanings Production of implicatures, irony, stance, affect, identity, power, metaphor, politeness, etc. This serves to convey a range of implied ( pragmatic) meanings in context. And this determines overall communicative effectiveness.

20. Model of CLA Graphological or Phonological Meanings Graphological or Phonological Forms Lexical Meanings Lexical Forms Morphosyntactic Meanings Morphosyntactic Forms Cohesive Meanings Cohesive Forms Info. Management Meanings Info. Management Forms Interactional Meanings Interactional Forms Sentence Level Discourse Level Language Ability Grammatical Knowledge Grammatical Form Grammatical Meaning

21. The OOPT Language Knowledge Section • Item distractors (especially in task 1) were written to reflect partial knowledge of the form and/or meaning being tested, when possible. • Specs for each task were drawn up, one for each CEFR level tested (yes, 24!). Items were designed “theoretically” to target the different levels. • Designed to measure grammatical & pragmatic knowledge. Tasks Task 1: Testing knowledge of forms & meanings 10 items (out of 300) Task 2: Testing meanings (literal & intended) 10 items (out of 142) Task 3: Testing forms & meanings 10 items (out of 91) Task 4: Testing pragmatic meanings (implied) To be determined

22. Task 1: Assessing Forms & Meanings

29. Item Foci for Grammar--abbreviated

33. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

34. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

35. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

36. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 Easy Hard A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

37. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 Easy Hard A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

38. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 Easy Hard A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

39. Item Descriptors for Grammatical Form with the Item Focus ‘Tense & Aspect’ Within the A2 CEFR Level for Task 1 Easy Hard A2 Key Item Difficulty Hypoth. CEFR Future progressive we'll be packing 24.74 B2 Negation with “do” aux I don't 25.51 A2 "will" used with offers I’ll take 28.64 B1 Future time in question form will you do 32.46 B1 Future tense I’ll do/I’m going to do 33.50 B2 Present continuous to express future time am going 34.24 A1 Present tense of verb "go" go 34.28 A1 Simple past we went 34.36 A2 Present continuous in fixed expression “Who shall I say is calling?” is calling 35.38 B1 Simple past felt 35.45 A2 Simple past in interrogative did 37.18 A2

40. A2 Level-- Hypothesized at B2 ID: 0104 Select a word or phrase to complete the conversation shown below. 0 0 1 0 Example: Dichotomous Scoring Woman: What are you thinking about? Man: I was just thinking-- this time next week _____ our suitcases to go on holiday. a. we’ve packed b. we’d be packing c. we’ll be packing d. we’ve been packing

43. Example: Partial Credit Scoring C2 Level ID: 0034 Select a word or phrase to complete the conversation shown below. 2 3 1 0 Woman: It’s time I _____ a new car. Man: Why? What’s wrong with the one you’ve got? a. buy b. bought c. should buy d. had bought

44. Example: Partial Credit Scoring Item ID 0034

45. Example: Partial Credit Scoring A1 Level ID: 0008 Select a word or phrase to complete the conversation shown below. 3 1 0 1 12.85 16.30 18.48 40.15 ??? Man: What do you usually _____ in the evenings? Woman: I like to watch television. a. do b. look c. have d. make

46. Dichotomous vs. Partial Credit Item Difficulty Dichotomous scoring

47. Dichotomous vs. Partial Credit Item Difficulty Partial credit scoring

48. Dichotomous vs. Partial Credit Item Difficulty Dichotomous scoring Partial credit scoring

49. Dichotomous vs. Partial Credit Item Difficulty Dichotomous scoring Partial credit scoring

50. Dichotomous vs. Partial Credit Item Difficulty Dichotomous scoring Partial credit scoring

53. Test taker performance #1

54. Task 2: Assessing Literal, Intended & Implied Meanings

55. Model of CLA Graphological or Phonological Meanings Graphological or Phonological Forms Lexical Meanings Lexical Forms Morphosyntactic Meanings Morphosyntactic Forms Cohesive Meanings Cohesive Forms Info. Management Meanings Info. Management Forms Interactional Meanings Interactional Forms Sentence Level Discourse Level Language Ability Grammatical Knowledge Grammatical Form Grammatical Meaning

56. Task 2: Example Item Literal meaning of the elliptical form “not really.”

57. Task 2: Classical Item Analyses The percentage of students who answered correctly The percentage of students who answered A & C The percentage of students who selected each option in the bottom scoring and the top scoring group. The discrimination index; top group – bottom group, so .96 -.63 = .33 The point biserial gives an overall index of discrimination

58. Task 3: Assessing Forms & Meanings

59. Model of CLA Graphological or Phonological Meanings Graphological or Phonological Forms Lexical Meanings Lexical Forms Morphosyntactic Meanings Morphosyntactic Forms Cohesive Meanings Cohesive Forms Info. Management Meanings Info. Management Forms Interactional Meanings Interactional Forms Sentence Level Discourse Level Language Ability Grammatical Knowledge Grammatical Form Grammatical Meaning

60. Task 3 Item

61. Task 4: Assessing Pragmatic (Implied) Meanings

62. Model of CLA Graphological or Phonological Meanings Graphological or Phonological Forms Lexical Meanings Lexical Forms Morphosyntactic Meanings Morphosyntactic Forms Cohesive Meanings Cohesive Forms Info. Management Meanings Info. Management Forms Interactional Meanings Interactional Forms Sentence Level Discourse Level Language Ability Grammatical Knowledge Grammatical Form Grammatical Meaning

63. Situation: A conversation between a customer & waiter at a restaurant about the menu. Task 4 Item: CEFR Level A1 (contextual, sociolinguistic & psychological meanings) Administered to all levels of the Community English Program .

64. Situation: A conversation between a two close friend . Task 4: CEFR Level B1 (contextual, sociolinguistic & psychological meanings) Circle the best answer.

66. The Listening Section

67. Listening Section Purpose: • To measure a range of literal, intended, and implied(pragmatic) meanings. More specifically… • To measure literal meanings with endophoric reference (“to understand the lines”—literal paraphrase) — The answer explicitly stated in the text.

68. Listening Section • To measure implied meanings with endophoric reference (“to understand between the lines”) — The answer is implied by what is stated in the text. it is implied by or through the text. • To measure implied meanings with exophoric reference (“to understand beyond the lines”) — The answer is neither explicitly nor implicitly stated in the text; rather it is implied by or through the text.

69. Listening for Literal Mng (Endophoric Ref.) “ I think I’ll just have a coffee” = “ The woman want some coffee.”

70. Listening for Implied Meaning (Endo. Ref.) By asking for the building location, she is implying that she wants to go there. The man understands her implied meaning, & gives her the address of the art gallery. The meaning is implicitly stated in the text (i.e., endophoric).

71. Listening for Implied Meaning (Exophoric. Ref.)

72. Listening for Implied Meaning (Exophoric. Ref.) Nothing in the text explicitly states the woman is addressing a group of students. This can be surmised, however, by the type of activities & locations the woman describes and by the shared knowledge that students are often required to take tests when they arrive at a new school. The woman’s implied audience is beyond what is stated in the text (i.e., exophoric reference). Therefore, the focus of the item is on implied contextual or sociocultural meaning, rather than the literal meaning. And the reference to this information is exophoric because meaning can only be derived from outside the spoken text.

73. Questions?