The Center for Applied Linguistics is moderating the Test Developers Panel Discussion with several assessments represented including ACT, CASAS, TABE, and BEST Plus and BEST Literacy.
Designing multi-device learning practice with Adobe Captivate
Test Developers Panel Discussion (2010) Full
1. Adult ESL Assessment
Test Developers’ Panel Discussion
Accountability and Standards
Saturday, March 27, 2010 10:00 AM – 11:45 AM
Boston Convention Center Room 157A
Moderator:
Deborah Kennedy
Director of Development
Center for Applied Linguistics
2. 2
Session Purpose and Goals
The purpose of this session is to provide participants with an
overview of ESL assessments currently being used in post-
secondary and adult education programs in the United
States
By the end of the session, participants will have an
understanding of
The tests available from each test publisher
What each test measures and for which student
population(s)
Administration and test security requirements
Program implementation considerations, including training
requirements and costs of materials
3. Presenters
John Downs, Test Development Associate, ACT
COMPASS ESL Placement Test
Linda Taylor, Director of Assessment Development, CASAS
Comprehensive Adult Student Assessment System (CASAS)
Enrique Pilleux, ELL Solutions Specialist, CTB McGraw-Hill TABE
Complete Language Assessment System – English (CLAS-E)
Michelle Ueland, Adult ESL Specialist, CAL
BEST Plus and BEST Literacy
3
4. Session Format
Each presentation is 20 minutes.
15 minutes to present
5 minutes for questions
After all presentations, there will be 20 minutes for discussion and
additional questions for specific panelists.
4
5. COMPASS
ESL Placement Test
Today’s presenter
John Downs (john.downs@act.org)
ACT Placement Programs
ACT National Office
Iowa City, IA
319/337-1617
6. ESL Placement Test
ESL is integrated into ACT’s
COMPASS System
Windows 3.2 Version
and
Internet Version
7. Listening
Test components:
Reading Grammar
and Usage
These are objective, computer-adaptive, multiple-choice
tests that can be administered as often as desired: For
initial placement, mid-course progress, and post-testing.
Student scores are tied to narrative proficiency
descriptors. The test is un-timed.
8. Test Components
New Feature: ESL e-Write Direct
Writing Assessment with Instant
Scoring (5 Analytic Scores +
Overall)
Students have 60 minutes to respond to a writing
prompt. Instant scores are based on a six-point
rubric. To derive the overall score, the analytic
scores are weighted as follows: Development
(35%), Focus (10%), Organization (15%),
Language Use (35%), and Mechanics (5%).
9. Test Features
• Developed for postsecondary ESL
students:
• Community Colleges*
• Four-year institutions
• Intensive English Programs
*Approximately 70% of COMPASS/ESL users
are community colleges or other two-year
institutions.
10. Test Features
• The test measures a wide range of student
proficiency–from near-beginner to near
native-speaker proficiency. This range
makes the COMPASS/ESL Test
appropriate for:
• Placement testing
• Progress reporting
• Post testing
11. Overview of test questions
COMPASS
Math
Reading
Skills
Writing Skills ESL e-Write English as a
Second
Language
(ESL)
Pre-algebra
351
54 passages 453 questions 3 initial essay
topics
Grammar/Usage
207
Algebra
269
243 questions Reading
208
College Algebra
163
Listening
190
Geometry
186
* 3.2 will include
20 more reading
passages and 100
more questions
Trigonometry
151
Math total: 1,130
ESL Total:
605 Items
12. Test Time
• Average student test-taking time:
un-timed test (results will vary)
• 15-20 minutes per test
• 5-15 minutes for tutorials and
demographic questions
• 1:00 - 1:15 total
• 60 minutes for ESL e-Write
13. COMPASS Advantages
• Untimed, computer-delivered format
• Immediate on-site scores for
students
• Computer-adaptive testing for
increased accuracy
• Easy data management
14. COMPASS Advantages
• Post-test capability
• Outreach capability
• Immediate advising with course
placement recommendations. These
can be customized locally.
15. COMPASS/ESL
• ESL Tests Approved for Ability to
Benefit Use for Students in ESL
Programs and Services
• Minimum ESL ATB Qualifying
Scores
– Grammar/Usage 64
– Reading 70
– Listening 70
16. COMPASS/ESL Investment:
• No annual site license fee
• $1.66 for each unit (volume discounts are
available)
• .4 unit for student demographic
information
• $5.64 per student (or less) if the student
takes the full set of three objective tests
and demographic information
• ESL e-Write uses 3.5 Units (about $5.81)
17. Next Steps
• Review the operational version of
COMPASS/ESL (no charge) - faculty and
student review
• ESL - www.act.org/esl
• Proficiency Descriptors
• Sample Questions
• ATB Requirements
• e-Write - www.act.org/e-write
• COMPASS – www.act.org/compass
19. CASAS ESL Assessment History
Non-profit organization started in 1980 to develop assessment
for CA adult education programs
Approved by US Dept. of Ed for National Diffusion Network
in mid-80s
Approved for NRS for Reading, Listening, Writing (and
Math)
• CASAS Reading and Writing assessments are the only ones
approved by NRS for both ESL and ABE
CASAS National Consortium
• State directors and trainers from 30 states
• Determine field-based development priorities
• Participate in development and field-testing
• Meet annually in June in San Diego at CASAS Summer Institute
20. Integrated System Approach
CASAS Competencies
Basic Skills Content Standards
• Reading and Listening available
• Writing and Speaking – draft versions for review
Assessment Tools
Paper and computer-delivered CASAS eTests
• Reading and Listening – multiple-choice
• Writing and Speaking – performance-based
Instructional Materials QuickSearch Online
TOPSpro software
• Automatically scores tests
• Produces diagnostic student and class level reports
• Generates program and state level reports
• Generates NRS Federal Tables
21. CASAS National Scale
Equal interval
scale
used to determine
• learner skill
levels
• learner gains
• transition to
ABE
Accurate from
beginning literacy
to advanced ESL
22. About CASAS ESL Assessments
Measure academic skills in functional
life and work contexts
Group or individual
administration is practical for large
and small programs
Re-usable test booklets are cost-
effective
Reading and Writing tests can be
used in both ABE and ESL
programs
Most assessments available in paper
and computer-delivered formats
Diagnostic reports inform instruction
23. CASAS ESL Assessments
Reading
• Life and Work
• Employability (ECS) & Workplace
• Beginning Literacy
Listening
• Life and Work
• Life Skills
• Employability (ECS)
Writing
• Functional Writing Assessment –
Picture Task
Speaking
• Workplace Speaking
• Citizenship Interview Test
Available on CASAS eTests
24. Testing Process for Reading and Listening
Tests
Intake, including initial screening and pretesting
Reading
• eTests: Short locator and pretest (65 minutes)
• Paper: Appraisal and pretest (85 minutes)
Listening
• eTests: Short locator and pretest (45 minutes)
• Paper: Appraisal and pretest (65 minutes)
Post-testing
• Administer post-test(s) after approx. 70 -120 hours of instruction
All tests scored locally
• manually or with TOPSpro or eTests
Sample Test Items for test preparation
• available at no cost on CASAS website and in CASAS eTests
25. Life and Work Reading – 80 Series
Appropriate for all NRS levels
CASAS test Levels A – D
Replaces Life Skills Reading series
Features story lines about adults from diverse backgrounds at
Levels A and B
Includes narrative reading passages
Developed based on both Reading Content Standards and CASAS
competencies
Available via computer in CASAS eTests
26. Life and Work Listening – 80 Series
Appropriate for all NRS ESL levels
CASAS test levels A – C
Revised listening series based on latest research
Available via computer in CASAS eTests
New Features
• Updated content and contexts, and more natural speech
• Developed based on both content standards and competencies
• No written response (answer) options – all aural
• Picture item type and test booklet at Level A only
• Only three item types per level
• Repetition at Level A and in first two item types of Level B
• Preview questions introduce the context of each question
28. More than 300 Agencies in 25 states are now using it
Standardizes and improves assessment practices
Most CASAS test series are on CASAS eTests
• All tests are on one CD
• Use CASAS eTests units (CTUs) for any test
• Administrations (CTUs) in increments of 50, 100, 500, or 1,000
• Very affordable (75 cents per test) with
bulk purchase
Better management of test security than
paper-delivered tests – with a “dongle”
(a USB thumb drive)
• Also serves as a test use counter
CASAS eTests – Computer-delivered
testing
28
29. Common CASAS e-Tests Scenario
1) Examinee can take a Practice Test
2) Examinee takes quick Locator (no charge) and continues
immediately into appropriate pretest
3) CASAS eTests scores and reports pretest score
4) Exports to TOPSpro for diagnostic reports
5) For post-test, CASAS e-Tests
automatically selects post-test form
CASAS eTests Sampler – on website
CASAS eTests
29
30. No need for test booklets, answer sheets or #2
pencils
• Easier to manage
• Reduces training of test administrators
• Improves test security
Automatically scores tests
• No staff time required to scan or score
• Reduces human scoring errors
Can be given individually or to a group
• In a computer lab or on a laptop
• Can be used only for post-testing
• Touch Screens – use with any test in CASAS eTests
Benefits of CASAS eTests
30
31. CASAS Functional Writing Assessment
Picture Task
• Assesses Beginning to Advanced narrative writing for ESL and
ABE learners
• Analytical scoring rubric, guidelines and anchors
• Diagnostic reporting in Content, Organization, Word Choice,
Grammar/Sentence Structure, and Spelling/Capitalization/
Punctuation
• Predicts GED essay performance
• Used to report NRS levels in 10 states:
CO, CT, DC, IN, IA, KS, MN, NV, OR, VT
• Colored prompts available
Picture Task in CASAS eTests– field testing now
Computer-delivered Written Prompt (Essay) Task
• field testing now
32. Using CASAS for Transition to
Training and Post-Secondary Education
Used in many states for transition
• Washington state iBEST program (transition to ESL
combined with post-secondary training)
• Other states: OR, CT, KS
CASAS Secondary Level Reading Assessments
• Forms 513 and 514
• Contain more academic reading content
• Can also use Level D Reading assessments
• Correlated to CASAS Reading Content Standards for
diagnostic reporting
CASAS Writing Assessment
• Important skill for success in post-secondary education
33. CASAS Assessments for Low Literacy ESL
Learners
CASAS is developing computer-delivered touch screen
literacy assessments for low literate ESL learners, with
funding from the Florida Department of Education
Assesses listening and reading
Includes native language locator process and short oral
screening
Working with a Florida-based and a national advisory
group
Field testing now – contact CASAS to participate
Currently available: Paper-delivered consumable low
literacy reading assessments – Forms 27 and 28 (NRS
approved)
34. CASAS Citizenship Assessments
Citizenship Interview Test
• Assesses speaking skills of adult ESL/Citizenship learners in a
simulated CIS oral interview
• Determines readiness to pass naturalization English interview
• Consistent with current US CIS test content
Government and History for Citizenship
• Measures knowledge of government, history and civics
• Revised version with new test content available
• Contains reading and writing dictation sections with detailed
scoring guidelines, revised according to new USCIS rubrics and
word lists
Reading for Citizenship, Level A
• Reading comprehension in citizenship context
• Used for NRS pre- and post-testing of beginning level learners
35. CASAS Speaking Assessments
Workplace Speaking
• Assesses High Beginning to Low Advanced functional workplace
speaking skills
• Simple 3-point scoring rubric
• Scoring guidelines and anchors
• Uses scale scores to report ESL levels
Citizenship Interview Test– see Citizenship Assessments
In development
General Proficiency beginning and advanced level speaking
tests
Computer-delivered videotaped prompts and remote scoring
36. QuickSearch Online
Available at no cost
Helps select materials that match
curriculum
Provides quick, easy access to a
database of more than 2,000
instructional materials
Is correlated to more than 300
CASAS Competencies
Includes print, audio, video, and
software materials
Is updated continuously
Online version—available since
June 2008
QuickSearch software provides an essential link between
assessment and instruction in the CASAS system
37. Initial Implementation
Select skill area(s) to be assessed (e.g., reading, listening,
writing, and/or speaking and math)
Match students’ goals to appropriate CASAS test series
Identify pilot implementation teachers and classes
Select professional development option:
• Find out if free training is available in your state
• Distance Training – call for options
• New self-paced Moodle courses – included with
implementation
• Certified CASAS Trainer goes to your agency
• CASAS Summer Institute
• June 17 - 19, 2010 in San Diego
39. Background
• CTB Founded 1924 – To Help the Teacher Help the Student
• Published the first criterion-referenced tests
• Published the first version on TABE in 1964
• Introduced the use of IRT methodologies
• Published the first test incorporating both multiple-choice and
constructed response items
• Developed the Bookmark Standard Setting method: the most
advanced approach to standard setting
• First release of our online assessment system in 2001
40. Importance of Adult ESL
According to a report from the U.S. Dept of Education,
there are
► 1.3 million adult ESL students enrolled yearly
► Approximately 90 Million adults without a High School
diploma or GED
► An estimated 44% are non-Native English speaking
41. 41
Overview of TABE CLAS-E
• Standardized assessment that provides teachers with a
reliable, easy-to-use system that accurately identifies the
instructional needs of their ESL students
• Aligned with the NRS ESL Educational Functioning Level
Descriptors, Student Performance Levels, and several
state standards: FL, TX, CA, NY, AZ, MD, MA
• Aligned to Common European Frameworks and Canadian
Standards
• Scores from TABE CLAS-E are linked to TABE 9&10,
designed to offer your students a successful transition into
adult basic education
42. Key Benefits of TABE CLAS-E
• Reliable and valid;
– Received full 7 year NRS re-approval in Feb. 2010
• Assesses four skill areas
– Reading, Listening, Writing, Speaking
• Aligns to NRS ESL Educational Functioning Level Descriptors
• Appropriate for all levels of Adult ESL
• Seamless transition to TABE 9&10
• Correlated to key instructional programs such as All-Star and
Excellent English (McGraw-Hill ESL/ELT)
43. Testing with TABE CLAS-E
• Step 1: Locator Interview
• Step 2: Administer Locator test (Part 1 and 2)
• Step 3: Administer appropriate Level of CLAS-E for a pre-test
• Step 4: Score CLAS-E Level to find out ESL Proficiency Level
• Step 5: Place student in appropriate ESL class
• Step 6: Administer the alternate form of CLAS-E as the post-test
44. 44
Overview of TABE CLAS-E
Content Area Total Items
Testing Time
(Minutes)
Reading 25 25
Listening 25 20
Writing 20 20
Expository Writing Folio 5 27
Total for group administration R, W, L 75 92
Total Items Testing Time
Speaking 16 15
45. 45
Reading
• Uses schedules, forms, flyers, store receipts,
advertisements, product instructions, and newsletters
• Lower literacy examinees answer discrete, picture-based
items
• Assessment objectives include quantitative literacy, read
words, and reading comprehension
46.
47. 47
Writing
• Selected-response items:
– Gauge knowledge of English language grammar and mechanics
– Test quantitative literacy, usage and conventions, and sentence
formation and coherence
• Constructed response items:
– Measure application of grammar and mechanics in expository
writing
– Are scored with holistic rubrics to assess expository writing ability
48.
49. 49
Listening
• Presents items in a spoken format of varying lengths, from pairs to
short passages
• Examinees are addressed in meaningful work, community, and
academic situations
• Quantitative literacy, listening comprehension, and interpersonal skills
are measured
• Lower level examinees are tested in discerning beginning and ending
sounds of words
50.
51. 51
Speaking
• Introduces an innovative multiple-criteria scoring rubric
– Measures grammar, meaning, and appropriateness
– Scale scores will be averaged for the composite Speaking score
• Individually administered by a proctor (10 minutes)
• Uses picture-based stimuli
• Work and life skills
– Phone competency
– Expressing opinions and abilities
– Providing descriptions and instructions
• Includes critical thinking
52.
53. 53
Scoring Options
• Hand-scoring with stencil
– Individual Diagnostic Profiles for Objective level data
• Hand-scoring with SCOREZE
• TestMate®
TABE – scan answer sheet for immediate reports
– Student Individual Report
– Item Analysis Report
– Student Pre-Post Report
– Student Prescriptive Report (to McGraw-Hill All Star)
– Rank List Report
– Group List Report
– Assessment Summary Report
55. 55
Instructional Guidance
Teacher’s Resource Guide, Volume 2:
• Classroom Tasks and Lessons
Three basic themes:
80 Classroom Tasks (Community & Work related)
20 Lesson Plans Work tasks
12 Education tasks: tied to one of four skill areas
– Lesson Plans
– Teaching Tips
– Checklists
56. Professional Development
- Staff Development Workbook with DVD
- On-Site Training or WebEx Training
– Explanatory and descriptive material about the assessment
– Examples of test items and score reports
– Extensive exercises in scoring with the rubrics
– How to interpret and apply test results to enhance classroom instruction and
student learning
57. Linking Study for TABE CLAS-E and TABE 9&10
• Goal: To establish a relationship between TABE CLAS-E and TABE
9&10 and to provide a predicted scale score for ESL students on the
TABE 9&10 Level M Reading and Language tests.
• The Study: 925 examinees in 22 locations across the country were
administered both TABE and CLAS-E.
• A univariate ordinary least-squares regression method was used to
estimate a linear relationship between CLAS-E and TABE.
• Results: Include an estimated TABE Reading and Language Scale
Score and also a Standard Error Estimate Range.
58. 58
Why TABE CLAS-E is Unique
• TABE’s 45 year reputation of high quality assessments
• Integration with top instructional resources
• Industry leading item development, research, and
psychometric processes (reliability & validity)
• Complete assessment system with assessments,
diagnostic reports, and teacher guidance
• Correlation/Linking to TABE 9& 10
• Introduces students to testing formats used in ABE and
GED programs
60. English as a Second Language
(ESL) Assessment for Adults
Presented by:
Michelle M. Ueland, Ph.D.
Adult ESL Specialist
Center for Applied Linguistics
61. About CAL
The Center for Applied Linguistics (CAL) is a non-profit organization
founded in 1959. We celebrated our 50th
anniversary in 2009.
CAL staff have many years of experience in working with adult English
language learners and designing ESL tests for adults.
CAL’s Language Testing Division specializes in the development of English and
foreign language proficiency tests.
The Basic English Skills Test (BEST) has been one of the most widely used
assessments in the United States since the 1980s.
CAL developed BEST Plus from 1999 to 2004, using rigorous test development
procedures to ensure the validity and reliability of the test. Funding was
provided by the U.S. Department of Education’s Office of Vocational and Adult
Education (OVAE).
62. Uses for BEST Literacy and BEST Plus
Determine student language proficiency
upon entry into a programPLACEMENTPLACEMENT
PROGRESSPROGRESS
PROGRAM
EVALUATION
PROGRAM
EVALUATION
DIAGNOSISDIAGNOSIS
Track the progress of individual learners
Evaluate program effectiveness when
used as a pre- and post-test
Provide feedback to learners and
instructors to improve instruction in
understanding and using written and
spoken English
Report outcomes to comply with the U.S.
Department of Education’s National
Reporting System (NRS) for accountability
purposes
REPORTINGREPORTING
63. Reliability and Validity
▪ BEST Literacy reliability and validity were assessed in studies
undertaken in 1984 and again in 2006.
▪ High internal reliability estimates show consistency between 3 forms.
▪ Interrater reliability was .884 to .989.
▪ Validity evidence provided by quantitative analyses validity evidence
supports the use of the test to measure survival-level competency of
adult English language learners.
▪ BEST Plus underwent rigorous test development procedures
beginning with a feasibility study in 1999 and ending with a full-
scale field test, reliability study, and standard-setting study in 2002.
▪ Average interrater reliability was .97.
▪ Test/retest reliability was .89.
More detailed information is contained in the BEST Literacy Test Manual and the BEST Plus and BEST
Literacy Technical Reports.
64. About BEST Literacy Updated in 2006
Assesses adult English language
learners at survival and pre-
employment levels
Emulates real-life reading and
writing tasks
Three parallel forms – B, C and D
Scores aligned with Student
Performance Levels (SPL) 0 to 7
and National Reporting System
(NRS) levels
Test manual guides proctors and
scorers - no additional training
required
65. Functional oral language assessment of
interpersonal communication skills of
adult English language learners
Available in two formats
- Computer-adaptive
- Semi-adaptive, print version
Computer-adaptive by its nature provides
a unique test every time
Semi-adaptive, print version has three
parallel forms (A, B, and C). Within each
form there is a locator and three levels
within the booklet. Requires one reusable
full-color picture cue book. Scores must
be entered into a computer to determine
the scale score.
BEST Plus is a face-to-face oral interview
administered by a trained test administrator.
About BEST Plus
66. BEST Plus Scoring Rubric
Assessing What Students Can Do in English
Measures how well the
student understands spoken
English
Listening ComprehensionListening Comprehension
Language ComplexityLanguage Complexity
CommunicationCommunication
Measures the depth and
breadth of the student’s use
of conversational English
Measures how well the
student can be understood
when speaking in English
67. BEST Plus computer-adaptive version
BEST Plus
software
customizes the
test based on each
examinee’s
language ability –
students do not
take the same test
twice.
The test takes 3
-20 minutes to
administer
depending on the
language
proficiency of the
examinee.
Scores are entered directly into the computer. The computer then updates its estimate of the
examinee’s ability and chooses the next folder and questions based on this estimate.
68. Score Management Software
Individual score reports show
a BEST Plus scale score and
a Student Performance Level
(SPL).
Each score report includes a
subscore for Listening
Comprehension, Language
Complexity, and
Communication.
BEST Plus scores are aligned
with the National Reporting
System (NRS) ESL
Functioning Levels. The Score Management Software provided with the
test allows administrators to view and print score
reports and transfer test data between databases.
69. BEST Literacy and BEST Plus Costs
▪ BEST Literacy
▪ Packets of 20 tests with Scoring Sheets $45.00
▪ Test Manual $25.00
▪ Shipping and Handling standard rates
▪ BEST Plus
▪ Test Administrations $1.00 to $1.50 per student
▪ Test Administrator Guide $30.00 (included in training)
▪ Picture Cue Booklet $15.00 (only for print-based)
▪ Shipping and Handling standard rates (applies to
print-based materials only)
70. Getting Started
BEST Literacy proctoring and scoring instructions are included in
the Test Manual. Training is not required.
BEST Plus begins with a six-hour Test Administrator Training to
ensure effective test administration and reliable scoring.
• Test administrators need strong English speaking and comprehension skills
but do not need to be native speakers of English.
• Test administrators benefit from experience working with adult English
language learners (ELLs).
• Test administrators benefit from knowledge of language teaching concepts
and vocabulary.
Trainer-in-Training Program certifies state trainers, thereby building
in-state capacity to deliver BEST Plus test administrator training.
71. BEST Plus Scoring Refresher Toolkit
▪ Available to refresh and
recalibrate trained test
administrators’ scoring
▪ Implementation guidelines for
program administrators
▪ Facilitator materials
▪ Training video/DVD with 50+
benchmark samples
▪ Separate video with scoring
activities
▪ Answer keys and instructions
for determining scoring
accuracy
▪ Test Administrator and
Facilitator’s Edition Workbooks
72. New Adult ESL Assessment Web Site
▪ Visit www.cal.org/aea to
navigate our new Web site
▪ Features a new framework for
current and prospective users
of BEST Plus and BEST
Literacy
▪ Access Frequently Asked
Questions about our
assessments
▪ Learn more about other CAL
Resources
73. Manager, Adult ESL
Assessment: Frank Finamore
ffinamore@cal.org
202-355-1586
Web: www.cal.org/aea
Phone: 1-866-845-BEST (2378)
Email: aea@cal.org
Write: Center for Applied Linguistics
4646 40th Street, N.W.
Washington D.C. 20016-1859
Contact Adult ESL Assessment at CAL
▪ Web www.cal.org
▪ Online store www.calstore.cal.org
▪ Phone (202) 362-0700 toll free at 1-866-845-2378
▪ Fax (202) 363-7204 toll free at 1-888-700-3629
Contact CAL