The role of the CAO in a “supportive” accountability framework

1. Welcome!
You can download this presentation at the following address

2. John Cronin, Ph.D.
Vice President of Education Research
NWEA
john.cronin@nwea.org
The role of the CAO in a “supportive”
accountability framework

3. NWEA
Is a not for profit assessment
organization that partners with over
7,000 school systems in the U.S. and
in 30 countries.
We are a leader in computer-
adaptive and formative assessment.

4. During this session you’ll
learn

5. What is the role of the chief
accountability officer?
Job Requirements - Chief Accountability Officer:
While performing the duties of this job, the employee is regularly required
to talk or hear. The employee frequently is required to sit. The employee is
occasionally required to stand and walk. The employee must occasionally lift
and/or move up to 25 pounds. There are no specific vision abilities required
by this job.

6. Accountability is not an inquisition
43:03

7. What is the role of the
accountability officer?
• Create dialogue and action to improve the
effectiveness of schools which centers on
improvement of systems.
• Provide a transparent common set of facts (mastered
by all leaders) that provide the basis for discussing
student achievement.
• Implement structures, policies and procedures to
ensure that data used to evaluate schools has
integrity.

8. Create dialogue and action to
improve the effectiveness of schools
that is focused on improvement of
systems.
1

9. Create dialogue and action to
improve the effectiveness of schools
that is focused on improvement of
systems.
The management letter is the
cornerstone.
1

10. The financial advisor metaphor

11. What is accountability?
To be “answerable” means that accountability is a
dialogue between the stakeholders and the leaders of
their schools. One part of that dialogue is understanding
the goals and objectives of the stakeholders. Another
part of that dialogue is discussing how school leaders are
doing in reaching those goals.
The most important part of the dialogue is what school
leaders are doing to improve performance based on this
information, which is leading.

12. 1 The management letter
Is an opportunity to show leadership.
It is a conversation with stakeholders about the
performance of schools. In it they explain their
business, their successes and failures, and discuss
their intended actions for improving the educational
experience of students.

13. 1 The management letter
Is not simply for the community…
It requires the school leaders to think seriously
about their work and communicate coherently
about the state of their school and their strategy.
It forces them to reflect and clarify their
thinking.

14. The four principles that guide your
communications to your audiences.
• Clearly identify who your audience is.
• Understand what that audience wants to know
about their schools and their performance.
• Be transparent about how you are performing on
these objectives.
• Communicate your strategy to improve
performance, and corrective action when strategy
fails.

15. The tone of the management
letter
• Should be optimistic
• Should be plain yet thoughtful
• Should be conversational
• Should be realistic in addressing successes and
transparent about problems
• Should be effusive in praising of subordinates,
critical of yourself.

16. • Be transparent about failures
• Have a strategy in place to address it
• Accept responsibility and temper expectations
• Demonstrate progress
How to be transparent without getting killed

17. Our bad news from 2014 comes from (Burlington-Northern Santa Fe) and is
unrelated to earnings. During the year, BNSF disappointed many of its
customers. These shippers depend on us, and service failures can badly hurt
their businesses. (transparency about failure)
BNSF is, by far, Berkshire’s most important non-insurance subsidiary and, to
improve its performance, we will spend $6 billion on plant and equipment in
2015. That sum is nearly 50% more than any other railroad has spent in a
single year and is a truly extraordinary amount, whether compared to
revenues, earnings or depreciation charges. (our strategy to address failure)
An illustration of good tone from the
Berkshire Hathaway Management Letter

18. Though weather, which was particularly severe last year, will always cause
railroads a variety of operating problems, our responsibility is to do whatever it
takes to restore our service to industry-leading levels. That can’t be done
overnight: The extensive work required to increase system capacity sometimes
disrupts operations while it is underway. (recognizing responsibility and
tempering expectations)
Recently, however, our outsized expenditures are beginning to show results.
During the last three months, BNSF’s performance metrics have materially
improved from last year’s figures. (reporting progress)
From the Berkshire Hathaway Management
Letter

19. One of our commitments is to help more students graduate prepared for college.
Advance Placement is a key piece of this strategy and our goals were to increase
participation in the program, increase the number of students sitting for AP tests,
and improving average scores on AP exams.
Our data show that we are doing well on two of the three. In the past three years
we’ve increased the number of students participating in the program by 20% at
Jefferson high school, and the average scores on the exams have generally
improved. In English for example, the average score improved from 3.5 to 3.7 and
In Calculus AB the average score improved from 3.1 to 3.5. That’s impressive
progress and all credit goes to Marsha White, who led this initiative and the
teachers and Jefferson high schools who made this happen. (crediting others for
success)
Unfortunately, the storyline isn’t perfect. While more students want to take
courses, we cut the number of available courses and the improvement the
improvement in the number of students sitting for the tests reversed last year
(Admit failure)
How this would read in a school
management letter

20. You may be aware of budget pressures that have impacted Jefferson, and because of
retirements and transfers at some of the high schools, we did not replace AP
teachers who retired or transferred and those courses were dropped. I should’ve
seen this problem coming and planned for it (accepting responsibility and tempering
expectations).
This year we reviewed teacher assignments and are reinstituting courses in areas
where we have a teacher with appropriate credentials. Because those teachers will
be dealing with new content and larger class sizes, we’re funding summer training for
them in both the AP content and in teaching strategies that better enable adaption
of instruction with larger student groups (demonstrating progress).
We can’t promise that this will entirely solve the problem, and our pass rate on AP
exams may take a hit while our new AP teachers master their assignment. I’ll keep
you apprised in my monthly newsletter as to how this is going.
How this would read in a school
management letter

21. What is the role of the
accountability officer?
• Create dialogue and action to improve the
effectiveness of schools which centers on
improvement of systems.
• Provide a transparent common set of facts (mastered
by all leaders) that provide the basis for discussing
student achievement.
• Implement structures, policies and procedures to
ensure that data used to evaluate schools has
integrity.

22. Provide a transparent common set of
facts (mastered by all leaders) that
provide the basis for discussing
student achievement.
2

23. Provide a transparent common set of
facts (mastered by all leaders) that
provide the basis for discussing
student achievement.
The report of learning indicators
serves this purpose.
2

24. Everyone is entitled to their opinion, but
not to their own facts.
Daniel Patrick Moynihan

25. Your annual report should be your “almanac”
when it comes to student achievement facts.
Any member of your leadership team who
represents educational data should be
thoroughly familiar with it.
In any data discussion, the discussion should
begin with agreement on what the facts actually
are.

26. Four types of metrics:
Achievement – How are students learning?
Growth – How much progress do students make
within a year?
Improvement – Are we improving achievement
and/or rate of growth of students over time?
Acceleration – Is the rate of change
accelerating, decelerating, going negative?

27. 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013 2014 2015 2016
Percent of students proficient in math
Achievement in mathematics for the current past
year shows slightly more than half of students are
proficient.

28. 35% 37%
41%
46%
52% 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013 2014 2015 2016
Percent of students proficient in math
Achievement in mathematics showed an
improving pattern through 2013 and
improvement actually accelerated during that
time period.

29. 35% 37%
41%
46%
52% 53% 54%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013 2014 2015 2016
Percent of students proficient in math
Achievement in mathematics improved but
decelerated between 2013 and 2015. Sometimes
this can be a leading indicator of trouble ahead.

30. 35% 37%
41%
46%
52% 53% 54%
43%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013 2014 2015 2016
Percent of students proficient in math
Achievement in mathematics declined
dramatically in 2016.

31. The choice of metrics matters. Poorly
chosen metrics can distort reality.

32. Which school achieved the best growth
in mathematics?
School Mean years of growth
M.L. King Elementary 1.34
Hamilton Middle School 1.46
Armstrong High School 2.14

33. 0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
AverageScaleScoreGrowthbySchool
Grade in School
Average Scale Score Growth in Mathematics by
School
The bars represent the median growth for the NWEA
norm group by school. When statisticians refer to a
“year of growth” the median scale score growth is the
common definition of a year. As you can see, what
constitutes a “year of growth” by that definition varies
immensely by grade

34. 0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
AverageScaleScoreGrowthbySchool
Grade in School
Average Scale Score Growth in Mathematics by
School
The lines represent the standard deviation of growth,
by grade, across the schools in the NWEA norming
group. The standard deviations are relatively even
across grades, which means that a school showing
growth that is 1 standard deviation above other
schools achieves roughly 2 to 2.5 scale score points
greater growth than the median school.

35. 0.0
0.5
1.0
1.5
2.0
2.5
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
YearsofGrowth
ScaleScoreGrowth
Average Scale Score Growth in Mathematics by
School
The line graph depicts the “years of growth”
associated with a school that performs 1
standard deviation better than the median school
for each grade. In grades 1-3 being 1 SD better
translates to about 1.3 “years of growth”. In
grades 8-10 it translates to 1.5 to 2 years of
growth. Given an elementary school and a high
school performing at the same level relative to
their peers, the high school will look better if

36. 0.00
0.50
1.00
1.50
2.00
2.50
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
ScaleScoreGrowth
Average Scale Score Growth in Reading by
School
The line graph depicts the “years of growth”
associated with a school that performs 1
standard deviation better than the median school
for each grade. In grades 1-3 being 1 SD better
translates to about 1.3 “years of growth”. In
grades 8-10 it translates to 1.5 to 2 years of
growth. Given an elementary school and a high
school performing at the same level relative to
their peers, the high school will look better if

37. 0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
-5
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
YearsofGrowth
ScaleScoreGrowthinReading
Average Scale Score Growth in Reading by
School

38. All students should be in
play relative to a chosen
metric.

39. Fifth grade math achievement in one
school system
NUMBEROFSTUDENTS
FALL RIT
Mathematics
No Change Down Up

40. NUMBEROFSTUDENTS
STUDENT’S SCORE IN FALL
NUMBER OF 5TH GRADE STUDENTS MEETING PROJECTED GROWTH IN THE SAME DISTRICT
Mathematics
Below projected growth Met or above projected growth
Fifth grade math achievement
in one school system

41. It’s important to have a
“racing form” for student
achievement

42. Serena is “one year” behind
proficiency for her grade
1.5
Her goal is to attain
years of growth this year.

43. 45%
43%
39%
34%
31%
26%
21%
18%
16%
12%
9%
55%
57%
61%
66%
69%
74%
79%
82%
84%
88%
91%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
10
9
8
7
6
5
4
3
2
1
K
Proportion of students meeting 1.5 years of growth in
mathematics
Met 1.5 years Did not meet
Students start at 50th percentile of 2015 norms

44. 13%
9%
2%
1%
1%
1%
1%
1%
1%
1%
1%
87%
91%
98%
99%
99%
99%
99%
99%
99%
99%
99%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
10
9
8
7
6
5
4
3
2
1
K
Proportion of schools averaging 1.5 years of growth in
mathematics
Met 1.5 years Did not meet
Students start at 50th percentile of 2015 norms

45. 44%
42%
38%
34%
30%
25%
21%
17%
14%
11%
6%
56%
58%
62%
66%
70%
75%
79%
83%
86%
89%
94%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
10
9
8
7
6
5
4
3
2
1
K
Proportion of low performing students meeting 1.5 years of
growth in mathematics
Met 1.5 years Did not meet
Students start at 20th percentile of 2015 school norms

46. 9%
6%
1%
1%
1%
1%
1%
1%
1%
1%
1%
91%
94%
99%
99%
99%
99%
99%
99%
99%
99%
99%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
10
9
8
7
6
5
4
3
2
1
K
Proportion of schools with low performing students averaging
1.5 years of growth in mathematics
Met 1.5 years Did not meet
Students start at 20th percentile of 2015 school norms

47. Implement structures, policies and
procedures to ensure that data used
to evaluate schools has integrity.3

48. In baseball, the goal of the general
manager is to hire players who all
hit .300. In reality, very few players
meet this aspirational goal.

49. What happens when
educators are held
accountable to unreasonable
improvement goals?

50. They cheat

51. Morale collapses

52. They focus on
metrics rather
than learning

53. The lesson from the Atlanta cheating scandal

54. Source: Aviv, R (2014, July 21). Wrong Answer. The New Yorker. Retrieved on June 16, 2016
from http://www.newyorker.com/magazine/2014/07/21/wrong-answer
“After more than two thousand interviews,
the investigators concluded that forty-four
schools had cheated and that a “culture of
fear, intimidation and retaliation has
infested the district, allowing cheating—at
all levels—to go unchecked for years.”
They wrote that data had been “used as an
abusive and cruel weapon to embarrass
and punish.”

55. Source: Aviv, R (2014, July 21). Wrong Answer. The New Yorker. Retrieved on June 16, 2016
from http://www.newyorker.com/magazine/2014/07/21/wrong-answer
“Several teachers had been told that they
had a choice: either make targets or be
placed on a Performance Development
Plan, which was often a precursor to
termination. At one elementary school,
during a faculty meeting, a principal
forced a teacher whose students had
tested poorly to crawl under the table.”

56. Differences in fall-spring test durations
An illustration of gaming
15%
25%
60%
Mathematics
Spring < Fall Spring = Fall Spring > Fall
0.0
1.0
2.0
3.0
4.0
5.0
6.0
Spring < Fall Spring = Fall Spring > Fall
GrowthIndex
Mathematics
Differences in growth index score
based on fall-spring test durations

57. What constitutes a reasonable
school improvement goal?

58. What makes a school improvement
goal realistic?
• What is the past track record of the
school?
• What is the range of improvement that
is normal for schools like this?
• What is the level of control that the
principal has over the conditions of
improvement?

59. Three takeaways

60. Accountability is a dialogue…

61. …not an inquisition

62. Metrics matter

63. Accountability isn’t about the absence of failure.
Failure offers an opportunity to learn

64. Thank you!
E-mail: john.cronin@nwea.org
Organization Website: www.nwea.org