1. Mission Skills Assessment
Assessing Character Traits in Middle School Students
Independent Schools Data Exchange
1
INDEX Mission Skills Assessment
www.indexgroups.org
617-413-8382

2. Mission Skills Assessment
Mission Skills Assessment: A Tool To Alter
The Way Schools Think About Education
MSA measures six character strengths—Teamwork, Creativity, Ethics,
Resilience, Curiosity, Time Management—that have proven essential
for success in school and in life. The assessment gives each school a more
scientific way to measure its curriculum’s success at meeting the goals
outlined in its mission.
INDEX Mission Skills Assessment 2

3. Who We Are
• INDEX Elementary Schools: National benchmarking and
best practices group of 28 Independent PS – 8/9 co-
educational day schools with 400+ students.
• Center for Academic and Workforce Readiness and Success,
Educational Testing Services (ETS).
INDEX Mission Skills Assessment 3

4. MSA Schools
Duke School (NC) Old Trail School (OH)
Episcopal Day School (GA) The Pike School (MA)
Far Hills Country Day School (NJ) Rippowam Cisqua School (NY)
The Foote School (CT) Rodeph Sholom School (NY)
Grace-Saint Luke’s Episcopal School (TN) Rumson Country Day School (NJ)
Green Vale School (NY) Saint Martin’s Episcopal School (GA)
Greenwich Country Day School (CT) Shady Hill School (MA)
The Lexington School (KY) Shore Country Day School (MA)
Marin Country Day School (CA) St. Patrick’s Episcopal School (DC)
New Canaan Country School (CT) Trinity Episcopal School (NC)
INDEX Mission Skills Assessment 4

5. What is the Mission Skills Assessment
The MSA is a longitudinal assessment measuring and
benchmarking student achievement and improvement in core
mission skills.
• Teamwork
• Creativity
• Ethics
• Resilience
• Curiosity
• Time Management
INDEX Mission Skills Assessment 5

6. Roadmap
• R&D • Cognitive Labs • Fall Test • Fall Test • Fall Test
• ETS • Pilot Test • Spring Test • 2nd Report • INDEX Schools
• Constructs • Horizons • 3rd Report
• 1st Report
INDEX Mission Skills Assessment 6

7. The Assessment
• Web-based.
• Student self-assessment
• 2 30-minute tests for approximately 60 minutes total.
• Assess once per year.
• SJT and other student-completed performance measures.
• 6th, 7th, and 8th graders.
• The test is fun! There are no wrong answers.
• Teacher-rater assessment
• Teacher rates each student individually
• Outcome data (e.g. test scores, grades, absences, etc.)
• Institutional focus. No tracking of individual performance.
INDEX Mission Skills Assessment 7

8. Noncognitive Skills Research
INDEX Mission Skills Assessment 8

9. What are Noncognitive skills?
• Not traditionally “tested” by standardized
assessments
• Not new but newly important; labeled “21st century
skills”
• Essential capacities necessary for success in school
and in life
• Often taught implicitly, but can be taught explicitly
INDEX Mission Skills Assessment 9

10. Social &
Attitudes & Habits &
Emotional Personality
Beliefs Processes
Qualities
Attitudes Emotion Time Openness to
(evaluations) Management Management Experience
Intelligence
Test Anxiety Goal Setting Conscientiousness
Theories
Test-taking
Self-Efficacy Self-Regulation Extraversion
Strategies
Coping with
Self-Concept Study Skills Agreeableness
Stress
Emotional
Confidence Teamwork Self-Monitoring
Stability
Lipnevich, MacCann, & Roberts (2011)
INDEX Mission Skills Assessment 10

11. Disposition Toward School
GPA Class Discipline Infirmary
Absences Visits
SSAT .27 -.04 -.05 -.04
Conscientiousness .21 -.31 -.23 -.22
Teacher ratings .38 -.35 -.42 -.26
(noncognitive)
INDEX Mission Skills Assessment 11

12. Grades
rho
Primary Secondary Tertiary
Conscientiousness .28 .21 .23
Openness .24 .12 .07
Agreeableness .30 .05 .06
Emotional Stability .20 .01 -.01
Extraversion .18 -.01 -.03
Cognitive ability .58 .24 .23
Poropat, A. (2009). Psychological Bulletin
INDEX Mission Skills Assessment 12

13. Academic Success
Noncognitive skills correlate positively
with achievement on NAEP and PISA.
Noncognitive factors predict college grades
as strongly as cognitive variables do.
ETS, (2008); Poropat, (2009)
INDEX Mission Skills Assessment 13

14. Life Trajectory
Early childhood interventions do not
raise IQ, but improve noncognitive
skills which affect education,
employment, earnings, and crime.
Heckman et al. (2010)
INDEX Mission Skills Assessment 14

15. Life Satisfaction
Noncognitive variables…have been
demonstrated to predict happiness,
health, marital satisfaction, and peer
relationships.
Diener & Lucas (1999); Bogg & Roberts (2004); Watson, Hubbard, & Wiese, (2000); Jensen-
Campbell et al. (2002)
INDEX Mission Skills Assessment 15

16. Demographics
The benefits of higher noncognitive skills
are demonstrable across IQ, socio-
economic class, gender, and race.
Tough, Paul. How Children Succeed: Grit, Curiosity and the Hidden Power of Character.
(2012)
INDEX Mission Skills Assessment 16

17. Ethics Love of Learning Resilience
100% 73% 67%
Analysis of
mission
statements of 20
independent
middle schools
Creativity
Teamwork
67%
40%
INDEX Mission Skills Assessment 17

18. Analysis of university mission statements
(Oswald et. al., 2004)
Intellectual Interpersonal Intrapersonal
Knowledge and mastery Appreciation for Social responsibility and
of general principles diversity citizenship
Continuous learning, Leadership Physical and
intellectual interest and psychological health
curiosity
Artistic and cultural Interpersonal skills Career orientation
appreciation
Adaptability and life
skills
Perseverance
Ethics and integrity
INDEX Mission Skills Assessment 18

19. * *
**
“Very Important”
Percent Rating
* *
The Conference Board, Corporate Voices for Working Families, Partnership for 21st Century
Working Skills, and the Society for Human Resource Management.
INDEX Mission Skills Assessment 19

20. The Character Imperative
1) Research shows that these skills lead to
positive educational and life outcomes;
2) Colleges, universities, and employers are
seeking these skills in students and
workers; and
3) We must demonstrate the value we add.
INDEX Mission Skills Assessment 20

21. Guiding Children to Achieve their
Full Potential
Students are provided with a distinct
advantage if their schools excel in training
noncognitive skills. Independent Schools
can lead the way in designing innovative
character assessments and intentional
classroom strategies.
INDEX Mission Skills Assessment 21

22. Constructs in MSA
INDEX Mission Skills Assessment 22

23. Teamwork
Models of teamwork include 100
• Cooperation 90
• Influence
80
• Conflict resolution
• Guiding others 70
2nd most valued work 60
attribute according to Are they 50
Really Ready to Work? report Ma
th
IT Wr E E C R W L C L O T
itte thics nglis ritic ead ritin ifelo reat eade ral C eam ork
nC h al
Th ing g ng
Le
ivit
y rsh
W
om wor Eth
om ink Com arn ip . k ic
. ing p ing
In high school students, teacher reports of teamwork were correlated with grades
across several subjects (Wang, MacCann, Zhuang, Liu, & Roberts, 2009)
INDEX Mission Skills Assessment 23

24. Creativity
• Originality, progressiveness, or imagination
• Dictionary.com
• Tasks measuring creativity, “require examinees fairly
quickly to think of…a series of responses fitting the
requirements of the task….”
• Carroll, 1993; p. 428
INDEX Mission Skills Assessment 24

25. Creativity
100
90
5th most valued work
attribute according to Are 80
they Really Ready to Work?
report 70
60
50 Ma IT Wr E E C R Wr Lif C L Or Te Wo
th itte thics nglis ritic ead i elo rea ead a a
nC h al
Th ing ting ng tivi
ty
ers l Co mwo rk E
thi
Le hip m. rk
om ink Com a rn c
. ing p ing
Creativity predicts graduate school outcomes beyond GRE verbal and mathematics
scores (Frederickson & Ward, 1975; Bennett & Rock, 1995; Bennett & Rock, 1998)
INDEX Mission Skills Assessment 25

26. Ethics
100
Defined in our study as
cheating and concern for
others 90
One of the most valued work 80
attribute according to Are they
Really Ready to Work? report 70
60
50 Ma IT Wr E E C R W L C L O T W
th itte thics nglis ritic ead ritin ifelo reat eade ral C eam ork
nC h al ing g ng ivit rsh om wor Eth
Th Le y
om ink Com a rn ip . k ic
. ing p ing
INDEX Mission Skills Assessment 26

27. Resilience
• A level of adaptability that allows individuals to survive
and thrive in adverse conditions
• Dent and Cameron (2003)
• One way we measure resilience in the MSA is to measure
core self-evaluation – or how capable, worthy, and effective
a person feels.
INDEX Mission Skills Assessment 27

28. 2002 Income
1979 Core Self Evaluation
Judge and Hurst (2007) INDEX Mission Skills Assessment 28

29. Intrinsic Motivation / Love of
Learning / Curiosity
• What people will do without external inducement
(Malone & Lepper, 1987)
• In a study of 3rd – 8th graders, intrinsic motivation was
positively correlated with:
• Grades: r = .34
• Standardized Test Scores = r = .27
• However…
INDEX Mission Skills Assessment 29

30. (Lepper, Corpus, & Iyengar, 2005)
INDEX Mission Skills Assessment 30

31. Time Management
Determining one’s needs, setting goals to meet needs, and
prioritizing and planning to meet goals (Lakein, 1973).
Planning Organization
Grades .21** .38**
Grades .25** .42**
(6 months later)
Liu, Rijmen, MacCann, & Roberts (2009)
INDEX Mission Skills Assessment 31

32. How are we doing it?
INDEX Mission Skills Assessment 32

33. Our GPS:
Multi-Trait Multi-Method (MTMM) Design
Self Ratings
“Triangulation”
Not directly
observable
Measurement
True
True Method
Resiliency
Resilience
Alternative Teacher
Method Ratings
The underlying view of measurement in the MTMM analysis is that to measure a theoretical construct,
different measures, each with its own bias, are selected. Bias that is due to method effects is reduced through a
triangulation process. Baron & Kashy, 1992, p. 170
INDEX Mission Skills Assessment 34

34. Item Types
INDEX Mission Skills Assessment 35

35. Self-Report
INDEX Mission Skills Assessment 36

36. Situational Judgment Test
INDEX Mission Skills Assessment 37

37. Biographical Data
INDEX Mission Skills Assessment 38

38. Creativity Performance Test
INDEX Mission Skills Assessment 39

39. Teacher Ratings
INDEX Mission Skills Assessment 40

40. Resilience:
Representative Items
Self-Report
Situational Judgment Test I overcome challenges and set
You are feeling stressed about the amount of backs.
homework that you have been given by your
teacher(s). Below are some ways that you
might think, feel, or act in this situation, right
at the time that you feel stressed-out. Rate how Resilience
often you do each activity when you feel
stressed. How do you think, feel, or act when
you are stressed from having too much
homework to do?
Teacher Ratings
1. I take control and say to myself: I can do This student is easily
this! discouraged.
INDEX Mission Skills Assessment 41

41. What did we find?
INDEX Mission Skills Assessment 42

42. Participation
• Fall 2011
• 18 INDEX schools
• 2,081 students
• Spring 2012 (optional assessment)
• 13 INDEX schools
• 1,677 students
• Fall 2012
• 20 INDEX schools
• > 2,600 students
• Data being analyzed
INDEX Mission Skills Assessment 43

43. Average'Construct'Scores'By'Gender
103
102
101
Average'Construct'Score
100
99
98
Male
97
Female
96
95
94
2011 2012 2011 2012 2011 2012 2011 2012 2011 2012 2011 2012
Creativity Ethics Intrinsic9Motivation Resilience Teamwork Time9Management
Construct
44
INDEX Mission Skills Assessment

44. The MSA Demonstrates
Evidence of Reliability
INDEX Mission Skills Assessment 45

45. Reliability
2011 2012
Construct! Reliability! Reliability!
Time Management! 0.86! 0.86! SAT Reliability
Resilience! 0.84! 0.85!
Math .91
Intrinsic Motivation! 0.89! 0.90!
Creativity! 0.88! 0.88! Critical
Reading .91
Teamwork! 0.77! 0.85!
Ethics! 0.90! 0.85!
INDEX Mission Skills Assessment 46

46. The MSA Demonstrates
Evidence of Validity
INDEX Mission Skills Assessment 47

47. The MSA Predicts
• Grades
• Teacher Ratings of Student Quality
• “How would you rate this student compared to other students in
your school?”
INDEX Mission Skills Assessment 48

48. r = .56
Creativity and Student Quality Rating
4
2012 Student Quality Rating
3.5
3
2.5
2
1.5
1
0.5
0
E - Bottom D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%
20%
2011 Creativity
INDEX Mission Skills Assessment 49

49. How “Big” Are Our
Correlations?
Examples from other research (Meyer et al, 2001)
Variable 1 Variable 2 Correlation
Sugar Consumption Child’s Behavior .00
Aspirin Heart Attack Death .02
Parental Divorce Child Well-Being .09
Lead Exposure Child IQ .12
Ibuprofen Pain Reduction .14
Alcohol Aggressive .23
Behavior
Viagra Sexual Functioning .38
Gender Height .67
INDEX Mission Skills Assessment 50

50. 6th r = .29
7th r = .59
8th r = .58
Time Management and GPA by Grade
4
3.8
3.6
3.4
Spring 2012 GPA
3.2
3
6
2.8
7
2.6
2.4 8
2.2
2
E - Bottom D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%
20%
2011 Time Management
INDEX Mission Skills Assessment 51

51. The MSA Predicts
• Student Well Being
• My#life#is#going#well#
• My#life#is#just#right#
• I#would#like#to#change#many#things#in#my#life#
• I#wish#I#had#a#different#kind#of#life#
• I#have#a#good#life#
• I#have#what#I#want#in#life#
• My#life#is#be;er#than#most#kids’#
INDEX Mission Skills Assessment 52

52. r = .33
Resilience!and!Well!Being!
5!
4.9!
4.8!
2012!Well!Being!
4.7!
4.6!
4.5!
4.4!
4.3!
4.2!
4.1!
A!-!Top!20%! B!-!60%-79%! C!-!40%-59%! D!-!20%-39%! E!-!Bo7om!20%!
2011!Resilience!
INDEX Mission Skills Assessment 53

53. r = .31
Ethics and Well Being
5
4.9
4.8
2012 Well Being
4.7
4.6
4.5
4.4
4.3
4.2
4.1
A - Top 20% B - 60%-79% C - 40%-59% D - 20%-39% E - Bottom 20%
2011 Ethics
INDEX Mission Skills Assessment 54

54. There is initial evidence that the
MSA predicts just as well as
standardized tests
INDEX Mission Skills Assessment 55

55. MSA and ERB Scores: Correlations with Student
Quality Ratings
Correlation with Student Quality
0.6
0.50
0.5 0.46 0.45 0.44
0.4 0.38
0.31
0.3
0.2 0.16 0.15
0.1
0
INDEX Mission Skills Assessment 56

56. MSA and ERB Scores: Correlations with GPA
0.7
0.59
0.6 0.56
Correlation with GPA
0.52
0.49
0.5
0.41
0.4
0.28 0.27
0.3
0.21
0.2
0.1
0
INDEX Mission Skills Assessment 57

57. MSA and ERB Scores: Correlations with Well Being
0.3 0.28
Correlation with Well Being
0.25 0.23 0.22 0.22
0.19
0.2
0.14
0.15
0.1
0.04
0.05
0.01
0
INDEX Mission Skills Assessment 58

58. Current and Next Steps
• Continual refinement of MSA assessment
(adding interests)
• Assessing underserved populations with MSA
• Gathering interest from additional schools
INDEX Mission Skills Assessment 59

59. Implications for Schools
INDEX Mission Skills Assessment 60

60. Implications for Schools
• Wide-ranging and deep:
• Why?
• What?
• How?
INDEX Mission Skills Assessment 61

61. Why? Mission and Vision
• Reexamine school mission and how
measured
• Strength of mind and strength of character
• Develop a Vision for Character
INDEX Mission Skills Assessment 62

62. How?
Programs-
intentional and
well-designed
High-quality Evidence-
training and based
support Evaluation
INDEX Mission Skills Assessment 63

63. What? Programs
How can we teach these skills better and how will we know we
are succeeding?
• Deliberate, explicit and implicit instruction
• Proven interventions
• Performance Character Audit
• Articulated, spiraled, integrated curriculum
• Grade-level goals
• Updated and analyzed curriculum maps
INDEX Mission Skills Assessment 64

64. Programs
• Social Emotional Learning, advisories, personal
awareness, and co-curricula
• Shared lexicon, common language-Glossary of
Terms for Character
• Best Practices-follow through, evaluation,
refinement
INDEX Mission Skills Assessment 65

65. What? Training and Support
• Ongoing education–school leaders, faculty,
parents
• Shared vision, vocabulary, and resources
• Participation in regional and national educator
character summits
• Ongoing partnerships and dialogue
INDEX Mission Skills Assessment 66

66. What? Evidence-based
Evaluation
• Updated Progress Reports
• Parent and student conferences
• Faculty and parent education summits
• Greater emphasis: Self-reflection and meta-cognition
• Surveys
INDEX Mission Skills Assessment 67

67. Evidence-based Evaluation
• Integrated assessments
• Participation in new assessments (MSA)
• Benchmarking
• Data for national accreditation and new
criterion
INDEX Mission Skills Assessment 68

68. Measuring what we value:
Criterion 13:
The Standards require a school to provide evidence
of a thoughtful process, respectful of its mission,
for the collection and use in school decision
making of data (both external and internal) about
student learning.
INDEX Mission Skills Assessment 69

69. Improvement Phases
Envision Implement Evaluate and Refine
Commitment tied to Needs and resources Professional
mission assessment Development &
Partnerships
Create clear vision Action Plans, Communication and
integration, and engagement
enhancements
Build engagement Evidence-based • What is working?
strategies and programs • How do we know?
• How can we improve
teaching and learning?
INDEX Mission Skills Assessment 70

70. Community Responses
• Excitement!
• Shared understanding
• Partnership
• Shared commitment to studying results and asking the next
questions…
INDEX Mission Skills Assessment 71

71. Your School, Your Mission
Look to your mission, look to your alumni, look
to your purpose and your school’s view of what
our society needs the most: creative and
innovative thinkers, people of fine judgment,
leaders, collaborators, and…then select or design
the instruments which will help you make these
predictions.
Martin, Jonathan. Lessons from Law School Admissions: More from USC Attributes that Matter, 21K12
Blog, January 23, 2013.
INDEX Mission Skills Assessment 72

72. Thank you!
Questions?
Comments?
FOR MORE INFORMATION, PLEASE CONTACT:
INDEX
INDEPENDENT SCHOOL DATA EXCHANGE
LISAPULLMAN@INDEXGROUPS.ORG
WWW.INDEXGROUPS.ORG
MSA Steering Committee:
Chuck Baldecchi Dave Michelman Lisa Pullman
Head of School Head of School Executive Director, INDEX
The Lexington School (KY) Duke School (NC) lisapullman@indexgroups.org
CBaldecchi@thelexingtonschool.org Dave.Michelman@dukeschool.org
Rich Roberts
Tim Bazemore Ned Murray Managing Principal Research Scientist
Head of School Head of School ETS
New Canaan Country School (CT) Episcopal Day School (GA) rroberts@ets.org
TBazemore@countryschool.net nmurray@edsaugusta.com
Jeremy Burrus Jennifer Phillips
Research Scientist Director of Educational Advancement
ETS Far Hills Country Day School (NJ)
jburrus@ets.org jphillips@fhcds.org
INDEX Mission Skills Assessment 73