1. Measuring Grit
Do Non-Cognitive Attributes Impact Academic
Success, Engagement, Satisfaction and Retention?
Dr. Mac Adkins, President
Provided by
SmarterServices

2. Measuring Grits
• We recommend a glass beaker for
measuring grits. We are from Alabama,
we know about these things.

3. What is Grit?
• Why does one student who had straight
A’s in high school drop out of college after
one year, and another one excel?
• Why does one single mom with three
children graduate Summa Cum Laude and
another one drop out?

4. What is Grit?
• Grit is that elusive quality that prompts one
student to stick with it while others quit.
• For over ten years we have measured
levels of grit in over 2,000,000 students at
over 500 colleges and universities.
• Today I want to share with you the results
of research related to the impact that grit
has on student success.

5. Some Students Seem To Have
More Grit Than Others

6. Three Approaches to
Measuring Grit
• Stick your head in the sand.
• Use a brief, non-prescriptive survey.
• Use SmarterMeasure Learning Readiness
Indicator.

7. What is SmarterMeasure?
• A 124-item online skills test and attributes
inventory that measures a student’s level
of readiness for studying online
• Used by over 500 Colleges and
Universities
• Taken by over 2,000,000 students

8. What Are The Ingredients
In Our Grits?

9. What Does The Assessment Measure?
INTERNAL EXTERNAL SKILLS
INDIVIDUAL LIFE FACTORS TECHNICAL
ATTRIBUTES
Availability of Time Technology Usage
Motivation Dedicated Place Life Application
Procrastination Reason Tech Vocabulary
Time Management Support from Family Computing Access
Help Seeking
Locus of Control TYPING
LEARNING STYLES Rate
Accuracy
Visual
Verbal ON-SCREEN
Social READING
Solitary
Physical Rate
Aural Recall
Logical

10. How Does The Beaker Work?

17. How Do You Spot a Bad Grit?

23. Adjusting Readiness Ranges
Adjusting the cut
points can make
the reporting a
more accurate
predictor of
success.

24. How Do Schools Use It?
• Orientation Course
• Enrollment Process
• Information Webinar
• Public Website
• Class Participation
• Facebook

25. Thermometer Analogy
• More important than taking your child’s
temperature is taking appropriate action
based on their temperature.
• More important than measuring student
readiness is taking appropriate action
based on the scores.

26. Measuring the Grits

27. Progression of SmarterMeasure Data Utilization
Predictive
Correlation
Comparison
Descriptive
Student Service

28. Research
Ideas on the
Research
Page of the
Website

29. Approaches to Research Projects
Internally Company Professionally
Conducted Assisted Assisted

30. Middlesex Community College
• 6% to 13% more students failed online
courses than on-ground courses.
• Intervention Plan
- Administer SmarterMeasure
- Identify which constructs best predicted success
- Provide “Success Tips” as identified
Distributed by website, email, orientation
course, records office, library, posters, and
mail

31. Research Findings
• Analyzed 3228 cases over two years
• Significant positive correlation between
individual attributes and grades
Motivation Impacts Grades

32. Results of Middlesex Research
Before SmarterMeasure™ was implemented, 6%
to 13% more students failed online courses than
students taking on-ground courses. After the
implementation, the gaps were narrowed: 1.3%
to 5.8% more online students failed than on-
ground students.

33. Results of Middlesex Research
Failure rates reduced by as much as 10%

34. Action Plan
• Empower eLearning staff, faculty advisors,
and academic counselors with student
data
Three
areas of Motivation
Self Time
Discipline Management
focus

35. Project Summary
“In summary, the
implementation of
SmarterMeasure
has helped students
to achieve better
academic success
by identifying their In essence, with various strategies
implemented to promote
strengths and SmarterMeasure™, a “culture” was
weaknesses in created during advising and registration
for students, faculty, and support staff to
online learning.” know that there is a way for students to
see if they are a good fit for learning
online.

36. CEC - The Need
• We need to know which students to advise
to take online, hybrid or on-campus
courses.
• We need to know which students to direct
to which student services to help them
succeed.
• We need to know how to best design our
courses so that new students are not
overwhelmed.

37. The Analysis
• What is the relationship between
measures of student readiness and
variables of:
– Academic Success - GPA
– Engagement – Survey (N=587)
– Satisfaction – Survey (Representative Sample
based on GPA and number of courses taken
per term)
– Retention – Re-enrollment data

38. The Analysis
• Phase One – Summer 2011
– Included data from all three delivery systems – online,
hybrid and on-campus
– Analyzed data at the scale level
• Phase Two – Fall 2011
– Focused the research on online learners only
– Analyzed data at the sub-scale level
• A neutral, third-part research firm (Applied
Measurement Associates) used the following statistical
analyses in the project:
– ANOVA, Independent Samples t-tests, Discriminant
Analysis, Structural Equation Modeling, Multiple
Regression, Correlation.

39. The Findings
• Academic Achievement
– The scales of Individual Attributes, Technical
Knowledge, and Life Factors had statistically
significant mean differences with the
measures of GPA.

40. The Findings
• Retention
– The measure of Learning Styles produced a
statistically significant mean difference
between students who were retained and
those who left.
• A 73% classification accuracy of this retention
measure was achieved.
– The scales of Individual Attributes and
Technical Knowledge were statistically
significant predictors of retention as measured
by the number of courses taken per term.

41. The Findings
• Engagement
– The scales of Individual Attributes and Technical
Competency had statistically significant relationships
with the four survey items related to Engagement.
– The scales of Life Factors, Individual Attributes,
Technical Competency, Technical Knowledge, and
Learning Styles were used to correctly classify
responses to the survey questions related to
engagement and satisfaction with up to 93%
classification accuracy.

42. The Findings
• Satisfaction
– Structural equation modeling was used to create
a hypothesized theoretical model to determine if
SmarterMeasure scores would predict
satisfaction as measured by the survey.
– Results indicated that prior to taking online
courses, student responses to the readiness
variables were statistically significant indicators of
later student satisfaction.
– Therefore, the multiple SmarterMeasure
assessment scores are a predictor of the Career
Education survey responses.

43. The Findings
• Statistically Significant Relationships
Academic Engagement Retention
Achievement
Individual X X X
Attributes
Technical X X X
Knowledge
Learning X X
Styles
Life Factors X X
Technical X
Competency

44. The Findings
• Student Categorizations
– Enrollment Status
• Positive – active/graduated (34.3%)
• Negative – withdrew/dismissed/transfer (65.7%)
– Academic Success Status
• Passing – A, B or C (48.9%)
• Failing – D, F or Other (21.1%)
– Transfer Credit – (21.8%)
– Not reported – (8.2%)

45. The Findings - Correlates
Readiness Domain Readiness Domain Subscales
Positive vs. Negative Pass vs. Fail
Life Factor Place, Reason, and Skills Place
Learning Styles
Social
and N/A
Logical
Personal Attributes
Academic, Help Seeking, Procrastination,
Time Management
Time Management, and Locus of Control
Technical Competency
Internet Competency
Internet Competency and
Computer Competency
Technical Knowledge
Technology Usage
and Technical Vocabulary
Technical Vocabulary

46. The Findings - Predictors
Readiness Domains GPA F p
Life Factor Place and Skills 12.35 .0001
Learning Styles Verbal a and Logical 3.95 .02
Personal Attributes
Help Seeking, Time
Management, and Locus of 21.11 .0001
Control
Technical Competency
Computer and Internet
22.75 .0001
Competency
Technical Knowledge
Technology Vocabulary
38.76 .0001

47. The Findings - Predictors
Readiness Domains Credit Hours Earned F p
Life Factor Place 12.37 .0001
Learning Styles Visual 6.81 .01
Personal Attributes
Academic Attributes, Help
Seeking, and Locus of 13.40 .0001
Control
Technical Competency
Computer Competency
12.23 .0001
and Internet Competency
Technical Knowledge
Technology Usage and
Technology Vocabulary 26.97 .0001

48. The Recommendations
• We need to know which students to advise
to take online, hybrid or on-campus
courses.
– A profile of a strong online student is one
who:
• Has a dedicated place to study online
• Possesses strong time management skills
• Demonstrates strong technical skills
• Exhibits a strong vocabulary of technology terms

49. The Recommendations
• We need to know which students to direct
to which student services to help them
succeed.
– An online student who should be directed toward
remedial/support resources is one who:
• Has a weak reason for returning to school
• Has weak prior academic skills
• Is not likely to seek help on their own
• Is prone to procrastinate
• Has low, internal locus of control
• Has weak technology skills

50. The Recommendations
• We need to know how to best design our
courses so that new students are not
overwhelmed.
– Limit advanced technology in courses offered early in
a curriculum
– Foster frequent teacher to student interaction early in
the course
– Require milestones in assignments to prevent
procrastination
– Clearly provide links to people/resources for
assistance

51. Argosy University
• Required in Freshman Experience course
• Students reflect on scores and identify
areas for improvement in their Personal
Development Plan
• Group reflection with others with similar
levels of readiness

52. Argosy University -
COMPARE
• Compared the traits, attributes, and skills of the online
and hybrid students.
• Substantial differences between the two groups existed.
• Changes were made to the instructional design process
for each delivery system.
Online Hybrid

53. Argosy University - EXPLORE
• Correlational analysis between SmarterMeasure scores
and student satisfaction, retention, and academic
success
Statistically Significant Motivation
Factors:
Technical Time
Technical Competency
Motivation Satisfaction
Availability of Time. Retention
Success

54. Argosy University - TREND
• Aggregate analysis of SmarterMeasure data to
identify mean scores for students.
• Comparison made to the national mean scores
from the Student Readiness Report.
National
Scores
Argosy
Scores

55. Argosy University - APPLY
• Findings were shared with the instructional
design and student services groups and
improvements in processes were made.
For example, since technical
competency scores increase as
the students take more online
courses, the instructional
designers purposefully allowed
only basic forms of technology to
be infused into the first courses
that students take.

56. J. Sargeant Reynolds
Community College
• Required as admissions
assessment Attributes
• Integral part of their QEP
• Computed correlations
with grades and Life
SmarterMeasure Factors Grades Technical
sub-scales of over 4000
students.
• P Learning
Styles

57. Findings
• Statistically significant correlations:
- Dedicated place, support from employers
and family, access to study resources, and
academic skills (Life Factors)
- Tech vocabulary (Technical Knowledge)
- Procrastination (Individual Attributes)
Scores Grades

58. Academic Success Rates
70
60
50
40 High Score
Low Score
30
20
10
0
Skills Resources Time
Less than 10% of students with low scores
experienced academic success.

59. Five Schools
What is the relationship between measures
of online student readiness and measures
of online student satisfaction?

60. Methodology
Incoming vs Outgoing
Data from 1,611 students who completed both the
SmarterMeasure Learning Readiness Indicator
and the Priority Survey for Online Learners were
analyzed.

61. Findings
• There were statistically significant
relationships between factors of readiness
and satisfaction.

62. Comparison to
Compass Scores
North Central Michigan College - Petoskey, MI

63. National Data
• 2012 Student Readiness Report
• Data from 690,927 students from 324
colleges and universities

64. Online Learner Demographics
• 70% were female
• 59% were Caucasian/White
• 54% had never taken an online course before
• 35% were traditional aged college students
• 52% were students at an associate’s level
institution

65. Online Learner Demographics
• Dominant Social learning style
• Highly motivated
• Moderate reading skills
• Pressed for time
• Fast typists
• Increasing technical skills

66. Profile of a
Successful Online Student
• Four demographic variables have had a
statistically significant higher mean for four
years in a row.
Females higher in
Individual
Attributes, Academic
Attributes, and Time
Management.
Males higher in Technical
Knowledge.

67. Profile of a Successful
Online Student
• Caucasians have had the highest means
for four years in Technical Knowledge.
• Students who have taken five or more
online courses have had the highest
means for four years in Individual
Attributes, Technical Knowledge, and
Procrastination.

68. Conclusion
• Statistically significant relationships exist
between measures of online student
readiness and measures of academic
success, engagement, satisfaction and
retention.

69. Conclusion
• Students individually benefit and schools
collectively benefit from measuring learner
readiness and appropriately responding.

70. SmarterMeasure.com

71. SmarterMeasure References
Middlesex, Argosy, J. Sargeant Reynolds:
http://smartermeasure.com/research/research-results/
Noel Levitz Study: https://www.noellevitz.com/papers-
research-higher-education/2011/2011-adult-and-online-
learner-satisfaction-priorities-reports
Student Readiness Report:
http://smartermeasure.com/smartermeasure/assets/File/
Online-Student-Readiness-Report.pdf

72. “Live as if you were to die
tomorrow. Learn as if you were
to live forever.”
Mahatma Gandhi

73. For More Info
SmarterServices.com
(877) 499-SMARTER
info@SmarterServices.com