Abstract: The ability to authentically capture and assess student opinions and growth has always been a challenge. Polls, surveys, and focus groups are some of the most common ways for administrators to gather assessment data at an institutional level; however, these indirect sources only serve as proxy indicators of student voices and experiences. And while ePortfolios may provide better and more authentic direct assessment, they are usually not faster or easier. How can we merge the efficiency of the forms, polls, and surveys with the power of ePortfolios to have a faster, better, and a deeper look into our students’ expectations? We will provide an overview of our current text mining assessment methods for over 2000 students ePortfolio reflections given specific prompts. These methods will provide a counter-narrative to indirect surveys that liberate all of our students’ unique voices and allow us to better align their aspirations in the arena of higher education. Presenters: G. Alex Ambrose, Program Director of ePortfolio Assessment, University of Notre Dame Trunojoyo (Atun) Anggara, Academic Advisor and Educational Data Assessment Postdoctoral Fellow, University of Notre Dame

1. Liberate Learning Through
Next Generation Assessment
G. Alex Ambrose, Ph.D, Professor of the Practice and ePortfolio Program Director
Trunojoyo (Atun) Anggara, Ph.D, Academic Advisor and Educational Data Assessment Postdoctoral Fellow
Kaneb Center for Teaching & Learning, University of Notre Dame
Closing Plenary at the 9th Annual Forum on Digital Learning and ePortfolios
January 27, 2018

2. Acknowledgments
Kaneb Center for Teaching & Learning
•G. Alex Ambrose: Digital Learning Scientist
•Kevin Barry: Director of Center
•Chris Clark: Assistant Director of Center
•Kevin Abbott: Educational Technology Specialist (OIT)
•Alison Lanski: ePortfolio Implementation Specialist
•Georges Alsankary: student worker
Office of Digital Learning (ODL)
•Elliott Visconsi, Chief Academic Digital Officer
•Laurie Kirkner, Interim Director and Senior Learning Designer
•Crystal DeJaegher: ODL Instructional Designer
Office of Information Technologies
•Xiaojing Duan: Platform Engineer & Data Czar
•Patrick Miller, Sakai Lead Learning Management Customer IT Solutions
•Laura Gekeler, Sakai Learning Management Professional Customer IT Solutions
First Year of Studies
•Holly Martin, Associate Dean
•Maureen Dawson, Assistant Dean
•Hugh Page, Dean
Interdisciplinary Center for Network Science & Applications (iCeNSA)
•Everaldo Aguiar: Data Scientist
•Munira Syed: PhD student
Notre Dame Faculty
•Nitesh Chawla: Full Professor in the Department of Computer Science & Engineering
and Director of iCeNSA

3. #LiberateLearning
#AACU18
@gAlexAmbrose
@Atun_Anggara
Access a Copy of our Slides
https://tinyurl.com/aacu2018liberate

4. Many meanings in this image...
What do you notice?
How do you react and with which emotions?
Can you relate and what does it mean to you?
What title or caption would you give it?

5. What does assessment in higher education
actually do to the American dream?
Free
Thrive
Liberate
Restrain
Harm
Limit
ASSESSMENT
LEARNINGTEACHING
HIGHER
EDUCATION
AMERICAN
DREAM

6. Assessment Design Questions
What role does technology play?
What problem does it address?
If technology is the answer, what was the question?

8. Digital Assessment Innovation #1:
Taking the ePortfolio Pulse

9. Context: Advising Top Down Program Assessment
Selected Responses Constructed Reflection

10. Ambrose, G. Alex, Martin, Holly, Page, Hugh (2014) “Linking Advising and ePortfolios for Engagement: Design, Evolution, Assessment, &
University Wide Implementation” AAC&U Peer Review Winter 2014, Vol. 16

11. ePortfolio Form Export
Survey ---> Emails ---> Online Form --->ePortfolio ---> ePortfolio (w/ Embedded Forms)

12. ● During academic year of 2016-2017:
First year advising at Notre Dame is likely to be different from the
advising you received in high school. Your advisor will assist you in making the
transition to college academics and accessing the resources ND has in place to smooth the way. He
or she will also help you understand and begin to make informed choices about the basics of the
University’s requirement and elective course system. You will talk with your advisor about your
academic hopes, concerns, and plans, and he or she will suggest paths and opportunities to fit your
goals. During your first year, with the assistance of your First Year advisor and others, you will begin
to build an education suited to your goals and interests. List the top three expectations you have for
your academic advising experience.
Advising Initial Thoughts prompt evolution:
2015-2016 to 2016-2017
● During academic year of 2015-2016:
List the top 3 of your expectations walking into an advising session.

13. What is the effect of prompt-priming with respect to student
response in Initial Thoughts?
● By the total number of words written:
○ 2015-2016: 78,479 total words
○ 2016-2017: 111,619 total words
○ Word increase of 42% between the two years, even though there is 17 less ePortfolios being
written in 2016-2017
Out of 1795 ePortfolios
Out of 1778 ePortfolios

14. What is the effect of prompt-priming with respect to student
response in Initial Thoughts? | Unigrams
2015-2016
Higher mentions since higher
number of words generated by
the students
2016-2017
Hope is less prevalent here
Major and academic related
words more prominent here

15. What is the effect of prompt-priming with respect to student
response in Initial Thoughts? | Bigrams
2016-20172015-2016
More mentions in:
● study abroad
● medical school
● career path
High school is mentioned less here

16. 1. Better prompt-priming resulted in more engagement (around 40% increase of
writing) from the First Year Students in 2016-2017 compared to 2015-2016
2. More substantive words (major, study abroad, medical school, etc. being
mentioned) in 2016-2017 compared to 2015-2016
First case Takeaways

17. Dashboard Demo

18. Demo Screenshot

19. Demo Screenshot

20. Demo Screenshot

21. Demo Screenshot

22. Demo Screenshot

23. Digital Assessment Innovation #2:
Boosting Learners to Thrive

24. Challenge
National Center for Educational Statistics https://nces.ed.gov/collegenavigator/
Can we identify those 2% (40 students) & help everyone thrive?
How can we ensure everyone has the opportunity to thrive?

25. Context: Course Bottom Up Student Assessment
Moreau First Year of Experience Course Scale & Stats:
- 2,000+ students
- 114 classes (class size of <19)
- 56,000 assignments turned in and graded with common rubrics per year
- 2.3 million click tracks and 15 million words text mined per year

26. Next Generation Digital Learning Environment (NGDLE)
Ambrose, G. Alex, Abbott, Kevin, Lanski, Alison (2017) “Under the Hood of a
Next Generation Digital Learning Environment in Progress” Educause Review.

27. Analytics & Assessment Data Process

28. Identify --> Notify --> Boost --> Evaluate

29. Identify --> Notify --> Boost --> Evaluate
Subject:
Request for Immediate Moreau Action Plan
Message:
Dear Moreau Student,
As Assistant Dean and Co-director of the Moreau First Year Experience, I recently reviewed a report on the progress of all Moreau students over the first five weeks
of the semester.
I am sending this message to you as a way of offering assistance and support since the report indicates that you might be showing signs of struggling. We want
to ensure that you have access to any and all resources you might find useful.
Out of the 2000+ students in the course, you are one of only 39 (2%) who have not submitted or scored a 0 on at least two weekly prompts. In the past
two years of this course, this type of early performance is often associated with low midterm and final Moreau grades as well as a low overall semester
GPA.
In order for me to understand your situation better and for you to learn about helpful resources, please take two minutes to fill out this personal action plan. You
can also reply to this message to get in touch with me directly. If you haven't provided an action plan response within 48 hrs of this message, I'll plan to share the
information above with your FYS advisor to ensure that you are succeeding at the highest possible level.

30. Identify --> Notify --> Boost --> Evaluate
(Early)
Early Semester Learner Action Plan

31. ReasonsforMissingAssignment
Technical/Grading
Forgot to submit
Stressed about things on
campus
Struggling to keep up w/
classes
Stressed about other things
at home
Talk to instructor about
submission
Be more careful in the future
Talk to instructor about
submission
Contact Rector
Contact care consultant
Contact peer advisor
Contact counselor
Contact FYS advisor
Talk to instructor
Contact Program in
Academic Excellence

32. Mid Semester Advisor Boost Report
Identify --> Notify --> Boost --> Evaluate
(Mid)

33. Thrive after Boost Impact
2,053 Total FYS
75 ID & Given a Boost
Identify --> Notify --> Boost --> Evaluate
1% Did not thrive with ≤B- grade
99% Thrived with ≥B Grade at final
13/75 Did not thrive after at least 1
boost
6 Students Crashed after Midterms
62/75 Thrived after at least 1 Boost

34. Wrapping Up

35. What role does technology play?
What problem does it address?
If technology is the answer, what was the question?
How can we move beyond efficient assessment to
more effective assessment thereby liberating the learning and the learners?

36. Liberating the American Dream of Our
“Barred Student at the Crossroads”

37. References
Related Presentations:
Ambrose, G. Alex, Martin, Holly, Page, Hugh (2014) “Linking Advising and ePortfolios for Engagement: Design, Evolution,
Assessment, & University Wide Implementation” AAC&U Peer Review Winter 2014, Vol. 16
Ambrose, G. Alex, Abbott, Kevin, Lanski, Alison (2017) “Under the Hood of a Next Generation Digital Learning Environment in
Progress” Educause Review.
Image Sources:
https://www.tolerance.org/magazine/spring-2013/the-schooltoprison-pipeline
https://www-03.ibm.com/ibm/history/exhibits/specialprod1/specialprod1_9.html
https://thumbs.dreamstime.com/x/find-needle-haystack-22905126.jpg
http://images.techtimes.com/data/images/full/25830/sven-sachsalber.jpg
http://media.gettyimages.com/illustrations/looking-for-a-needle-in-a-haystack-illustration-idiwdayala0001c?s=170667a
http://mitchellsfrontpage.com/wp-content/uploads/2017/02/800x400-elite-daily-robot-lawyer.jpg

38. How to Cite this Presentation:
Ambrose, G. Alex, Anggara, Trunojoyo (2018) “Liberate
Learning Through Next Generation Assessment”
Association of American Colleges & Universities, 9th
Annual Forum on Digital Learning and ePortfolios.
Washington, D.C.

39. Questions & Discussion
Access our slides
https://tinyurl.com/aacu2018liberate
Connect or Collaborate
gambrose@nd.edu tanggara@nd.edu