Deriving value from analytics requires much more than purchasing technology. University of Kentucky's analytics journey utilized fostering a bottom-up emergent community of practice as well as top-down organizational maneuvers. This presentation shares different aspects of the University of Kentucky score.
1. Organizing to Ret Analytics RightVince Kellen, Ph.D. Senior Vice Provost, Analytics and TechnologiesVince.Kellen@uky.eduThis is a living document subject to substantial revision! September, 2014
2. Silos
Are recursive
Get reproduced across time
and space reliably, without
effort
Arise naturally due to human
sociological/biological
tendencies
It takes constant effort to
mitigate their adverse effects
Sharing data and analysis
widely requires a
reconceptualization of silo
structures
2
3. Organization dysfunction
Information as power
Defensiveness
Data hoarding
Process separation
Empire building
Excessive control
Fear of scrutiny
Loss of power
3
4. We are competitive animals
Information becomes a [tool, weapon]
We instinctually manage information to enhance our competitiveness
Competition relies on information hiding
IT tools become part of our body
How we personally utilize information is part of our biological heritage. This is hardto change, if at all
4
5. Shift from production concerns to consumption ones
Production
•
Collecting, integrating, cataloging, categorizing, transforming, abstracting, analyzing, model-building, visualization, dashboarding, distributing, publishing. If you build it they will come (hopefully)
Consumption
•
Motivating, collaborating, expressing, integrating, improving action, increasing ambition, desire, recognition. If theybuild it everyonewill come
5
8. A.
Merging of mobile and BI strategy
B.
Merging of IR and BI units
C.
Super high-speed infrastructure
D.
Single analytic value chain
E.
Analytics community of practice
F.
Data transparency
G.
Community sourcing and norming
H.
Community rules of etiquette
8
9. Our Community of Practice Rules of Etiquette
Be safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please have
good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and university
information appropriately.
Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with
the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for
greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what absolutely
needs to be private, the way the university runs it's business often involves multiple colleges and units at the same time. Don't
use your access to take unfair advantage of another unit.
Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff
dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data management
processes that might need to be changed to improve data quality.
Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking great
care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In time, we
can accommodate more ways of looking at the same data while not interfering with other views or taxonomies.
Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a few
select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the university
can improve accordingly.
9
10. Organizing IT
Our organizational model makes a big difference. Other universities fail to take advantage of a tool like this for purely political reasons
Making key data transparent to all does not help those who made their living being the data ‘go to’ person
We had to merge two units (Institutional Research and Business Intelligence), losing 1/3 of the staff. This let us hire three data scientists with different analytic backgrounds
The tool let the staff transition their skills
10
11. What we have done and what we would like to do
First steps over the past year
•
Mobile micro-surveys: Learning from the learner. In one year, 134,458 surveys harvested. Survey response rates are holding at about 40%. We can instantly analyze all responses for retention and progression issues
•
Student enrollment, retention, demographics, performance, K-Score, facilities utilization, instructor workload, student revenue and financial aid, student progression and more
•
High speed, in-memory analytics architecture. Lowest level of detail, maximum semantic expressiveness, one- second per click for analyst are key design philosophies
•
Open data and organizational considerations
Coming down the road?
•
Micro-segmentation tool to enhance user and IT productivity, develop personalized mobile student interaction/intervention
•
Models for learner technographics, psychographics, in addition to behaviors, performance, background
•
Advanced way-finding for streaming content like lecture capture
•
Content metadata extraction and learner knowledge discovery
•
Real-time measures of concept engagement and mastery
•
Real-time learner recommendations and support engine
•
Use graphing algorithms to perform more sophisticated degree audit what ifs
11
13. List builder
Iteratively query any/all fields of your choosing, linking in an AND or ORfashion
Combine different lists using SET manipulations
Refresh lists regularly (nightly or otherwise)
Apply the set name as a filter on ALL models
This provides advanced filtering and combining that works regardless of the user interface
Our AA team can build and maintain Lists easily. So can some users
Since lists are refreshed nightly, we can keep track of each time a student (or other entity) as it added or removed from a list
We can develop workflow apps using this. Backend, front-end agnostic
13
16. 2009201020112012201320142015Academic year81012141618202224262830323436 Avg 19109107810716016111836154551551654716214195110916216556Fast/Slow ProgressionStudent headcount850100165Cohort YearFall 2008Fall 2009Fall 2010Fall 2011Fall 2012
List builder visualization example
Found all students who take a lot of classes at one point in their career and then took less classes at another point in their career.
Interpretation: These students start with a bang but fade at the finish
How long did this analysis take? Start to finish with this visualization:
25 minutes
16
18. Identifying smaller segments of students
In addition to our work on difficult student cases, we needed to find a way to reach a ‘murky middle’ group of students
Identify students who are just as likely to come back as they are not
The predicted reenrollment was about 50%
After interventions, the actual enrollment was about 65%
19. The whole enchilada
Personalize learning, learning analytics and IPAS analytics into one real-time architecture
•
Real-time personalized interactions
•
Target on-demand peer tutoring based on student’s profile
•
Deliver micro-surveys and assessments to capture additional information needed to improve personalization
•
Give students academic health indicators that tell students where they can improve in study, engagement, support, etc.
•
Let students opt their parents in to this information so the family can support the student
•
Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user temporal proclivities, mix and match messages across learning, support and progression areas
•
Allow for open personalized learning
•
How content gets matched to students is psychologically complex
•
Several theories of how humans learn give many insights
•
Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective memory, etc.
•
We need an open architecture to promote rapid experimentation, testing and sharing of what works and what doesn’t
University of Kentucky
20. Herding cats
We shared with everyone that we are building the bridge as we walked on it
We established a community of practice and rules of analysis etiquette
We built tailored objects for colleges, let users choose their own front end tool
We relied on word-of-mouth adoption and some teasing-revealing
Guess what happened?
20
21. Top-down versus bottom-up
Doing this top down is like pushing water uphill. Its harder than pushing a rock uphill
The great leader is one who the people say “We did this ourselves”
Consider analytics to be a process of self discovery. Each person has to go through the stages of maturity
Paradoxically, this also requires strong top-down commitment and action! Organizational maneuvers like reorganizations are [normally] required
21