Jackie S. Zajac of American University's presentation on UX Results Through Research at Penn State, June 2013.

1. Good UX Work That
Won’t Go to Waste
A Case Study of American.Edu/Kogod

2. Jackie Zajac
Director, Strategic Communications
• Journalism —> Marketing & Communications
• 4 years at AU/Kogod
• Never really heard of Usability until 2012

3. Inspiration
• PSUWeb12
• User Interactions with Information Systems
• Declining graduate enrollment
• Bickering over web strategy

4. School Background
• Kogod School of Business
• 1,200 undergraduate, 600
graduate students
• 120 Faculty & Staff
• 8 Graduate Programs
• 40,000+ visits/month

5. “1,000 More Grad Students in 10 Years”

6. Reporting Structure
8 Faculty Program
Directors
6-person Admissions
Team
Dean, Provost,
University Marketing Advisory
Council, Web Steering Committee...

7. Study Parameters
• Time (< 3 weeks)
• Modest budget ($250)
• Technical limitations due to AU’s CMS
• Only tested American.Edu/Kogod

8. Usability Test

9. Tools!
1. Silverback ($69.95)
2. Treejack (Free version)
3. Audio recorder
4. Pen + paper
5. Gift cards

10. Test Design
Tasks (6)
• Specific to Prospective Graduate Students
Users (5)*
• Demographic matches
*Jakob Nielsen said it was OK!

11. Test Design
Test outline
• Homepage Impressions
• Tasks
• Post-Task Completion Survey
• Final Thoughts?
• 30 Minutes

12. Tasks
1. How many credits are in the MS in Accounting program?
2. Find the average GMAT score for a Full-time MBA student
3. What is the cost of the Part-time MBA program?
4. When is the next Information Session being held?
5. Find the graduate admissions application
6. Find the graduate admissions’ office phone number
*Abbreviated wording

13. Completion Survey
After each task, the user was asked:
1. Did you complete the task? (Yes/No)
2. How satisfied are you, on a scale of 1 to 5? (5 being
very satisfied)
3. How easy was it, on a scale of 1 to 5? (5 being very
easy)

14. Test Participants
• 5 users
• Washington, D.C.-based
• Well-educated
• Bachelor’s Degree Minimum
• Age match (average 28.8 y/o)
• 4 Domestic, 1 International Background

15. 40%
60% 60% 60%
80% 80%
100%
80% 80%
60%
80% 80%
Find the number
of credits in the
MS of Accounting.
Find the average
GMAT score for a
Full-Time MBA
student.
Find the cost of the
Professional (Part-
time) MBA
program.
Find when the
next information
session is being
held.
Find the graduate
admissions
application.
Find the graduate
admissions’ office
phone number.
Task completion rate
Task completed from testers' perspective
Task completed from users' perspective

16. 0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Satisfaction and Ease from User Perspective
Task completion
satisfaction rate from
users' perspective (1 to
5)
Easiness rate from users'
perspectives (1 to 5)

17. 0 1 2 3 4
Site is attractive and has a
good balance of graphics/text.
Site was easy to navigate.
Information is easy to read.
I always felt I knew what was
possible to do next.
My mistakes were easy to
correct.
Number of user responses
Users’ Final Impressions
Maybe
No
Yes

18. Key Finding:
Too Many Navigational Options

19. User Comments
“My eyes are drawn immediately to the big large picture. It
feels like there’s a lot of stuff framing it, going around…it’s
kind of confusing.”
“FTMBA…I’m not really sure what that acronym means.”
“Would I give up right now? If this was a school I wanted to
go to, No. …But it’s not easy to find.”

20. Page Load Time

21. Page Load Time
• Average homepage load time = 6.1 seconds
• Slower than 76% of tested websites
• Industry standard = 4 seconds or less (June 2012)
• Many Javascript and CSS files built into the design
• Some homepage images were >500kb

22. Card Sort

23. Card Sort
• 22 participants in a closed card sort
• Median age 25-35
• 28 cards, 4 categories
• Graduate Admissions
• Graduate Programs
• Contact Us
• About Kogod

24. Card Sort Results
• The majority of users had similar ideas P
• 3 cards consistently sorted in different categories
• Frequently Asked Questions
• Information Sessions
• Visiting Campus

25. Card Sort
Results—22
participants

26. Kogod Home
Page
About
Contact Us
Dean's Blog
Facts & Rankings
Kogod Advisory
Council
Kogod Brochure
School History
Welcome From
the Dean
Visiting Campus
Graduate
Admissions
Admissions
Deadlines
Application Guide
Enrollment
Application
Graduate Degree
Requirements
Information
Sessions
International
Applicants
Military
Applicants
Test Scores
Request More
Info
Tuition and Fees
Visiting Campus
FAQs
Graduate
Programs
Full Time MBA
Full Time MBA
Career Outcomes
Professional MBA
Graduate
Certificates
MS in Accounting
MS in Finance
MS in Real Estate
MS in
Sustainability
Management
MS in Taxation
Proposed IA

27. Kogod Home
Page
About
Contact Us
Dean's Blog
Facts & Rankings
Kogod Advisory
Council
Kogod Brochure
School History
Welcome From
the Dean
Visiting Campus
Graduate
Admissions
Admissions
Deadlines
Application Guide
Enrollment
Application
Graduate Degree
Requirements
Information
Sessions
International
Applicants
Military
Applicants
Test Scores
Request More
Info
Tuition and Fees
Visiting Campus
FAQs
Graduate
Programs
Full Time MBA
Full Time MBA
Career Outcomes
Professional MBA
Graduate
Certificates
MS in Accounting
MS in Finance
MS in Real Estate
MS in
Sustainability
Management
MS in Taxation
Proposed IA
X

28. What’d We Do
With All This?

29. Recommendations
• Add an Admissions tab to the top nav (Duh.)
• Link directly to each graduate program from the
homepage
• Restructure Admissions nav
• Create a new, central MBA “landing” page
• Allows users to view both formats before click

30. Recommendations
• Decrease jargon
• Global Learning vs. Study Abroad
• FTMBA vs. Full Time MBA
• Make functional info clearer + easier to find
• Breadcrumb navigation?
• Social media more prominent
• Better process for Information Sessions
• Reduce image file sizes
• Delete homepage touts

31. For example

32. All Aboard!

33. All Aboard!
Dear all,
You may have noticed some changes to the Kogod website recently. They are the
result of a usability study of the website we conducted this summer. The focus of our
testing was to determine how easy the website is for our primary audience—
prospective students—to use. Based on the study's results, we've implemented
several changes to the website's layout, navigation and terminology. A few more
changes lie ahead.
We expect that this process will make it easier for our prospects to find the
information they're looking for, and to take action. A follow-up study will be conducted
in 2013 to determine whether further improvements are needed.
If you have any questions, do let me know.

34. Your BFFs <3
1. Data
2. Admissions
3. School Goals

35. Results @ 6 Weeks
Admissions Homepage: 7,637 pvs (+3%)
MS Finance: 3,819 pvs (+50%)
MS Accounting: 3,115 pvs (+60%)
MS Sustainability Mgmt : 1,325 pvs (+5%)
MS Taxation: 1,106 pvs (+84%)
MS Real Estate: 1,095 pvs (+63%)
Renamed “Tuition & Fees”: 2,147 pvs (+150%)
New MS Applicants page: 2,515 pvs, 02:19 TOP

36. Results @ 6 Weeks
New MBA “landing” page: 2,617 pvs (n/a)
FTMBA page: 2,621 pvs (-10%)
PMBA page: 1,733 pvs (+11%)

37. Results @ 6 Weeks
New MBA “landing” page: 2,617 pvs (n/a)
FTMBA page: 2,621 pvs (-10%)
PMBA page: 1,733 pvs (+11%)

38. Results: 6 Weeks

39. Then Came…
More Changes
Redesigned Degree Pages
• Clearer Calls to Action
• Consistency
Redesigned Graduate Programs index page

42. Remember:
Always send a status update to Program Directors.
Don’t forget SEO.

43. Now @ 9 Months
Overall: No substantial change in visitors
But we’re DOWN:
• Pageviews -6.7%
• Avg visit duration down (slightly)
• Pages per visit -7.3%

44. Now @ 9 Months
Admissions homepage: +3.8% over same time period
MS Finance: +58.8%
MS Accounting: +48.9%
MS Sustainability Mgmt: +12.4%
MS Taxation: +72.4%
MS Real Estate: +74%
Tuition & Fees: +190.7%

45. Now @ 9 Months
New MBA landing page: 18,451 pvs (n/a)
FTMBA degree page: -2.5%
PMBA degree page: +7%

46. What’s Next?
Summer 2013
• Usability testing with same tasks
• 2-3 additional tasks
• Test ad landing pages
• Test on desktop + mobile devices

47. Questions?
@jackiesauter