Presentation by Renee Kart, Director of Project Strategy at Simpson Scarborough, given at the 2011 SUNYCUAD Conference held in Saratoga Springs, NY on June 9.

1. Will They Love You Tomorrow?
Conducting Effective Alumni Research
Renee Kart
Director of Project Strategy
SimpsonScarborough
June 9, 2011

2. What We Want to Know
• What type of relationship do alumni seek to have with our
institution?
• How can we do a better job of engaging alumni through
communications and activities?
• What services do alumni need and want from us?
• How do our alumni want us to communicate with them?
How do they want to communicate with us?
• How effective are our institution’s print and electronic
communication strategies?
• How do our alumni feel about their experience at our
institution?
• What messages about our institution build a sense of
pride for our alumni?
• Are there ways that we can model alumni engagement to
make better decisions about resource allocation?
• What types of events will strengthen relationships with
alumni?
• How should our institution segment our alumni to
maximize the effectiveness of our efforts to engage them?
• Who are the leaders among our alumni?
2

3. Establishing Guiding Questions
• “Guiding Questions” are the 5-8 broad questions
you want your research to answer
• If you have more than 8, your study probably Many survey research studies try
lacks focus to gather information about too
• Make the hard trade-offs (a “must know” vs. many topics rather than delving
“nice to know”); no laundry lists deeply into one or two pressing
• They are not actual questions that would be asked issues.
during a focus group, in-depth interview, or survey
• Imagine they could be the section headers on your After you develop your guiding
PowerPoint of the findings questions, you will be able to
• Get buy-in from other stakeholders decide if qualitative or
quantitative research (or both) is
• List the guiding questions on all communications best to address your needs.
regarding the project; include them at the top of
every draft of your moderator guide, in-depth
interview script, or survey instruments
3

4. Choosing Research Methods (1 of 3)
Research Methods
Focus Group
Qualitative (In-Person or Online)
In-Depth Interviews
Direct Mail
Quantitative Phone Survey
Online Survey
Alumni Network Analysis
Other Segmentation Analysis
Discrete- Choice Modeling
4

5. Choosing Research Methods (2 of 3)
Methods of Quantitative Research
Method Advantages Disadvantages
• Less bias • Significantly longer There are pros and cons to every
• Respondents work response time form of data collection.
Direct
at own pace • No probing
Mail
• Ensures anonymity • No control over
• Less expensive response pool The decision on which method is
• Strong response • No visual aids best for your study requires you to
rates • Difficult to accept the limitations of your
• Quicker than direct establish rapport chosen approach.
Phone
mail • Interviewer bias
• Sequence is flexible • Overused
• Allows for probing • Expensive
• Quickest response
time • Difficult to ensure
• Generally less representation
Online
expensive • Difficult to identify
• Allows for complex usable sample
branching
5

6. Choosing Research Methods (3 of 3)
Example of an actual research plan for a project currently underway.
Research Plan
Target
Sampling Frame Qualitative Research Quantitative Research
Audience
4 online focus groups (2
outside NYC-area
undergraduate alumni/2 Online Survey
outside NYC- area
All living alumni of
graduate alumni) (Discrete-choice
undergraduate and
Alumni modeling and
graduate programs with segmentation analysis
2 in-person focus groups
an email address on file will be included in online
(1 NYC-area
survey)
undergraduate alumni/1
NYC-area graduate
alumni)
6

7. Focus Group Basics
• Used to understand our audiences:
• Feelings, perceptions, and needs
• Motivations to give/not give Focus groups are generally more
• 8-12 members of target audience gathered for an effective than surveys at exploring
open-ended discussion about topic of interest emotional drivers.
• One of the most frequently used market research
techniques Focus group generate qualitative
• Typically lasts 1 to 1 ½ hours; no longer data which means the findings
are presented as key themes
• To ensure accuracy, conduct multiple groups rather than through the use of
• Always audio or video tape statistics, as is the case with
quantitative research.
• Typical project has at least four groups
• Same target audience, same guide, same
moderator, etc.
7

8. Focus Group Examples
• Fordham University
• 4 online focus groups (2 with outside NYC undergraduate alumni; 2 with outside
NYC graduate alumni)
• 2 in-person focus groups (1 with NYC-area undergraduate alumni; 1 with NYC-area
graduate alumni)
• Used as precursor to quantitative analysis
• MIT Sloan School of Management
• 4 online focus groups (1 with domestic undergraduate alumni outside Boston
area, 1 with domestic graduate alumni, 1 with international undergraduate
alumni, 1 with international graduate alumni)
• 2 in-person focus groups (1 with undergraduate alumni in Boston area; 1 with
graduate alumni in Boston area)
• Used as precursor to quantitative analysis
• Alfred University
• 4 online focus groups with alumni from the 1970’s and 1980’s (2 with alumni in
western NY; 2 with alumni outside western NY)
8

9. In-Person vs. Online Focus Groups
Advantages • Formal, traditional focus groups are
conducted in a focus group facility with
In-Person Online two-way mirror
Ability to observe Cheaper incentives • Online focus groups are becoming an
body language can be used industry standard
Viewers can observe Save on cost of
• Online focus groups typically involve:
using facility with focus group facility
two-way mirror • The use of “webinar” software like
WebEx or Microsoft Live Meeting
Stronger “show” Shorter recruiting
rate time • Participants dial into a toll-free
number and log onto a secure
Don’t have to deal No geographic
server at the same time
with “technical limitations
difficulties” • The moderator can push
content/visuals via the computer
More flexibility to
screen and take polls
use exercises and
worksheets • Audio/video recorded and accessed
online
9

10. Developing the Moderator Guide (1 of 2)
• Must contain thoughtfully crafted questions to get
the responses you need
• Rushing through a long list of questions will Most moderator guides are no
not yield in-depth insights more than two pages long.
• Developing these questions is a unique skill
learned chiefly through experience
The moderator guide should be
• Open with simple, easy, fun, and compelling filled with open-ended questions
questions that get the group participating or ones that begin with “how,”
• Address general questions toward the beginning, “why,” “what,” “where,” and
specific questions toward the end “when.”
• Use exercises to break up “straight questioning”
and avoid “group think” (polls, sticker book, role A question that should be asked
playing, visual aids/websites) frequently during the group is,
• Moderator’s role is to develop the right questions “Why do you say that?”
and facilitate the discussion to get at this
information
10

11. Developing the Moderator Guide (2 of 2)
Excerpt from actual moderator guide.
11

12. Analyzing Results & Report Preparation
• Have the moderator and another reviewer listen
to all tapes; compare notes
• Focus group findings are typically presented as Clip from actual focus group.
bullet points in a PowerPoint Question about decision to move
• Report key characteristics of participants to printed Alumni Magazine to
provide context online.
• Key findings are presented as themes; report only
those themes that emerge very commonly across
all groups
• Report all interesting non-themes as isolated
opinion or “good ideas”
• Avoid temptation to present focus group findings
using charts, graphs, and/or tables
• Use direct quotes and audio/video clips to
reinforce key themes
12

13. Survey Research Basics
• Close-ended questions
• Conducted over the phone, face-to-face, direct
mail, or online Quantitative research is about
• One of the most frequently used market research getting the hard measures of a
techniques market
• Typical instruments last 10-15 minutes, but alumni • How important is…
can withstand up to 20 minutes • What is your level of interest
• Good questionnaire design is extremely important; in…
wording can dramatically change measurements • What is the likelihood you
• Typical project has one online survey will…
13

14. Stimulating Participation
• Sell value of research
• Guarantee confidentiality
• Avoid non-vital questions and long questionnaires Incentives are a critical condition
• Avoid questions you already have the answers to in when working to generate survey
your own data records response.
• Use an attractive design and layout
• Use preliminary notification Offer an incentive that your
target audience values and be
• Utilize reminders/call-backs creative.
• Include return envelope with postage
• Use incentives; cash is the universal incentivizer
14

15. Sampling
• Generally, seeking to secure a “representative
sample” of our target population
• All else being equal, a larger sample size n If you want 400 completed
leads to increased precision in estimates of surveys and expect a 40%
various properties of the population response rate, select a sample of
• But, larger samples are not necessarily better 1,000.
because of the opportunities for non-sampling
error
• To achieve desired sample size (n), make an
educated guess as to what your response rate
(r) will be; then select a sample that is n/r
15

16. Response Rates
• Response rates directly linked to your target audience’s interest in you and your topic
• The higher their interest, the higher your response rate
• Typical response rate for alumni is between 15% to 25%
• Online tools such as Raosoft will generate recommended number of respondents
determined by population size, desired margin of error (typically 5%), and desired
confidence level (typically 95%)
16

17. Designing Survey Instruments
• Exclude interesting, but not vital, questions
• Pay close attention to design, layout, and perceived ease of completion
• Avoid leading or ambiguous questions; maintain objectivity
• Open with simple, interesting questions
• Progressively narrow the scope of questions
• Place difficult, sensitive, or complex questions toward the end
• Include personal information/demographic questions last
• Plan time into the research process for extensive re-writing of the instrument
• Ask the HARD questions
• Typical instruments should not take longer than 10-15 minutes, but alumni can
withstand up to 20 minutes (regardless of method of collection)
17

18. Structuring Survey Questions (1 of 5)
WHAT’S WRONG HERE? SOLUTION
Q1. How interested are you in attending Q1. Rate your level of interest in attending the following.
Reunion 2012? 1 = Low
2 3 4 5
6 = High
interest interest
• 1 = Low interest Reunion 2012
• 2
• 3 Homecoming 2012
• 4
Sports outing with
• 5 fellow alumni
• 6 = High interest
Cultural outing with
fellow alumni
Q2. How interested are you in attending
Homecoming 2012?
• 1 = Low interest Group like questions together to
• 2 reduce respondent burden.
• 3
• 4
• 5 Respondents can get
• 6 = High interest “burned out.”
18

19. Structuring Survey Questions (2 of 5)
WHAT’S WRONG HERE?
“Select your level of agreement with this statement. I am proud to be
an alumnus of SCHOOL because it provided me with job training,
allowed me to meet new people, and looks good on my resume.” Three questions are being
asked here.
• Strongly disagree
• Disagree
• Agree
• Strongly agree
SOLUTION
“Select your level of agreement with this statement. I am proud to be
an alumnus of SCHOOL because it provided me with job training.” Be specific and isolate
• Strongly disagree issues.
• Disagree
• Agree
• Strongly agree
19

20. Structuring Survey Questions (3 of 5)
WHAT’S WRONG HERE?
“How far would you be willing to travel to an event sponsored by
SCHOOL?”
• Less than 50 miles
• 50-100 miles These response options are not
• 100-150 miles mutually exclusive.
• 150-200 miles
• More than 200 miles
SOLUTION
“How far would you be willing to travel to an event sponsored by
SCHOOL?”
• Less than 50 miles
• 50-100 miles Structure answer options so they
• 101-150 miles are independent of one another.
• 151-200 miles
• More than 200 miles
20

21. Structuring Survey Questions (4 of 5)
WHAT’S WRONG HERE?
“What college/school within INSTITUION did you receive your degree
from?” _________________________
This is too broad; respondents
might use jargon or acronyms.
SOLUTION
“Select which college/school within INSTITUION you received your
degree from. ”
• College of Arts and Sciences
• Arthur Miller School of Business
• Nancy Peete School of Education This allows you to control the
• College of Engineering responses, which will result in
better data analysis.
21

22. Structuring Survey Questions (5 of 5)
WHAT’S WRONG HERE?
“It is a good idea for SCHOOL to stop printing the Alumni Magazine.”
• Agree
This question includes an applied
• Disagree
assumption.
SOLUTION
“In your opinion, should SCHOOL transition the printed Alumni
Magazine to an online format?”
• Yes This questions asks, rather than
• No assumes, the opinion of the
respondent and is clear about the
implications of the resulting data.
22

23. Data Analysis
• Think about your data analysis as
Question Data Analysis Plan
you are preparing your instrument
Indicate which information
• Create a “Data Analysis Plan” as about SCHOOL you would
Analyze results to build
strategy for print and
you finalize your instrument be most interested in
electronic communications.
hearing about.
• Create “dummy tables” which
Rate the likelihood that you
mock up the tables and charts you Analyze results to build
will participate in each of
strategy for on-and off-
will prepare once your data is the following activities in
campus events.
collected and analyzed the next 2 years.
• Include all information the reader
needs to interpret the finding
accurately when you are not
around
23

24. Alumni Network Analysis (1 of 3)
• The true leaders of your alumni pool may not necessarily be the ones who are current
engaged with your institution
• Alumni network analysis reveals and nurtures the existing social networks among
alumni
• Alumni are significantly influenced to attend or ignore events based on who is attending
the program
• Alumni are asked to identify their most cherished relationship with peers as well as
other at your institution
• Data is aggregated and the social network analysis, in addition to traditional statistical
analyses, is conducted to reveal the maps of relationships among alumni
• Using the information revealed by this analysis the ones who are connected to the
greatest number of fellow alumni can be tapped for event planning and other initiatives
24

25. Alumni Network Analysis (2 of 3)
25

26. Alumni Network Analysis (3 of 3)
Mr. Barry P is
a “leader”
because he is
connected to
many other
alumni.
26

27. Segmentation Analysis
• Segmentation analysis takes the
alumni’s attitudes about the
institution (specifically their types
of previous interaction and level of
interest in future interactions) and
seeks natural groupings of alumni
based on those attitudes Biggest Fan Quiet Supporter
Alumni Segments
Unengaged Critic
Supportive and Supportive but not Unengaged and not Unengaged and
• Those segments can be examined engaged engaged paying attention intentionally not supportive
with an eye towards identifying
how they differ, both on the
attitudes and all other measures of
Segment Characteristics
alumni captured in the research •
•
3+ alumni events in last 2 yrs
Member online alumni commy
•
•
Hired MIT Sloan alum
Lives in Northeast
•
•
MBA graduate
Lives in Midwest or abroad
•
•
MBA graduate
Lives in Midwest or abroad
process •
•
•
Graduated 2000+
Lives in Boston
Email address on file
•
•
Graduated <2000
Email address on file
•
•
Earns <$150K annually
No email on file
•
•
Low student satisfaction rating
No email on file
• This helps you to understand the • 18 contacts per year •
Communication Flow
16 contacts per year • 12 contacts per year • 5 contacts per year
• News updates • News updates • News updates • Selected news updates
groups based on their differences •
•
Social and networking invites
Invited to mentor students
•
•
Networking invites
Invited to meet with students in
small groups
• Invited to review resumes of
graduating students
on a number of different
attitudinal, behavioral, and
demographic characteristics
27

28. Discrete-Choice Modeling (1 of 2)
• Discrete-choice modeling describes In-Person Opportunity
consumer decisions as they would be in Factors Levels
the “real world” where we always make For making professional connections
Why
Primarily social in nature
“trade-offs” in our decisions An opportunity is
To increase my knowledge about topical
offered
issues
• By observing respondent choices, you Other SCHOOL alumni
can determine: Who
Other SCHOOL alumni and faculty
Other SCHOOL alumni and current students
• What is important to them in the attends the
opportunity
SCHOOL President
Other SCHOOL alumni and non-SCHOOL
decision-making process regarding speakers
attendance at in-person and online Monday through Thursday evening
between 5 p.m.and 9 p.m.
events and opportunities Friday, Saturday, or Sunday evening
When
• Each respondent is exposed to four the opportunity
between 5 p.m. and 9 p.m.
Saturday or Sunday between 12 p.m. and 5
scenarios with two options each occurs
p.m.
• They are forced to choose the activity in Monday through Friday morning between 8
a.m. and 10 a.m.
which they would be most likely to Where Less than 30 minutes from your home
participate an opportunity 30 minutes to an hour from your home
occurs 1 to 2 hours from your home
• One level within each of the factors 0-$10
identified to the right were included in How much
$11-25
$26-50
each option it costs
$51-75
Over $75
28

29. Discrete-Choice Modeling (2 of 2)
29

30. How Much Will It Cost?
• Cost influenced by
• Method of data collection
• How much qualitative and quantitative Alumni research can cost $35,000
research is done to $150,000 for research strategy
• Amount of work you are able to do and development.
internally
• In-house vs. Outsourcing
• What expertise exists internally?
• Can internal provider be objective?
• Can internal provider meet ambitious
deadlines?
30

31. Resources
• The Green Book – http://www.greenbook.org/
• Sample Research RFPs - http://simpsonscarborough.com/resources/sample-requests-
for-proposals/
• Survey Monkey – http://www.surveymonkey.com/
• Question Pro – www.questionpro.com
• Inquisite Online Survey Software – http://www.inquisite.com/
• Qualtrics Online Survey Software - http://www.qualtrics.com/
• Marketing Research Association – http://www.mra-net.org/
• Survey Sampling – http://www.surveysampling.com/
• International Code of Marketing and Social Research Practice –
http://www.esomar.org/uploads/pdf/ESOMAR_Codes&Guidelines_ICCCode.pdf
• Raosoft Sample Size Calculator - http://www.raosoft.com/samplesize.html
31

32. Thank You!
32