Ioannis Andreadis presented research on who responds to website visitor satisfaction surveys. The research questions examined whether survey samples are representative of website visitors and whether surveys overestimate satisfaction. Andreadis analyzed data from a Greek voting advice application to compare survey respondents to all application users on characteristics like gender, age, and education. Some differences existed but were small. Satisfaction levels may be overestimated as more satisfied users are more likely to complete surveys. Andreadis concluded measuring computer literacy could improve representativeness but preliminary results support hypotheses that differences exist between users who start and finish surveys.

1. General Online Research Conference
GOR 13
March 04‐06, 2013, DHBW Mannheim, Germany
Ioannis Andreadis, Aristotle University of Thessaloniki
Who responds to website visitor satisfaction surveys?*
Contact: john@auth.gr
* sponsored by WEBDATANET
(COST Action IS1004)

2. Research Questions
1. Is an exit survey sample representative
of the population of all website visitors?
– Focus on visitor’s sex, age and education
level
2. Can we use website exit surveys to
estimate users’ satisfaction level?
– Do we get a better response rate by users
who are satisfied and (as a result) do we
overestimate the overall satisfaction level?
Ioannis Andreadis
Aristotle University of Thessaloniki

3. Voting Advice Applications
• Users read a series of
statements on
political issues and
they express their
level of agreement
with each of them
• Users can compare
their political views
with the positions of
the political parties
Ioannis Andreadis
Aristotle University of Thessaloniki

4. VAA Statement
• Example from
Wahl-O-Mat 2009
– Railway privatization
– German railway
operator Deutsche
Bahn should remain
entirely owned by the
state.
– Answers (Buttons)
• Agree, Neutral,
Disagree, Skip
Ioannis Andreadis
Aristotle University of Thessaloniki

5. Why a VAA can be used to test the
aforementioned hypotheses?
• HelpMeVote.gr is a Greek Voting Advice
Application different from Wahl-O-Mat
– Before the presentation of the result, users
are asked to fill-in a form with their:
– Sex, Age group, Education Level, Vote
Choice, Confidence
– Recently added field: Level of Computer and
Internet Experience
• Almost all users fill-in this form
Ioannis Andreadis
Aristotle University of Thessaloniki

6. Website exit survey
• Questionnaires are attached to outcomes,
i.e. each outcome can have only one
completed web questionnaire
• Outcomes are session based, i.e. the user
needs to close the browser or click a
special link to start from the beginning. In
any case the user has to answer the whole
series of VAA questions before
participating to the web survey.
Ioannis Andreadis
Aristotle University of Thessaloniki

7. Sample-Population Comparison:
Sex
Male
Female
0 20 40 60 80
VAA Population Survey sample
Ioannis Andreadis
Aristotle University of Thessaloniki

8. Sample-Population Comparison:
Age
>64
50-64
35-49
25-34
18-24
0 10 20 30 40
VAA Population Survey sample
Ioannis Andreadis
Aristotle University of Thessaloniki

9. Sample-Population Comparison:
Education
Postgraduate
studies
Tertiary
Secondary
Compulsory
0 10 20 30 40 50 60
VAA Population Survey sample
Ioannis Andreadis
Aristotle University of Thessaloniki

10. Odds ratio (Two logistic regression models)
male 1,27 1,09
18-24 (>64) 1,71 1,44
25-34 (>64) 1,85 1,62
35-49 (>64) 1,95 1,66
50-64 (>64) 1,49 1,37
Secondary (compulsory) 1,21 1,10
Tertiary (compulsory) 1,31 1,18
Postgraduate (compulsory) 1,51 1,36
male by substantive 1,15
18-24 by substantive 1,39
25-34 by substantive 1,34
35-49 by substantive 1,32
50-64 by substantive 1,15
Confidence by Secondary 1,11
Confidence by Tertiary 1,15
Confidence by Postgraduate 1,16
Constant 0,02 0,02
Ioannis Andreadis
Aristotle University of Thessaloniki

11. Sex – Vote Choice Interaction on
Web Survey Participation
7%
6%
5%
4%
3%
2%
Undecided Substantive Answer
Male Female
Ioannis Andreadis
Aristotle University of Thessaloniki

12. Estimated Satisfaction Level
(subset of respondents)
• VAAs rank parties according to their
proximity/similarity with the user e.g.: Party
A 80%, Party B 70%, Party C 30%:
• If the selected party appears first, the user
will be very satisfied
• If the selected party appears near the top,
the user will be somewhat satisfied
• If the selected party appears near the
bottom, the user will be dissatisfied
Ioannis Andreadis
Aristotle University of Thessaloniki

13. Sample-Population Comparison:
Satisfaction
Not at all
Somewhat
Very
0 10 20 30 40
VAA Population Survey sample
Ioannis Andreadis
Aristotle University of Thessaloniki

14. Conclusions
• Small differences between sample and
population regarding Sex, Age and Education
– Significant interactions: (Sex and Age)*Vote choice,
Education*Confidence
– 6th ISM workshops: Would it be better if I had a
measure of computer literacy? Preliminary results
appear to support this hypothesis (but, there are
differences between those who start and who finish)
• Satisfaction
– Overestimation of satisfied users and underestimation
of unsatisfied users
Ioannis Andreadis
Aristotle University of Thessaloniki

15. Thank you!
Ioannis Andreadis
Aristotle University of Thessaloniki
john@auth.gr
* sponsored by WEBDATANET
(COST Action IS1004)