Presentation by Stephen Hole in collaboration with Christopher House (University of Wales: Trinity Saint David, Swansea) and Gavin Bunting (University of Wales) at the Research-Teaching Practice in Wales Conference, 10th September 2013, at the University of Wales, Gregynog Hall. Slidecast edited by Professor Simon Haslett.

1. It’s time to ‘Face’ the truth. Is Facebook’s
Survey Monkey a legitimate research and
pedagogical tool?
Christopher House, Gavin Bunting
and Stephen Hole
.
1

2. Survey Monkey
• Survey Monkey is:
–
–
–
–
–
Well known;
Free for small surveys;
Can niche non spatial data;
Special interest groups;
A growing mechanism of choice for undergraduate
research.
2

3. Sources Surveyed
• Faculty of Applied Design & Engineering:
– PhD Students (some with 1,000 + replies);
– MSc & BSc class exercises and dissertations;
– HEI staff awareness survey.
• School of Applied Computing:
– Approximately 80 undergraduate projects and
dissertations per year for the last 5 years;
– 20% Dissertations;
– 80% Projects;
• Approximately 50% of students will use a survey to
gather primary data;
• Numbers are growing year on year.
3

4. Surveys: Setting the Scene (1)
• Surveys are probably the most commonly used
data gathering instrument world wide.
• Students’ perceive survey based research:
– To be straight-forward;
– An easy means of gathering primary data.
• Students' often fail to consider:
– Response rate;
– Reliability;
– Validity.
4

5. Surveys: Setting the Scene (2)
• They do not expect to encounter
problems and often produce surveys
that do not:
–
–
–
–
–
Take into account survey design;
Ask the right questions;
Gather appropriate samples;
Provide data that can be meaningfully analysed;
Take into account relevant statistical analysis
techniques.
5

6. Surveys: Setting the Scene (3)
• Badly designed surveys when administered:
– Render the analysed results:
• At best inconclusive;
• At worst producing information that results in flawed
decision making.
• Due to the time frame of an undergraduate
dissertation there is limited opportunity to
redesign and administer a further study.
6

7. Survey Process Activities
• The survey process typically includes the
following activities:
–
–
–
–
–
–
–
–
–
–
Setting specific, measurable objectives;
Planning and scheduling the survey;
Ensuring that appropriate resources are available;
Designing the survey;
Preparing the data collection instrument;
Validating the instrument;
Selecting the participants;
Administering and scoring the instrument;
Analysing the data;
Reporting the results (Pfleeger and Kitchenam, 2001)
7

8. Student Survey Dilemma (1)
• Not looking before they leap:
– Data gathering equates to a questionnaire;
– A questionnaire equates to Survey Monkey;
– Sample population equates to Facebook,
Twitter and other Social Media contacts;
– Data Analysis equates to Bar Charts, Pie
Charts, Line Graphs etc.
8

9. Student Survey Dilemma (2)
• Not looking before they leap results in:
–
–
–
–
–
Poor quality data;
Poor analysis;
Weak conclusions;
Inappropriate recommendations;
Low grading.
9

10. Question & Data Types
• Open Format;
• Closed Format:
–
–
–
–
–
–
–
Dichotomous;
Range;
Radio Button;
Rank Order Scaling;
Multiple Choice;
Likert Scale;
Numeric.
• Question type will determine data type:
– Survey Monkey constrains the options and
flexibility of survey design.
10

11. Statistical Methods
• Parametric
– Numeric data that is from a normal (bell shaped)
distribution.
• Non-parametric
– Data is ordinal in nature and does not fit a normal
distribution;
– Data is reliant on some form of ranking and is not
numeric in nature;
– Likert Scales generate ordinal (non-parametric
data).
11

12. Typical Statistical Measures
• Parametric
–
–
–
–
Chi Square;
Linear Regression;
ANOVA;
T Test.
• Non-parametric
– Kendall’s Tau;
– Mann-Whitney U;
– Spearman’s Rank
Correlation Coefficient;
– Anderson-Darling;
– Kolmogorov-Smimov.
12

13. Recognising Data Types
• Students may not understand the
difference between parametric and nonparametric data and as a result:
– Use a parametric statistical test on ordinal
data;
– Draw inaccurate results from their analysis.
13

14. Student Checklist (1)
• Survey Monkey is a legitimate research and
pedagogical tool provided the following checklist is
used:
– Questionnaire must be a suitable data collection
method;
– Administration by use of a web interface will not skew
the sample population;
– Sample population must be identifiable:
• Not an invite to all friends on Facebook, Twitter, etc;
– Quality design principles are incorporated into the
questionnaire construction;
14

15. Student Checklist (2)
– Ethical approval has been granted;
– No questionnaire to be administered without
supervisors approval;
– Questions meet the objectives of the research
hypothesis;
– Collected data can be suitably analysed;
– Statistical measures used in the analysis match the
type of data being collected:
» Parametric for normal bell-shaped data;
» Non-parametric for ordinal data.
15

16. Student Checklist (3)
– Select appropriate statistical analysis
software:
•
•
•
•
•
SPSS;
Microsoft Excel;
Weka Data Mining Tool;
SQL Server Data Mining Tool;
Etc.
– Understand the output generated by the
software with sufficient background
reading.
16

17. P& K Enhanced Survey Guidelines
• The survey process typically includes has the
following activities:
– Setting specific, measurable objectives:
• Administration by use of a web interface will not
skew the sample population.
– Planning and scheduling the survey:
• Consider innovative completion drivers.
– Ensuring that appropriate resources are available:
• Less significant in SM surveys.
– Designing the survey:
• Question type/data.
– Preparing the data collection instrument:
• Consider technical synchronisation of software
packages.
17

18. P& K Enhanced Survey Guidelines
– Validating the instrument:
• Vary depending on level of academic qualification.
– Selecting the participants:
• Carefully following methodological practice.
– Administering and scoring the instrument:
• Controlled by SM.
–
–
–
–
Analysing the data;
Reporting the results;
Ethical approval has been granted;
Supervisory approval has been granted.
18

19. Case Study
• Handout to be supplied.
19

20. Any Questions?
20