Trend Spotting Workshop. A practical guide to making sense of large information sources. Workshop run with Gemma Long (QAA) at etc.venues Maple House, Birmingham, 23rd February 2017.
General Principles of Intellectual Property: Concepts of Intellectual Proper...
Trend Spotting Workshop
1. Trend Spotting:
A practical training day to help you make
sense of large information sources
Marieke Guy and Gemma Long, QAA
Maple House, Birmingham
23rd February 2017
4. Agenda for the day
Time Session
10:00 – 10:30 Registration and coffee
10:30 – 10:40 Presentation: Introduction to day
10:40 – 11:00 Practical session: Introductions and aims for
the day
11:00 – 11:20 Presentation: Introduction to qualitative
(language-based) data & thematic analysis
11:20 – 11:50 Practical session: Sharing examples of
research analysis
11:50 – 12:20 Presentation: Bringing together sources
12:20 – 13:00 Lunch
5. Agenda for the day
Time Session
13:00 – 13:30 Practical session: Considering scenarios
13:30 – 14:00 Presentation: Carrying out data analysis
14:00-14:30 Practical session: Thematic analysis
14:30-14:45 Coffee break
14:45-15:15 Presentation: Outputs of language-based
analysis
15:15-15:45 Practical session: Comparing good and bad
report examples
15:45 -16:00 Presentation: Conclusions and resources list
Feedback sheets
16:00 Finish
7. • Help you better organise and make sense of large
information sources
• Help you carry out language-based research and
development
• Help you with market research
• Help you with business enhancement
• Help you with report planning
• Help you write better, more engaging reports
Aims for the day
10. Key terms
Term Definition
Project The piece of work to be carried out.
Research question Question that identifies what needs to be found out.
Collecting and
organising sources
The process of pulling together information sources and
organising them in some way.
Coding (into nodes) The process of pulling out sections of text and allocating
words or short phrases that give it meaning.
Thematic analysis Form of analysis in that involves examining and recording
themes.
Report writing Writing up a report that pulls together all the outcomes
from your analysis.
12. • Founded in 1997
• Offices in England, Scotland & Wales
• Higher Education Agency
• http://www.qaa.ac.uk/
About QAA
13. Our mission is to safeguard
standards and improve the quality
of UK higher education, wherever
it is delivered around the world
14. QAA strategic aims
Aim 1: enhance the quality and secure the academic standards
of UK higher education, wherever delivered, in order to
maintain public confidence
Aim 2: provide leadership, through knowledge and resources,
in assuring and enhancing the quality of higher education
within the UK and internationally
Aim 3: extend and enhance the value and reach of QAA’s
services, within and beyond UK higher education
16. QAA: future directions
i) Secure our role as the UK’s agency of choice for
higher education
ii) Develop as a client-focused organisation
iii) Be an agile organisation, able to adapt quickly
iv) Offer innovative services, for different market sectors
v) Deliver tangible benefits for the sector
QAA’s Board has begun the process of developing our
next strategy to 2020
17. • Non-subscriber events on:
Student Engagement
Managing Quality & Enhancing Quality
Professional, Statutory and Regulatory Body (PSRB)
Life after Degree Awarding Powers (DAP)
Transnational Education post EU
Degree Apprenticeships
• Quality Enhancement Network (QEN):
Two events in April on the theme of Entrepreneurship and
Apprenticeships (1 London, other tbc)
Two events in June on Postgraduate Research (1 Leeds, 1 London)
Quality Code consultation event in June
Upcoming events
19. • Introduce yourselves
• Who are you?
• Where do you work?
• What do you do?
• Share a reason why you are here
• Share an aim for the day
To the wider group
20. • One willing volunteer should empty their purse,
wallet or bag on the table
• Arrange and cluster the content into categories
• Label each pile
• Discuss
In small groups
25. • Information that is not in a numerical form i.e.
language-based data, descriptive data…
• Examples include: survey responses, diary
accounts, open-ended questionnaires,
unstructured interviews, unstructured
observations, collections of reports
• Often about interactions and relationships
• Analysis of such data tends to be more difficult
than looking at quantitative data (numbers)
Qualitative data
26. • To identify themes and patterns and share in the
form of reports
• To answer particular questions (or theories)
• To help inform decision making and business
planning
How is it used?
27. • Using data that you have access to as an
organisation to help guide decisions that improve
success
• Informed because should be based on more than
just numbers – contextualised and use staff
intelligence
• Important part of strategic planning
• Important to have data that backs up the decisions
that are being made
Data-informed decision making
28. • Anything more than you can easily read during the
work time available
• Perhaps more than 20 pages?
• It’s all about organisation and process
• It’s also about reproducibility and reuse
• Big data – volume, velocity, variety
• Tools, tools, tools…
What are large volumes??
29. • Why have you been asked to do this work?
• Who is it for? Who will see it? Where will it go?
• Is there an agenda behind it? Where are the
sensitivities?
• Who is leading on the work? What about sign off?
• What will be the output?
• What is the business enhancement purpose?
• How will success be measured?
Starting point
30. “If you do not know how to
ask the right question, you
discover nothing.”
W. Edwards Deming
31. • What do you need to produce?
• Who is it for?
• Is it for internal or external viewing?
• When should it be delivered?
• How long should it be?
• How can it be promoted?
End point
32. The template is designed to help you when planning
the production of a report or publication. It asks all the
questions you need to have considered before starting
work. It also ensures that you have thought through
the sign off process and considered the resources you
will need to publish the report/publication. The time
spent planning at the beginning should reduce the
time spent later on.
Publication report template
33. Name/title of
publication/report:
Purpose/objectives for the
publication
Why is this report being commissioned?
What does it hope to achieve?
How will the report benefit the organisation?
Is this report part of a contractual obligation?
Is this report part of a series?
Target audience(s) Who is this report being written for?
Who else will find it interesting?
Potential risks/issues for
this publication
Are there any controversial issues that are likely to be
raised in this report? Are there any sensitive areas
around this report’s subject?
Report writer Who is responsible for writing this report?
Budget code
Project manager Name of individual who will manage the project
overall.
Which format will the
report be published in?
Where will it be published? What format?
Word count/ no. of pages.
Key terminology Use of terminology.
Publication deadline Potential timing issues? Marketing lead times?
Project approval by
director
Signed: Date:
35. • Visual information management tools
• Method of storing, organising. Prioritising, learning,
reviewing and memorising information
• Stimulates creativity
• Helps you see bigger picture and detail
• Many software tools:
Mind maps
36. • What assumptions do you hold about the research
question?
• What values and life experiences may shape your
interpretation of the data?
Reflexivity exercise
38. • Think about the ‘research’ you have worked on in
the past that has looked at large information
sources
• Share one recent example with the group
• Each write down the following on a post-it note:
• One challenge of carrying out this work
• One lesson learnt of carrying out this work
• One success in carrying out this work
• Share your post-its on the wall
In small groups
41. • Existing reports
• Case studies
• Magazines, journals, newspapers, books
• Grey literature
• Web pages
• Databases
Using existing data sources
42. • Importance of taking a critical approach to
appraising sources
• Who wrote it or said it?
• Do they have an agenda?
• Were they given a template or series of questions?
• Questionnaire bias
Importance of context
43. • It takes skill – attend a course ;-)
• Think about what you want to find out
• Keep it short
• Think about your audience
• Use simple words and avoid being too formal
• Expand acronyms
• Avoid leading questions
• Balance not bias – Likert scale…
• Avoid double negatives
Writing a good questionnaire
44. • Use of more than one method of data collection or
research
• Bringing together quantitative (numbers) and
qualitative (words) data
• Most research ends up using some numbers
Mixed methods research
46. • Look at the two scenarios allocated
• Brainstorm a plan for the project manager
• To think about:
• Starting point – research question
• Middle – how can they analyse this data? What do they need to keep in
mind?
• End point – what do they need to produce as an output?
• What challenges can you identify?
In small groups
48. Two Approaches
Deductive Research Inductive research
Aimed at testing an idea Creating a new theory
Similar to grounded theory
(constructing a theory through
analysis of data)
Top down approach Bottom up approach
Theory driven Data driven
Have an idea in mind about
what you want to say or show.
Perhaps you have an agenda
already?
Am open to seeing what the data
shows.
Much more exploratory.
52. • Common form of analysis in social science
research
• Involves examining and recording themes
• Importance of organising data
• Key element is ‘coding’ – recognising important
moments in the data and highlighting them
Thematic analysis
familiarisation
with data
generating initial
codes
searching for
themes among
codes
reviewing themes
defining and
naming themes
producing the
final report
53. • Occur numerous times across the data – but
frequency not always related to importance
• Researcher judgement is key tool
• Try to avoid preconceptions
• Semantic and latent themes – look beyond what
people say – underlying ideas
• Themes and codes are different
Themes
54. • Trends are the general direction of travel: “our
customers are starting to prefer…”
• Patterns are series of data that repeats: “Time has
shown that customers like x”
• Trendlines
• Upwards and downwards
• Trend analysis
Trends and patterns
55. • Actively look for patterns within your environment,
industry and people’s behaviour
• Look at how information is structured
• Look for relationships between different pieces of
information
• Think about cause and effect relationships
Actively looking for patterns
56. • Things that are similar
• Things that are different
• Things that are frequent
• Things that are sequential or run in cycles
• Things that are opposite
• Things that are caused by one another
• Things that are in relation to one another
Recognising patterns
57. • Chronology
• Key events
• Settings
• People
• Places
• Processes
• Ideas
Things to look at…
58. What is a code?
“A word or short phrase that
symbolically assigns a summative,
salient, essence-capturing, and/or
evocative attribute for a portion of
language-based or visual data.”
Saldaña, J (2009). The Coding Manual for Qualitative Researchers.
59. • Gathering all the information about a topic together
for further exploration – you code into nodes
• Nodes can be topics, people, places, sections of a
report, positive feedback etc.
• Coding is heuristic
• Different projects require different approaches
• Need for consistency across projects
• Can be carried out in cycles
Coding
60. • Automated semantic analysis
• Good for very large volumes of data
• Provides numerical indication of relevance
• Highly inclusive and objective
• Clustering algorithm
• e.g. PTES work using
Leximancer
• Still requires researcher
interpretation
Concept mapping
61. • Think about the sentiments associated with a
concept
• Certain words connected with certain sentiments
• Favourable or unfavourable?
• e.g. difficult = unfavourable, enjoyable = favourable
• How will you deal with this?
Sentiment identification
62. • Call them what you like: log notes / self-memos /
summaries / reflexivity journal / notes
• Supports transparent process and team working
• Keep notes on:
Project aims: goals, assumptions, concerns, research question
Sources: where they are from, what is their remit, how they are organised
Project progress: why you have chosen the codes and theme, insights
etc.
Ideas for future analysis
• Make sure notes written in logs are different from
the data – different font etc.
Research logs
63. • Kick off meeting where you define terms and nodes
• Keep in contact – IM, Yammer, Skype etc.
• Regular catch ups – daily stand ups?
• Share what you have learnt – share early and often
• Log your findings
• Have a retrospective to share lessons learnt
• The benefits of different perspectives
Team working
64. • Nvivo – from QSR
http://www.qsrinternational.com/nvivo-product
• Atlas-ti – from Scientific Software development GMbH
http://atlasti.com/
• MAXQDA – from VERBI
http://www.maxqda.com/
• Leximancer
http://info.leximancer.com/
• Excel – Part of MS Windows
• Many tools out there – some open source e.g. RQDA
• None analyse the data – just help organise!!
Language-based analysis tools
65.
66. • Collect together categories
• Structure them in some way (perhaps using a mind
map)
• Think about headings and sub-headings
• Start thinking about the conclusion – what have
you discovered
• Think about key findings
• List your favourite quotations
Before writing up
68. • Look at the source material given
• Decide on your coding approach
• Start to code the text using the highlighter pens
• Cut out the coded content and place into piles
• Write a list of the codes you have identified on
post-it notes and label your piles
Individually
69. • Merge categories if appropriate
• Staple the piles of categories
• Arrange the categories on the table thinking about
hierarchy and structure
• Feed back to the wider group
In small groups
71. • Internal reports
• Summaries, recommendations
• External reports
• End of year review
• Grant applications/tenders
• Consultation responses
• Infographics
Outputs
72. “The value of an idea lies
in the using of it.”
Thomas A. Edison
73. • If you ask for feedback you should act
on it
• Pick the areas you can respond to
• Offer a strategy for dealing with them
• Don’t ask if you don’t want to hear the
answer
• “You said – we did” campaign
• #YouSaidWeDid
• e.g. at universities based on NSS
Feedback loop
74. • Reports look good with a few numbers in!
• Think about key stats from your project:
How many data sources?
When were they collected?
How many participants?
What percentage of overall participants was this?
Answers to any yes/no questions?
• Bar and pie charts
• Graphs and sparklines
• Tables
Combining with numbers
75. • Placing data in a visual context
• Helps users understand the significance of the data
• Want users to think about substance rather than
methodology
• Use the art of comparison: time-series, ranking,
ratios, deviation, frequency, correlation,
geographical location
• Dangers of spurious accuracy – avoid 34.567%,
use about a third
• Think about story telling approaches
Data visualisation with numbers
76. • Think about story telling approaches
• Word tags, bubble clouds, tree maps
• Word counts
• Venn diagrams
• Cluster analysis
• Using quotes
• Using photos and icons
Data visualisation with words
https://www.behance.net/gallery/7526739/Nineteen-Qualitative-
Data-Visualization
https://infogr.am/
77. • Infographics
Side by side
https://visage.co/turn-qualitative-data-visual-storytelling-content/
78. • Systematic approach to
presenting clear and
user-focussed information
• Focus is on layout
• Information is chunked up
into digestible chunks
• Uses banners, tables,
headings, margins etc.
Information mapping
79. • Answers the brief (or the research question)
• Consider the audience
• Well structured and coherent
• Offers clear examples
• Starts with an executive summary
• Ends with a sound conclusion
• Make appropriate recommendations
• Gives appropriate references
Writing a good report
80. • Honesty
• Agendas – being upfront about them
• Biases, values and judgements of researchers
• The messiness of reality
• Timing – familiarise yourself with the data
• Volume of data and work
• Prioritise the practical
• Conflicts of editing
Challenges
82. • Look at the selection of reports
• What works?
• What doesn’t work?
• List the possible faults of a report
• Think about the type of reports you create and
how they could be improved
• Feed back to the group
In small groups
84. • What have you learnt today?
• Did we cover all areas you wanted to hear about?
• What are your next steps?
Please fill in the feedback form!
To the wider group
Full image slides: A selection of full-page image slides are available to use. Images are available in Qmmunity, in the QAA Brand site. These slides are effective as divider slides, to introduce new sections, and headings can be added with a coloured panel behind the text to ensure legibility.
The coloured panel is 22cm wide with a 40% opacity. Text should be a maximum of two lines (for longer titles, use the full-colour slide). The transparent coloured background should match the colour palette.
QAA is the independent body entrusted with monitoring and advising on standards and quality in UK higher education
Our mission is to safeguard standards and improve the quality of UK higher education, wherever it is delivered around the world
We act in the public interest for the benefit of students and support higher education providers in providing the best possible student learning experience
We are dedicated to checking that the three million students working towards a UK qualification get the higher education experiences they are entitled to expect
QAA strategy (2014-17):
Aim 1: to enhance the quality and secure the academic standards of UK higher education, wherever delivered, in order to maintain public confidence
Aim 2: to provide leadership, through knowledge and resources, in assuring and enhancing the quality of higher education within the UK and internationally
Aim 3: to extend and enhance the value and reach of QAA’s services, within and beyond UK higher education
Future directions for QAA …
Important that QAA continues to evolve, to play a central UK and international role in the future.
Programme of change underway:
Organisational change (summer 2016)
Governance review
Ensuring QAA is well-placed to be the designated quality body under the new HE & Research Bill for England
Subscription consultation during 2016-17
Development of new services
M5 Group – efficiency savings and collaboration
To deliver the objectives set out here [see slide]
And our Board has begun the process of developing our next strategy to 2020.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
QAA is the independent body entrusted with monitoring and advising on standards and quality in UK higher education
Our mission is to safeguard standards and improve the quality of UK higher education, wherever it is delivered around the world
We act in the public interest for the benefit of students and support higher education providers in providing the best possible student learning experience
We are dedicated to checking that the three million students working towards a UK qualification get the higher education experiences they are entitled to expect
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.
Bullet points: Bullet point text should be at least 24pt to maintain legibility.
No more than four bullets points per slide is recommended. Keep bullets points concise. Don’t try to fit too much on one slide – use two slides instead.