2. Hello, I’m Kate
‘80s – bookish child of an IT dev and
a PM
‘90s – linguist, magazine writer
‘00s – marketer, journalist,
technologist, content strategist
I design content, and I design the
teams and systems that make it.
10. Testing enables content to become a
quantifiable, manageable asset with genuine
business value
11. The Challenge
is to create a content testing process
that will yield regular, usable data
12. Testing is part of a cycle, not a process
Plan
Make
TestReview
Publish
13. Testing content needs to be a repeatable exercise
Plan
Make
TestReview
Publish
Plan
Make
TestReview
Publish
Plan
Make
TestReview
Publish
Plan
Make
TestReview
Publish
14. It’s not the testing, it’s what you do with the results
Repeated testing builds
data around how content
works on your site, and
what you should be making.
That data needs to be part
of your decision making
processes
Plan
Make
TestReview
Publish
15. Old production processes + a bit of testing
won’t deliver the benefits you’d like.
You need a plan to change your processes,
before you start testing.
17. How I do testing
Image courtesy of Shutterstock
• Content testing principles
• Mapping the landscape
• Testing the things that matter
• Writing test cases
• Running tests
• Writing up findings
18. Content Testing Principles
Image courtesy of Shutterstock
• Content is there to do a job. Know what that job is, and test
for how well the content is doing that.
• Test by asking questions that are important to your key
people. These could be customers or the business, or both.
• Test using questions with definitive answers.
20. Test for the things that matter
Image courtesy of Shutterstock
• What is the significant influencing driver within the business?
• How do your testing goals align with that driver?
• What will you do with the results of the testing?
• Who will potentially be impacted by the testing?
21. Writing test cases
• Write tasks: one task per question
• Questions must have ONE definitive answer
• No clues in the questions
• Use plain language and common speech
• Be concise
• Make tasks independent from each other
• Make sure tasks don’t require confidential information.
22. Writing test cases: samples
• How many days holidays is a new, full-time employee
entitled to in UK offices?
• Who should be the first person you contact if you have an
accident while on site with a client?
• How many bottles of hair dye did the company sell in the UK
in 2015?
24. Running Tests
• Choose volunteers carefully: screen for prior knowledge and
bias.
• Give volunteers plenty of notice, and tell them in advance
what will happen.
• Stress that you’re testing the site, not them.
• Check your environment! Equipment, connections, facilities,
room bookings
• Have water, pens, paper, tissues, food available.
25. Recording results
• Use VIDEO! Jing is free, easy screen recording software.
There are many others, and for remote testing,
GotoMeeting has worked well.
• Ask participants to comment and narrate as they go
• Have one test observer sit beside the participant to guide
and lead the test
• Have another observer sit behind them to one side, so
they can look over their shoulder, and note down any
behaviours or comments.
26. Writing up results
Image courtesy of Shutterstock
• Use a meaningful format. Powerpoint, doc, video: choose a
format that will best deliver the message to the intended
audience
• Exec summaries are essential, and recommendations.
• If quoting from participants, maintain anonymity, and quote
accurately.
• Provide the data in appendices
32. Mapping the landscape
Image courtesy of Shutterstock
• A global intranet site hosting over 100,000 pages of content
• One (rather ancient) implementation of Sharepoint
• Thousands of employees, across multiple countries
• A head of employee comms fighting to get her message across
• No way of showing the impact that the intranet was having.
34. Mapping the landscape
Image courtesy of Shutterstock
How effective is the Bank’s intranet in supporting Global Standards? Does the
intranet help employees implement their day-to-day activities, or actively contribute
to risk within the Bank?
Cognifide, a technology and consulting firm, was commissioned to conduct content
usability research to look answering these questions.
The research was conducted between July and August 2016, and focussed on how
well Bank employees could use the intranet to answer 5 questions related to
Global Standards.
It found that, on average, employees were only able to find the answers to just over
half (51%) of the questions.
This means that for this test the failure rate was 49%, a significant exposure to risk
for the Bank.
35. Writing test cases
Image courtesy of Shutterstock
• Asking each user to find the answers to a set of questions relating to Global Standards.
They had to use the intranet only to complete the task.
• Questions were worded to have an unambiguous answer.
• Additional self-assessment questionnaire to provide a subjective view on how easy it was
to do the tasks
Outcome Meaning Metrics
Success The information can be found in a reasonable amount of time.
User can complete the task/give the right
answer in the allotted time.
Fail
The information cannot be found in a reasonable amount of time. User cannot complete the task/give the right
answer in the allotted time, or gives up.
Disaster The information is confusing or misleading.
User completes a different task/gives the
wrong answer.
36. Participants & Location
Image courtesy of Shutterstock
• Participants were recruited among Bank new joiners who had completed their mandatory
training. They took part anonymously and they are identified in this report by a progressive
number (Participant 1, Participant 2, etc.).
• Anonymity is a pre-requisite for this type of research, to ensure participants can freely
express their opinion. All participants were already familiar with the intranet, and most of
them stated that they use the intranet daily for their job.
• Sessions were run at the Bank in a common meeting room, without external observers, and
they were attended by the participants, one at a time. Two facilitators from Cognifide were
attending as well, one running the session and another one taking notes.
37. Running tests
Image courtesy of Shutterstock
Task # Question
Task 1 What are the goals of the Bank’s Global Standards?
Task 2
If you suspect an account is being used for money laundering, can you tell me one of the ways to escalate
your concern?
Task 3 Can you name three unusual activities that might indicate a customer is involved in money laundering?
Task 4 A customer wants to send money to their relatives in Syria: are they allowed?
Task 5
Following an Unusual Activity Report, a customer asks to see their financial crime risk rating. Can you
informally tell them?
38. Writing up results
Image courtesy of Shutterstock
• Report written, with exec summary, methodology, test cases, results per question,
overall results, outcomes and recommendations
• PowerPoint of recommended next actions, with timelines and rough costs
39. Results into Actions
Image courtesy of Shutterstock
• Data used to build business case for
$10m investment in a new intranet CMS
• Data used to acquire new funding for UX
review, Content Strategy work and new
intranet governance
• Repeat testing scheduled to show
changes following updates.
43. Repeat, often
Don’t overthink it
Hold your ideas
lightly
Expect to retest
monthly, quarterly,
yearly
Image courtesy of Shutterstock
44. Final things!
PLAN:
agree on what the goal is
plan how to get there
know what you’ve got to work with
know how you measure ‘good’
REPEAT:
keep moving on, like water
RUN:
brief producers fully
keep track of what’s live
own your errors, and successes
45. @ k a t e _ k e n y o n •
Kate.Kenyon@cognifide.com
Notas del editor
Content makes so much sense…and yet...why is it so hard?
Content is delivered by platforms, people and their processes. Platforms change easily: people and processes, not so much.
Before you can capitalize on the benefits of content, you need to work out if you have the right measurements for success – know what you’re making, why, and have a way to test whether it’s working or not. Today focusses on that last piece.
The aim is to define the ideal way of testing that will deliver the changes you need, at a pace the company can achieve.
In this session we look at what tools to use, where to look for risks, and models for new content success.
It doesn’t need to be a big plan, but do be aware that you need something.
Things I’ve cover now, and also some notes from past test cock ups that will help your future tests run well.
What’s the issue? What the wider goal here? What are you trying to change? No one tests for fun.
The questions were drafted and reviewed by the Global Communication team in charge of Global Standards. Cognifide consultants advised on the final wording. As all queries related to Financial Crime Compliance, they were ordered to minimise the risk of having the reply to one question implicitly replying to the following one.
The fact that all tasks required browsing the same section meant that the learning curve was likely to decrease with each task, and that implied a slight bias towards a positive result. On the other hand, the level of complexity also raised from Task 1 to Task 5, as the first three questions were strictly about Global Standards policies, while the last two were about implementing these policies in real-life scenarios.
Draw up a RACI of who will be involved in your testing – who decides the testing theme, who writes the questions, who signs them off, who reviews the results, etc.
By using a RACI you can also decide outcomes. If content performs really badly, what happens? Don’t test for the fun of it. Test for making hard choices.
Think of it this way – if you want to lose weight, run a marathon, do something different, you aren’t going to get it without change. You aren’t going to get it by just thinking about that goal.
The questions were drafted and reviewed by the Global Communication team in charge of Global Standards. Cognifide consultants advised on the final wording. As all queries related to Financial Crime Compliance, they were ordered to minimise the risk of having the reply to one question implicitly replying to the following one.
The fact that all tasks required browsing the same section meant that the learning curve was likely to decrease with each task, and that implied a slight bias towards a positive result. On the other hand, the level of complexity also raised from Task 1 to Task 5, as the first three questions were strictly about Global Standards policies, while the last two were about implementing these policies in real-life scenarios.