As HCI researchers, we are taught and encouraged to find fault with solutions in exhaustive, scientific detail. This approach is essential in our origin field of industrial human factors, but in a creative environment producing a long list of problems is rarely useful or inspiring. In this talk, Andrew addresses some of the problems he's seen in presenting the results of UX research, and draws on the art school method of critique to illustrate some alternative ways for researchers to engage with designers to help explore what could be next.

1. Hi everyone. I’m Andrew Harder and I want to talk to you about how researchers
can have a better impact when working with designers.
I work for Nokia Design in London. Previously I’ve worked with them as a design
researcher on pre-roadmap concepts like the 2012 range of phones; and right now
I’m running a Nokia-wide UX evaluation process for products that are launching as
soon as next month.
Before that I worked for an agency
1

2. And I want to talk about the change in perspective that I had when I made this jump that I made
from being agency-side at Flow delivering research, to going to Nokia and managing external
researchers to deliver work that I would then use to try and be useful to designers and Nokia’s
design process.
Massive shift in my perspective about what good research looks like, and how research can be useful
to design, when researchers THINK they’re being useful to design and they’re not and when
research can totally hijack design.
I think it starts with education.
As a researcher I did an HCI masters in London, and what we learned boiled down, to me, to being
exposed to evaluation techniques fundamentally based on cognitive psychology or ergonomics.
Techniques like user tests or cognitive walkthroughs draw directly from cognitive psychology.
And the ultimate result of these kinds of evaluation projects is a big list of issues with a design. You
know, it’s a list of what is wrong with the current state of things.
The designers I worked with at both Flow and Nokia have a completely different approach to their
work – that is broadly speaking a seen is a focus on the generation and development of new ideas
that resonate with social need.
So as we all know this can be a great pairing – one capability focusing on what is wrong now, and
one focusing on what can be done next.
But often the results of research and evaluation aren’t presented in a way that can help designers
take the next step, and actually in my opinion show little respect for the role of designers and their
responsibilities. This is what I want to talk to you about today – how we can work well together,
trying to draw some lessons from the pitfalls that I’ve seen myself as a researcher fall into, then see
researchers that I’ve managed fall into too.
Reasons for this:
•What kind of people go into research
•The research process, and what it does to researchers
•The education that we get and tools that we use,
But first I want to consider the issue of leadership in the field. So I’ve picked Don Norman to start
with
2

3. Look at that face, who could find fault with him?
Stating the obvious, Don Norman is the guru of modern usability
All but created the field with Design of Everyday Things
Personally inspired me to join this field
Undisputed leader in UX
3

4. Even Newsweek says so. Well his website told me that Newsweek said that at least.
So as well as being an HCI practitioner, he is also, you might say primarily in public
an educator and an opinion leader.
His very successfully evangelises our field
Laid the basic frameworks of UX in ways that are still relevant today
Now also deliberately provokes us to think critically about what it is that we do.
Before I tell you what my beef is with his work, let me take you on an aside.
Like I’ve said, this talk comes from my own experience and seeing common
attitudes in researchers
Problem is that I can’t share specific client examples, so I was trying to think about
how I could ground this talk in some way so everyone could understand my point
Then I found some this essay of his
4

5. This piece he wrote.
Trapped in a Lufthansa Airline Seat – Don Norman
Perfectly encapsulates the worst of presenting research. So when I found this, I was
so relieved. Being a nearly digital native, feeling some emotion at my desk made me
need to express it
5

6. On Twitter. I was so relieved. Imagine my surprise when the next day on my way to
work I saw this…
6

7. Stupid Twitter!
So I kind of feel like Don is in the room with me now. And before all of you go
tweeting “thevagrant hates the Don” let me make one thing really really clear
I love the Don. I love him.
But he’s not perfect.
7

8. And articles like this drive me up the wall.
It’s basically a documentation of the problems he found with this Lufthansa Airline seat.
Trapped in a Lufthansa Airline Seat – Don Norman
The audio and video selections are generous even though it took me five minutes to figure
out how to get the menu. There is a remarkable variety of video available, and each one
seems to be playable on demand… (As far as I can tell, once started, it cannot be paused,
fast forwarded or backed up, but maybe I just didn’t find those controls.)
And there is even a hidden panel for computer support: regular AC power… It even has
jacks labeled Internet and USB. The person across the aisle soon discovered that the
internet is not yet implemented. “Real Soon Now,” is the promise, but hey, give
Lufthansa credit.
There is even storage space for shoes. It’s wonderful. Except it doesn’t work. We were on
a ten-hour flight, so my wife and I took off our shoes…. But only one shoe would fit into a
storage space, and there were only two spaces for each pair of passengers — two spaces
for four shoes. Maybe German use smaller shoes than Americans? More likely, that’s all
the room they had.
And you know that nice, reclining bed? It doesn’t disturb the neighbors, but it certainly uses
every inch of space in the compartment. Leave a pillow, blanket, or heaven forbid …your
computer laptop in it on the floor and, crush, grind, scrunch. No more pillow, no more
blanket, no more laptop. “Please do not leave hand luggage on the floor when using this
position,” says the manual next to the heading “Sleep.” But where does the hand luggage
go? Up into the overhead compartment? Pain. Between the two seats? Maybe, but the
space is only about six inches wide, occupied by the two shoes we couldn’t fit into the
platform. Sure the briefcase can fit on top of the shoes, but if it leans even slightly to one
side, it gets crunched.
8

9. So all in, these are all the problems he finds with this airline seat.
This is basically what the article is about, this one-sided list of problems. He tries to
balance some positives by says that he’s praising Lufthansa for trying hard, but I
have to say my reading of it never gets far away from sarcasm.
The problem I have with this is that it epitomises a form of writing that is
considered legitimate in our field. It is somehow acceptable to write articles like this
that describe a user’s experience of a product or service and just list fault after
fault. I’ve used Don’s work because as a leader in our field he legitimises this as a
valid outcome of a UX review.
At a basic, fundamental level, by reading articles like this especially when we are
learning our trade, we implicitly believe that this is an acceptable way for us to talk
about a product or service.
My point is that this is NOT acceptable.
This type of discourse is ugly, uninformative, its tone and presentation style drives
good designers and service providers AWAY from us as UX researchers. In Australia
we’d call this whinging. This is nothing more than complaining about a service. And
complaining is not useful in a design environment.
And as researchers, it is far, far too easy for us to drop into complaining about
something rather than having a more impactful approach.
9

10. These are the bad habits & practices that we get into that can make it feel like we’re
complaining.
Tells everything – We want to say everything that we found
Suggests solutions – We do this to be helpful, but this is a problem
Ignores – Because it wasn’t in the test, right? Users didn’t see it.
Documents and formalises – Because we don’t want to forget anything, and we
want everyone to remember everything
Shows how smart we are – We learned a lot, and we want to be valued by our peers
Introduce the critique:
Designers has a highly developed practice for address problems in their work, the
design crit that is a fundamental skill taught in design schools.
•Two skills, giving the crit and taking the crit, they’re both very important. But I’m
only going to talk about GIVING the crit today
As researchers we need to learn more about giving critique, because that’s more
often the side we’re on
10

11. Critique
Fillet the findings – We only talk about the VERY MOST IMPORTANT THINGS
Creative Authority – Even if they don’t have it in the rest of the organisation, we
need to elevate designers to the people who can respond to our research
Intent – We research, analyse and talk about the hidden ideas about what is
BEHIND the prototype or system that we test
Interpersonal – A critique is ONLY EVERY interpersonal
Makes everyone else feel smart
Talks about the design and the qualities of the design, not the endlessly complex
experiences users can have.
11

12. So where do these differences come from? There are two basic models that explain
A LOT of this.
Common model, Preece Rogers & Sharp Interaction Design book, written in an HCI
faculty at Sussex University
•Research plays a primary role, but the easy implication to draw is that evaluation
and needs are the only thing that the design needs to follow.
•No brand, no portfolio of other products, no strategy, no COMPETITION etc.
•This was outside the scope of their book, but that’s part of the problem. We’re
training people to ignore the most important parts
The biggest fallacy I see people reading into models like this is that research can
provide all the answers
•Research provides the requirements and identifies the gaps between the prototype
and what users want.
• This model IMPLIES that successive iterations of design exist to address and
remove usability problems
•Design, therefore, is an activity that introduces error into this process
12

13. Immediately this is a different model of design – no flow chart for a start!
Problem – initial brief or problem, stimulus to first round of activity. It’s a single point so it
might be as little as a sentence, but it’s certainly not a thesis.
Discover is a phase that searches for ideas that respond to this brief.
•We deliberately explore multiple different avenues on how to respond to the brief
•What are the possible dimensions to explore this brief?
•What are the underlying user needs?
Define aligns the discovered user needs to the basic business objectives & project
management
Ends in the problem statement, a second brief about what needs to be done.
Develop – multiple design solutions are developed, iterated and evaluated
Deliver – resulting design is finalised and launched in the market.
Primary focus of this model is to scope and define the exploration of ideas in the process.
This view also shows the nature of design ideas as they progress through their
development.
HCI is strong at the converging parts of this diagram.
Loads of methods, approach, practice, publications, and so on.
Lots of methods, even ethnography has been brought out in this
We have very little to say about the diverging parts of this diagram.
That previous model says we have to gather requirements, in the form of needs – but it
implies that researchers are in charge of this phase of the project and that needs are just
out there waiting to be discovered.
We can apply the same methods that we apply in the diverging sections, without a
prototype – look at behaviours, identify pain points, etc. This is all useful stuff, but the real
value of this divergent phase – looking at what CAN BE DONE we have little to say about.
This is what, at the moment at least, I believe that designers uniquely bring to the table.
Their training in generating and developing ideas
Patience with ideas
13

14. Not everyone thinks these diverging parts of the diagram are good and that starting
from a single point rather than a thesis is good.
14

15. The Don again.
Remember, I love the man.
I love him so much that I love arguing with him even more.
15

16. Published on Core 77 last year – you might have read it
He makes no statement of the problem beyond his problems with design
competition winners, and a dogmatic view that everyone should understand
cognitive science. He also gives no examples of any of these designs he finds naïve.
So really, as an argument, it’s total horseshit.
But that last paragraph that I’ve snipped rings some bells – designers don’t
understand the complexity of the issues and the depth of knowledge already
known.
Researcher’s argument, it prioritises the explicit reading-knowledge that
researchers like me value
Let me tell you a bit of a secret – in ANY multinational consumer company, they
are drowning in research. They have just too much research. And yet we always
have to commission more to get what we need to know!
In particular, it is his view that not understanding the last degree of complexity in a
situation is an indefensible failure that is the most illuminating.
This is the complaint of someone who passionately believes in that iterative
process of research, and is deliberately marginalising the ideation work.
16

17. But there are appropriate questions to ask at each stage. Understand the purpose
of the design activity that’s happening in the moment and take it from there.
We need to know when to mix in the complexity of the real world, and when to
protect the design process from all the challenges that the real world presents.
17

18. ... But... There are some researchers who don’t agree with The Don either
18

19. In this paper, Usability Evaluation Considered Harmful (Some of the time)
19

20. We’re not trained how to deal with ambiguity in interfaces, we can criticise them in
light of being perfectly mature interfaces which they aren’t.
The double diamond model helps us situate them in their purpose during the
design process.
Radical new ideas may be immature
If there are social or cultural issues, today’s rejection may be tomorrow’s
acceptance
20

21. Which brings me back to my original question
What do we do?
21

22. So, a tangent to make this point. I’m going to tell you a story about two authors,
Ernest Hemingway and Kurt Vonnegut. One of Hemingway’s definitive works is
The Old Man and the Sea, which is a novella about Santiago, a very old
fisherman
22

23. In the story, Santiago goes out to sea, much further than other fisherman go, and engages
in a herculean struggle to catch an enormous marlin. He thrashes all day and all night, the
fight nearly kills him, but eventually he catches the marlin only to discover that it is too big
to fit into his boat. Because he can’t imagine not landing this fish, he strats it to the outside
of his boat and beings rowing back to shore. The problem is that there are sharks in the
water, and they begin attacking the marlin to eat it. The fisherman tries to fight them off,
but he’s already exhausted by catching this fish and he’s no match for them. By the time he
gets back to short, these sharks have eaten all the fish, all that’s left is a bit of skeleton, and
this being Hemingway he’s scorned and ridiculed by the people who have stayed at shore,
for doing all this struggle and then having nothing to show for it. .
So another author, Kurt Vonnegut took up this story and offered a meaning for it.
Hemingway published The Old Man and the Sea 10 years after his previous novel, which he
had struggled to finish but he considered the great work of his life – but the critics
lambasted it and ridiculed it. So in Vonnegut’s version, the old man was Hemingway
struggling to finish his life’s work, going out on a mammoth task that nearly kills him,
catching a prize fish but having the critics as shark decimate his work before he could land
it.
And I think in any large research project I’ve been in a similar situation – this might sound
odd to non-researchers, but even a 10 person user test can sometimes feel like an odyssey,
meeting so many new people, learning about their attitudes, behaviours, even lives, and
my head is swimming with the richness of what I’ve learned and the possibilities of what
could be done in response to this information. If you haven’t done user research it might
sound weird, but even coming up with a list of usability bugs can be really exciting and
inspirational.
23

24. The struggle is then to land this big catch of information and insight back to shore
where clients can use it. And if I look back to my time in agency,
Still remember trying to deliver 100 page powerpoint, met with blank stares about
for what designers should actually do next. And when you deliver a 100 page ppt,
the implicit response is – well, all of this!
Because we fall so in love with our findings, we lose perspective about it. We know
all this stuff and incomprehensible that anyone could change a single feature or
flow without understanding every last detail we found
And the problem here is that if you try to say everything, you end up having no
impact whatsoever – especially in an evaluation context – presenting every usability
bug = nagging.
By telling them everything you’re telling them nothing.
It’s never the case that designers need to know everything. Never. We have to
simplify and fillet our findings to deliver, meaningful, critical stuff that will engage
people.
24

25. The interesting thing about Vonnegut’s take on the story is that he told Hemingway’s story
to a fisherman, who basically said – the guy’s an idiot. Of course sharks are going to eat
your fish if you strap it to your boat. What you have to do in that situation is choose the
very best bit of marlin, cut that out and throw the best overboard – leave the sharks to it.
That way, at least you get a fish soup when you get back rather than people scratching their
heads at you thinking, what was all that effort for?
We have to fillet our research. We have to! But how?
Still remember trying to deliver 100 page powerpoint, met with blank stares about for what
designers should actually do next. And when you deliver a 100 page ppt, the implicit
response is – well, all of this!
Because we fall so in love with our findings, we lose perspective about it. We know all this
stuff and incomprehensible that anyone could change a single feature or flow without
understanding every last detail we found
And the problem here is that if you try to say everything, you end up having no impact
whatsoever – especially in an evaluation context – presenting every usability bug = nagging.
By telling them everything you’re telling them nothing.
It’s never the case that designers need to know everything. Never. We have to simplify and
fillet our findings to deliver, meaningful, critical stuff that will engage people.
The interesting thing about Vonnegut’s take on the story is that he told Hemingway’s story
to a fisherman, who basically said – the guy’s an idiot. Of course sharks are going to eat
your fish if you strap it to your boat. What you have to do in that situation is choose the
very best bit of marlin, cut that out and throw the best overboard – leave the sharks to it.
That way, at least you get a fish soup when you get back rather than people scratching their
heads at you thinking, what was all that effort for?
We have to fillet our research. We have to! But how?
25

26. Empathy is a critical skill for all UX professionals
But after a big research project, it can make us feel a monkey to its organ grinder.
26

27. We can get so passionately attached to our users and in love with our findings, that
we can let our empathy with them rule our reactions to them. And we want our
clients or partners to feel the same way that we do with them
•Forgetting that a lot of the time, we need to treat them as customers rather than
people
•And they say a lot of things that just aren’t relevant to what we are looking for
•Simply, that after meeting a lot of people we sometimes can’t see the wood for the
trees
•We celebrate the complexity of their users at the expense of providing actionable
insight
What to do:
Draft, space, review
Executive summaries
Remember that the most important thing about personas is their segmentation of
the user market, not the 10th bullet point that we put on them.
There are also some ways that we can think about what our user say that helps us
identify what is likely to unlock the most value
27

28. Japanese academic Noriaki Kano developed different categories to explain what
kind of product qualities satisfy people.
Threshold – Things people expect as a credibility requirement, also called hygiene
factors. Take away satisfaction if they’re not present, but no matter how much of
them you add or improve, satisfaction doesn’t improve.
•Ryanair defines threshold qualities. Made a lot of money out of figuring out exactly
what these are and delivering precisely no more than that. Airline: seat, you go in
the air, arrival at a new airport – this is undeniably a flight that you’ve been on
•Conference: airconditioning, basic time keeping
•Amazon: payment processing, ability to find what you’re looking for
28

29. 29

30. 30

31. Performance: classic top of mind things about what satisfies people about a
product or service. The more this need is met, the more satisfied you are.
•Ryanair: Price; BA perform on service too.
•Conference: Quality of speakers & workshops, networking opportunities, etc.
•Amazon: price, width of offering, delivery time, customer service
Excitement: the hardest to grasp in the abstract, excitement qualities make you
really satisfied if they’re present, but they don’t make you dissatisfied if they’re not
around.
•Ryanair: Well the only thing I’m excited about on a Ryanair flight is leaving it. BA,
an upgrade can be an excitement factor… hard to think of an ordinary situation
where you’d be disappointed to not get one
•Amazon: Black Friday sales
31

32. 32

33. 33

34. So what?
Clients understand their product already
•Clients have a really good understanding of what their product qualities are and
where they fit in this, even if they don’t know the Kano model per se
•Agencies often “discover” threshold factors and think that it’s new and important
•They also may not realise that they are really threshold factors.
•Example from phones – in my first user research for Nokia, it was in the states
where durability and battery life are king. Threshold factors… if you add more
durability then it won’t make people more satisfied.
HCI methods lean us towards discovering dissatisfiers rather than satisfiers.
• We identify and gravitate towards tangible problems and blocks – language like
‘pain points’ ‘severity’
There is a difference between experience and design
•We also conflate delivery problems with design problems and pretend that they’re
threshold factors. Like this
34

35. So this is a threshold product quality – accurate and reliable spell check.
This is clearly a bad experience, but my point is that it WAS NOT A DESIGNED
EXPERIENCE.
If you find the guy who designed the messaging view in the iPhone, or the keypad –
that guy did not design the autocorrection. Engineers did.
We need to prioritise the design in our evaluations rather than talking about
‘experiences’.
The implication is that spell check needs to go from being 90% accurate to 100%
accurate – well guess what, they knew that already.
And what would the impact on satisfaction be? Minimal. We can invest a lot of
money and effort without gaining significantly more satisfaction.
While big companies have a lot of resources, they have very scarce attention and
focus – this is very precious in big companies, so rather than thinking ‘oh my client
can just pass on my findings about spell check, offline mode and the manual to the
right people’ we have to help them focus on the single most important things!
35

36. So what? 2
FOCUS ON THE RIGHT KIND OF FACTORS FOR YOUR DESIGN OBJECTIVES
•Kano helps us target our research findings against the stage of design ideation we’re at
•Left diamond
•Typically looking for excitement qualities
•Performance qualities can be used to define the project
•Threshold qualities should just be ignored
•Right diamond
•Manage performance and threshold qualities
•This explains how in innovative designs, threshold qualities may not be perfect when they launch
•Windows 7 phone – no way to group your applications
•Palm Pre performance sucks – it is a slow phone
•Should threshold qualities be perfect, of course they SHOULD
•But the managed underdelivery of threshold qualities can allow teams to deliver products that
excite people
•Empathic, idealistic UX people would rather run a mile than suggest cutting back on threshold
qualities, yet Ryanair proves you can make millions by doing this.
•It feels heretical to be suggesting we use our user research to do this… but isn’t it true?
In terms of researching them, this is a very useful filter to use to understand what your consumers
have said and talked about. If we were researching Ryanair, then every single person would talk
about how awful the service is, and can you believe the charges that they land you with, and did you
hear that the staff are on commission for all those charges (which is true), etc etc. If we were doing
this research for an airline that wanted to explore low-cost alternatives, we could NEVER
recommend doing this stuff, cutting back services, adding fees, etc etc. And yet, there is an
enormous business built out of giving users as little as possible. Michael O’Leary even publicises the
fact that they’re always looking for other things to cut back on. Is their idea of charging for toilets,
not giving people a kind of sling rather than seat – are these real threshold qualities or aren’t they?
UX as a field would rather run a million miles than help businesses cut back on threshold qualities,
and yet there is millions and millions to be made in this.
36

37. The other big problem I see is when researchers making deliver their work to
designers with a list of recommendations or even worse, design concepts.
At Nokia specifically, rarely does a project go by without agencies wanting to ‘add
value’ as they see it, by giving us some concepts. You know, what if there was a
phone that did this, you could detach this part and it would do this, it could be a
customized phone for the elderly, or trend-following teenagers, or business people
in a new city for the first time, whatever. We’ve learnt all this, and this is what you
could do with it as a result. Sometimes this comes off as good-intentioned but
toothless, but to be honest most of the time it’s seen as patronizing or naïve.
37

38. It’s EASY to make recommendations to fix these problems that you’ve found, but
unless it’s explicitly part of your scope. It’s not our job as researchers to define the
solution; we have to explore and define the PROBLEM, not the solution.
 This happens again and again when I engage agencies – even if I explicitly
instruct them not to do it, they just can’t help it
Deliberately provocative with the headline
38

39. Or a slightly more scientific look at it
39

40. You can see why I want researchers to avoid making recommendations… because
when you do you’ve just shortcut this entire process.
What about at the diverging phases? Can’t we make recommendations even then?
Don’t understand the product, intent, business, industry, competition, suppliers,
etc. When you make recommendations in isolation, you’re bound to fall foul of
these gaps.
A more useful model for what researchers can do is here:
40

41. Useful for evaluation work
Problem is what was reported – “Users don’t see that there are multiple
homescreens on the Nexus One”
Problem Space is an explanation of the nature and structure of the problem, so we
go up a level in the hierarchy of the problem.
41

42. Problem: “Users don’t see that there are multiple homescreens on the Nexus One”
42

43. Problem Space is an explanation of the nature and structure of the problem, so we go up a
level in the hierarchy of the problem.
So one definition of the problem space might be:
No affordance for moving homescreen.
More precisely: Lack of affordance creates a discoverability problem for moving
homescreens.
And again: Users who have no experience with smartphones may have a discoverability
problem with recognising the system offers multiple homescreens and controlling them.
And again: Feedback that the phone gives isn’t sufficient
Components of a problem space:
• User experience (emotion, judgement, behaviour etc)
• Design element
• [Contextual factor]
• Linked together in a causal chain
• In a general state that explains general events not individual comments.
Solution Space is the area of possible responses to the problem. This is an active area in
which we ideate, it’s a brainstorming activity.
And then of course, the solution is what emerges from the solution space as the best
option for whatever reason, even time.
If you make recommendations without exploring the problem space or the solution space,
you go straight from problem to solution
•Miss out better solutions than the one that first occurred to you
•Reflect your imperfect knowledge of the design intent
•Disempower the creative authority of designers
So a recommendation for this could be to have page indicators on the screen with
numbers.
43

44. Which is what Samsung does
This might test well, but without further input, how do we know that this is the best
solution?
If we explored the solution space more, we’d find solutions like this:
44

45. 45

46. Exploring potential solutions gives us the perspective that real users don’t really
care what the navigation method they use for their screens are, they just want to be
able to get around their damn phone!
46

47. 47

48. 48

49. 49