In September 2013, the Metropolitan Museum of Art launched a new audio guide. This presented an opportunity—and a need—to take a more strategic look at this long-standing service. Although the Museum has provided an audio-guide service for more than fifty years, thus far it has had a limited understanding of who uses the service and what kind of impact it might have. Who is the audio guide serving? What is and is not working? In an effort to answer these questions, the Museum conducted an evaluation of its new audio guide in collaboration with Frankly, Green + Webb. We took a comprehensive look at the service, seeking to understand the entire visitor journey from initial awareness to the effect of the guide on the overall museum experience. This service design approach—though well established in the commercial sector —is relatively new in the museum field. Our paper includes findings from the evaluation and insights we gained from the research process, as well as how these insights are applicable to the wider museum field. What can other museums learn from our experience? What is the value of evaluating an audio guide as a service rather than a product? How can museums move from evaluation data to insights and actionable recommendations? And perhaps most challenging: how can we design successful digital products and services in complex organizations with multiple stakeholders who have different levels of comfort with innovation and risk? These issues are not unique to the audio-guide service or to the Met; they are common to digital projects across all museums. Paper available here: http://mw2015.museumsandtheweb.com/proposal/a-new-look-at-an-old-friend-re-evaluating-the-mets-audio-guide-service/

1. T: @lhmann @franklygw @iamgracetung @metmuseum
A New Look at an Old Friend:
Re-evaluating the Met’s Audio
Guide Service
Laura Mann, Frankly, Green + Webb
@lhmann @franklygw
Grace Tung, Metropolitan Museum of Art
@iamgracetung

2. T: @lhmann @franklygw @iamgracetung @metmuseum

3. T: @lhmann @franklygw @iamgracetung @metmuseum

4. T: @lhmann @franklygw @iamgracetung @metmuseum
Audio guide = product

5. T: @lhmann @franklygw @iamgracetung @metmuseum
We need an
app
A more modern
device
New tours,
more options
Beacons!

6. T: @lhmann @franklygw @iamgracetung @metmuseum

7. T: @lhmann @franklygw @iamgracetung @metmuseum
Audio guide = product

8. T: @lhmann @franklygw @iamgracetung @metmuseum
Audio guide = product
service

9. T: @lhmann @franklygw @iamgracetung @metmuseum
Audio guide = product
A service helps someone do
something
service

10. T: @lhmann @franklygw @iamgracetung @metmuseum
Design Principle
Build digital services,
not websites
Our service doesn’t begin and
end at our website. It might
start with a search engine and
end at the post office.
UK Government Digital Service
https://www.gov.uk/design-principles

11. T: @lhmann @franklygw @iamgracetung @metmuseum
Design Principle
Build digital services,
not websites
Our service doesn’t begin and
end with the audio guide device.
It might start with the website
and end at the gift shop.

12. T: @lhmann @franklygw @iamgracetung @metmuseum
Attract
Mobile Service Design
User Journey
Decide Use Deliver
Value

13. T: @lhmann @franklygw @iamgracetung @metmuseum
Attract
Mobile Service Design
User Journey
Decide Use Deliver
Value
1. Attract: Who are the users and what are their
needs?
2. Decide: Do visitors understand the offer?
3. Use: Is it easy to use?
4. Deliver value: Does it add value?

14. T: @lhmann @franklygw @iamgracetung @metmuseum
New Audio Guide Launches
September 2013
• Long-standing service
• 250,000 users annually
• 3000+ audio messages

15. T: @lhmann @franklygw @iamgracetung @metmuseum
Goals
1. Actionable
recommendations
1. Evidence base for
future decision-
making
2. Tools for on-going
evaluation

16. T: @lhmann @franklygw @iamgracetung @metmuseum
• Quantitative Survey
• Audio guide users and
non-users (n=492)
• English, Mandarin and
Spanish
• Qualitative Interviews
• Usability Testing
• Think aloud method
• Structured & unstructured
How We Went About It

17. T: @lhmann @franklygw @iamgracetung @metmuseum

18. T: @lhmann @franklygw @iamgracetung @metmuseum
User Journey Part I:
Attract
Who is being attracted to
of the service and what
are their needs?

19. T: @lhmann @franklygw @iamgracetung @metmuseum
Let’s meet the users
cc images courtesy of Ed Yourdon on Flickr
International
and Domestic
Tourists
New York City
and Tri-State
(NY, NJ, CT)
19%
81%
Audio Guide Users

20. T: @lhmann @franklygw @iamgracetung @metmuseum
Tourists are the ‘natural’ market for the audio guide
cc image courtesy of Ed Yourdon on Flickr
The audio guide offer:
 Helps them navigate the
Museum, conceptually and
physically
 Helps them find the
‘Treasures’
 Provides content in their
own language
International
and Domestic
Tourists

21. T: @lhmann @franklygw @iamgracetung @metmuseum
Locals are a more diverse audience–one size will not fit all
cc image courtesy of Ed Yourdon on Flickr
New York City
and Tri-State
(NY, NJ, CT)
The audio guide offer:
 Special exhibitions
✖ Perceive the guide as a
service for tourists
✖ Different touch points

22. T: @lhmann @franklygw @iamgracetung @metmuseum
What does this
mean for you?
1. Service design is a
strategic cross-departmental
process
2. Services need to be designed
for specific audiences. Who
is your service targeting?
3. Understanding the visitor
journey will allow you to
target visitors at the right
moment

23. T: @lhmann @franklygw @iamgracetung @metmuseum
User Journey Part II:
Understanding
Is the service easy to
understand?

24. T: @lhmann @franklygw @iamgracetung @metmuseum
No. Why is that?
• Audio guide is large and
ambitious: 25+ tours for
highlights, collections
areas, themes and families
• Describing the audio guide
offer is challenging for
staff
• Complex offer means visitors
may not be able to make
informed choices
• I think there's a
lot more on [the
audio guide] than
I realized.
Emily, age 35-44

25. T: @lhmann @franklygw @iamgracetung @metmuseum
Can visitors make an informed choice?
Natalie looks for a highlights tour
…I don’t know what’s in
any of these so I don’t
know where to go first…

26. T: @lhmann @franklygw @iamgracetung @metmuseum
What does this
mean for you?
1. More content and features:
not necessarily better
2. Give visitors a basis on
which to make informed
choices – that reflect their
needs
3. Design for short transaction
times and no learning curve

27. T: @lhmann @franklygw @iamgracetung @metmuseum
User Journey Part III:
Use
Is the service easy use?

28. T: @lhmann @franklygw @iamgracetung @metmuseum
Visitors reported that the guide was very easy to use
76%Rated the guide “very
easy” to use
I just type in the
number
-Jason

29. T: @lhmann @franklygw @iamgracetung @metmuseum
Say ≠ Do

30. T: @lhmann @franklygw @iamgracetung @metmuseum
Digging a little deeper
I think that they’re just
dual numbers and I don’t
understand why they would
have that because I’m
sure 3806…is going to
give me the same audio
(enters 3806 and listens)
Yeah, it was the same
thing except a different
voice.
Lucy

31. T: @lhmann @franklygw @iamgracetung @metmuseum
I was looking at some of the
tours but I wasn't sure
where they started relative
to where I was so I just
kept going with the stop
labels
Sarah, age 35-44
Usability is not only about what’s on the device’s screen

32. T: @lhmann @franklygw @iamgracetung @metmuseum
What does this
mean for you?
1. Ensure that in-gallery, on-
screen and in-ear are
aligned
2. ‘Start mobile and stay
mobile’ - ideate and test
the experience and content
in the gallery space
3. Innovation should be driven
by user needs, making things
easier for users, not harder

33. T: @lhmann @franklygw @iamgracetung @metmuseum
User Journey Part IV:
Value
Does the service
deliver value?

34. T: @lhmann @franklygw @iamgracetung @metmuseum
Overall, visitors are satisfied with the audio guide
90%would recommend the
guide to a friend
81%rated their overall
audio guide experience
as good or very good
Yes!
Best $7 we’ve spent
Louise,
age 35-44
It was awesome. Totally
converted to the audio.
Anna, age 35-44

35. T: @lhmann @franklygw @iamgracetung @metmuseum
The guide makes for a more meaningful visit
Q: As a result of taking the audio guide today, have you…
Learned about art
Noticed details in the
art works
Enjoyed your visit
Explored the Museum more
Discussed what you saw
and learned
Felt inspired
It draws your
attention to things
you might not have
noticed yourself.
Jake, age 18-24
…It's like getting
to know someone
rather than just
finding them
pretty.
Caroline, 25-34
N=341

36. T: @lhmann @franklygw @iamgracetung @metmuseum
0%
10%
20%
30%
40%
50%
60%
Much worse than
I had expected
Worse than I
had expected
Same as I had
expected
Better than I
had expected
Much better
than I had
expected
Guide Users
Non-Users
It improves the overall experience of the Museum
N=341Q: Overall, how satisfied are you with your visit to the Museum today?

37. T: @lhmann @franklygw @iamgracetung @metmuseum
Foreign language speakers are having a less positive
experience
90%Rate their experience
good or very good
73%Rate their experience
good or very good
English Other Languages

38. T: @lhmann @franklygw @iamgracetung @metmuseum
English
Stops
2601
Spanish
Stops
141
Mandarin
Stops
44
Main Building
The offer looks different depending on what language you speak
Focusing on the needs of
foreign language
visitors will improve
visitor experience and
audio guide take up rate

39. T: @lhmann @franklygw @iamgracetung @metmuseum
What does this
mean for you?
1. Guide users have a better
overall museum experience
2. Raising take up rate =
improving the visitor
experience
3. Visitors value close looking
and meaning over information
But, dig deeper…
4. The service may not deliver
the same value for all users

40. T: @lhmann @franklygw @iamgracetung @metmuseum
The impact of the
process and the
path forward

41. T: @lhmann @franklygw @iamgracetung @metmuseum
THE KEY LESSON
Who are our users?
What do they need?

42. T: @lhmann @franklygw @iamgracetung @metmuseum
3 OBJECTIVES
• Improve User Journey Fluidity
• Attract the Local Repeat Audience
• Provide for First Time, Foreign-Language Speaking
Audience

43. T: @lhmann @franklygw @iamgracetung @metmuseum
CONSOLIDATION
The User Journey is
not smooth. The User
is overwhelmed.
• Simplify interface, limit
choices
• Condense labeling, clean
metadata
• Cohesive signage, logos,
supplementary materials
• Clear, multiple distribution
points
WHAT WE’LL DOTHE PROBLEM

44. T: @lhmann @franklygw @iamgracetung @metmuseum
PERCEPTION + ACCESS
Local, repeat visitors
are not attracted to the
Audio Guide offer.
THE PROBLEM
Try it yourself! Go here: www.metmuseum.org/audioguide
• Focus on special
exhibitions
• Audio Guide Web Feature
• Research and test
WHAT WE’LL DO

45. T: @lhmann @franklygw @iamgracetung @metmuseum
CONTENT DEVELOPMENT
First-time, foreign
language speaking
visitors need more
content.
• Produce 300 translated
stops in 9 languages per
fiscal year
• Highlight permanent
collection
• More thorough analytics
THE PROBLEM WHAT WE’LL DO

46. T: @lhmann @franklygw @iamgracetung @metmuseum
AUDIO GUIDE = SERVICE