Presentation given at Columbia University on guerrilla usability by David Walczyk.

1. Usability Testing in the Wild :
Mindset and Framework
David Walczyk
Client: Columbia University, NYC

2. situation & myths
/
precepts & methods

3. situation & myths
/
precepts & methods

9. situation & myths
/
precepts & methods

10. OLD MYTHS
information is...
scarce, linear, concrete, analog, surface,
closed, non-visual, non-spatial, discrete,
hierarchical, mutually exclusive, phonetic,
etc...

12. NEW MYTHS
information is...
abundant, holistic, non-linear, deep,
abstract, computational, executed,
exploratory, behavioral, symbolic,
recursive, networked, emergent, etc...

14. situation & myths
/
precepts & methods

15. when media change people change.

16. new media are extensions & amputations.

17. new media create new environments.

18. preconceived figures lead to
unperceived grounds.

19. old methods confirm old ways and conceal
new ones.

20. insight and understanding
trumps a point of view.

21. situation & myths
/
precepts & methods

22. Methods
First Thinking and Then Doing

23. Steve the Student Heather the Hipster Tom the Teacher Adam the Architect Helen the Hippie
Steve isn't looking for Heather is going to Tom's primary focus is Adam will be Helen will be on BFI to
much exploration want to explore the going to be collecting interested in read Design Science
here. He wants a clear site. Her major ideas for his class – networking/exchanges News and to listen and
path to the concerns are going to activities, lesson plans, with like minded watch lectures.
information sought so be finding biographical audio/video materials. designers and their
he can complete his information and how A secondary concern is projects.
homework. to get involved. going to be
networking with other
professionals to learn
Searching for Heather is visiting the about what worked for
Tom is looking for Adam is looking for Helen is looking for
biographical site to learn more them.
curriculum ideas and networking the new issue of
information about about Buckminster maybe some insight/ opportunities on the Design Science News.
Buckminster Fuller Fuller. success stories from site.
from an authoritative others in his field.
Grew up
source. with WWW. Very tech savvy . Very comfortable. Very comfortable. Can get by.
Steve is coming to the Heather is coming to Tom is coming to the Adam will come to the Helen is looking for
site for information the site for information site from a site to brush up on current news and
about Bucky the man. about Bucky the man, sustainability Buckminster Fuller the updates to the archive.
but is going to perspective. He's going man, network with like
explore: community to need education- minded professionals,
content, current specific resources and and keep abreast of
events, ways to get community content. sustainability news and
Steve is going to Heather is going to
involved, etc. Tom is going to expect Adam is
events. going to Helen is going to
expect to find familiar expect clear navigation logical navigation, and expect logical expect things to be
navigation and web 2.0 features. grade-specific navigation, where she’s used to
mechanisms, like educational resources. biographical and finding them. The
dynamic menus, that historical resources, an structure of the site
lead him to the easy registration may not make sense,
information sought. process, and but she has learned
community content. how to navigate it.
BFI Personas and Scenarios 1

24. BFI Personas and Scenarios 4
Demographics Comfort with Technology Needs
Age: 24 Computers: Native. • Biographical/
Education: liberal arts BA, thinking Web: Native. historical
about grad school Social web: Belongs to all and a few you haven't head information
income: <$30,000 of. • Current events
Bucky familiarity: Vague • Community content
• Information on
getting involved
• Clear and logical
Personal Background navigation
Heather is tech savvy and hyper-connected like most of her lot. She's into sustainability & is
coming to the site to learn more about the man and his connection to the broader
HEATHER sustainability movement, though she hasn't really heard of him beyond being "the dome guy".
The Hipster After exploring the site, Heather is going to want to organize a community chapter. She
Googled "Buckminster Fuller Dome" to get to the site.
"Isn't he the dome guy?"
Motivations Scenarios Site Features Behaviors on Current BFI Website Behaviors on Redesigned BFI Website
To learn more Read about geodesic • Clear and logical Googles domes and finds what appears to be a Lands on content page from Google. Skims it.
about Bucky domes and thought navigation. landing page for articles about domes at BFI. Realizes she wants to check out more. Using
and his Bucky sounded familiar, However navigation out of that page to other bread crumbs she goes up a level to see what
connection to so she Googled parts of the site is unclear. other content may be of interest to her.
the "Buckminster Fuller
sustainability Dome" and landed on a
movement bfi.org content page
Wants to get Inspired by her
about domes. reading • Clear and logical Goes to BFI homepage via her delicious.com Goes to BFI homepage via her delicious.com
involved the other day, she navigation. bookmarks. Clicks on the Community Content bookmarks. Selects the "Local Chapters" button.
locally decides she wants to button, but is unsure as it also reads "Connect Taken to local chapter landing page where she
contribute locally. She • Easy and secure with our network of designers." Clicks on the picks a chapter in her local region. Is excited by
goes back to the BFI registration "Use our forums" page and is taken to a How to all the evens listed on the homepage, and wants
homepage to find out process (doesn't use forums page. Clicks the back button. Sees a to participate. Clicks the "register" link at the
about local chapters. ask for too much link in the red box in the upper right corner top right corner (where it's usually found) to
or too little info). labelled "BFI Forums". Clicks that and is taken to sign up for the email newsletter and to join the
• Clear opt-out the forums. forum.
options.
Wants to keep Gets too much email • RSS/Atom Goes to the BFI website and clicks on the RSS Heather realizes she doesn't want to fill up her
abreast of any and newsletters end up (customizable link. She's unsure exactly what content she is inbox with emails, so she goes to the BFI
events and spam. Worried about feeds). subscribing to. She clicks "More options homepage and clicks the RSS link in the upper
news from missing an event at her (experts only)" and finds an overwhelming list right corner. She is taken to the RSS page where
BFI. local BFI chapter. of potential RSS subscription options. She goes she chooses what content she wishes to
back to the RSS page and subscribes to the subscribe to in her feed reader.
basic feed.

25. Demographics Comfort with Technology Needs
Age: 16 Computers: Grew up with them. • Access historical
Education: High school sophomore Web: Doesn’t know a world without it. information about
Income: None (parents >$90,000) Social web: Facebook is lame now that Mom’s joined. Bucky in order to
Bucky familiarity: None complete an
assignment
Personal Background
Steve is a 16-year-old high school student finishing his sophomore year. He is very comfortable
with online environments and has the latest video game consoles. He has no familiarity with
STEVE Bucky, and was assigned to write a paper about him for a class. He doesn't read web pages
The Student thoroughly – he's more of a page scanner and usually clicks the first link. If he doesn't see
something quickly he gives up. He is frustrated that he can't just use Wikipedia for his
"I don't know, my teacher assignment.
just
assigned it."
Motivations Scenarios Site Features Behaviors on Current BFI Website Behaviors on Redesigned BFI Website
Learn He couldn't use • Clear path to Clicks the "Who is Buckminster Fuller?" button Steve clicks on the About Buckminster tab and
historical Wikipedia as a source. biographical on the BFI homepage. Scrolls down to "Basic is taken to a landing page that includes a Bucky
information Clicked on BFI.org as an information. Biography" clicks link -- finds a timeline. Clicks biography. Links to more in-depth resources
about external link from Bucky the back button. Clicks on the "Introduction to via the Education page are included.
Buckminster article. Buckminster Fuller" article. Is frustrated that the
Fuller article has some biographical information but it
is not complete.
BFI Personas and Scenarios 3

26. BFI Personas and Scenarios 2
complex
interactions
ct r
ite p ste
ch Hi
Ar t he
th
e er
a th
a m He
Ad
frequent
Steve the Student
interactions
er
infrequent a ch
e Te
interactions p pi e
Hi th
e m
th
n To
le
He
simple
interactions
Key
Usage of current BFI website Usage of redesigned BFI website
Change in website usage

27. Usability Professionals
Make Grist Space Collective Surfrider Foundation
Association
http://www.makezine.com/ http://www.grist.org/ http://spacecollective.org/ http://www.surfrider.org/ http://www.upassoc.org/
From the consistent browser Browser title, site ID, and An initial Flash animation is Evocative header images are As befits a website for a
title, site title, and subtitle, it subtitle make clear that the difficult to read and potentially background to the professional association of
is fairly clear that Make covers website provides annoying to users without the foundation's logo and usability specialists, no initial
do-it-yourself technology "environmental news and plug-in or who are just illustrate its motto, "Protecting ads detract from the clarity of
projects. Ad placement commentary." But a large impatient. A large text block Our Beaches For Over 20 a Web standard top left logo
unfortunately dilutes the banner ad is placed at top of about the Space Collective text Years." Throughout the page, and subsequent tabbed global
message somewhat – banner the page, where many users then loads first in the middle the use of blue and gray navigation. Navigation labels
and Google ads loom above expect to find the main of the page, signaling that this evokes the ocean and the use recognized syntax, such
the global navigation – and a navigation. Instead, main site is "different." The text foundation's focus. Global as "About UPA," so that no
large ad for McDonald's does navigation is located on the outlines the purpose, but not navigation in a Web-standard, mouseovers are required to
not instill initial trust in the left. Site hierarchy is displayed the nature, of the site, and top of the screen location uses understand them. Most other
site. But global navigation through the use of section moreover does so in an clear, action-oriented labels to links are either standard
follows Web standards, titles, but section titles manner that reinforces the orient the user and convey site underlined blue text or
counteracting the profusion of themselves are not clickable site's arty/science-fiction contents. The site's appeal for clickable graphics. Text about
ads: a logo home button is on and the list is so long that the identity: users are user action continues with the organization is
the upper left and utilities user cannot scan what "terrestrials." Navigation is other command labeling: "Stay prominently placed in the
such as search and RSS are on amounts to a summary of the difficult to pick out, as dark Informed," "Join Surfrider," center just below the global
the upper right, with a row of site without scrolling. gray elements do not contrast "Support Surfrider." The center navigation, and is followed by
multicolored labels across the Navigation labels such as with a black background of the page features a slide content that is current and
page presenting navigation "News/Analysis" communicate (though some elements pop show that outlines the useful to the association's
options to the six main a news-oriented site, but out in red on mouseovers). A foundation's strategic membership: conferences,
sections of the website other labels, such as "Victual login utility is in a Web initiatives and links to the events, and news. A short
beneath. Labels such as "Blog," Reality" impart a kind of standard upper right location, "What We Do" area of the site, section then follows which
"Videos/ Podcasts," and "hipness." Although the latter but a barely visible hover label and is followed by links to introduces usability concepts
"Forum/Community" signal type of labels are opaque to states that membership is by current campaigns and to potential clients. Site
that Make is a participatory the uninitiated, mouseovers invitation only. Navigation featured content. On the left utilities, including search,
site. The labels are highlighted yield some clarification. A labeling is also an issue, both are site utilities such as an most visited links, and
when moused over, and clearly labeled toolbox links to in design (small and low email newsletter signup box, a membership registration/log-
remain so when selected, useful site utilities, and is contrast) and vocabulary (what search box, and membership/ in are offered at the left and
acting as "breadcrumbs" to located where most users is a "timecapsule"?). However, chapter links. Membership right sides of the screen.
orient the user. Other links are would expect it, at top right. the blog/channel, multimedia links are also grouped in and Above UPA's contact
not visually uniform: some are The rest of the page follows a nature of the site is beneath a color-coded portion information and privacy policy
indicated by boxes and others layout fairly typical to news immediately implied by the of the top menu on the right. at page bottom are what
merely with Web standard organizations on the Web: label "recent posts" and the Beneath this second appear to be ads in the form
underlined text. An option to several prominently featured obvious image gallery layout. membership area are links to of logos/links to UPA's
"Suggest a Site!" does not images/videos plus other Also, higher contrast large the foundation's blogs, corporate supporter websites,
specify why, and simple footer areas corresponding to global labels effectively separate podcasts, and other but these logos establish
navigation is preceded by yet navigation categories page areas and lead the eye publications, coastal news UPA's credibility by
more Google ads. But the containing titles and links to down and across the page. items, and two vaguely labeled announcing its acceptance by
home page's large featured the latest stories. The right Space Collective's invitation- but visually appealing links to these organizations. Clear
content area reinforces Make's side of the screen contains only, members-only policy, "The Ring" and "SocialVibe." labeling, standard placement
message, flashing features more featured content, both and its use of a black Overall, the site's clear of page elements, and an
from the site and magazine, as internal and external to the background, slick layout, and organization and layout, unremarkable, business-
well as recent blog posts. The site, including a section for arcane vocabulary convey imperative labeling, and friendly color scheme convey a
design in general is not "this week's most popular" "coolness." The site is judicious use of images sober dedication to usability
particularly slick, which items, as is typical for news deliberately less than communicates the principles.
reinforces the you-can-do-it, sites. Footer navigation welcoming, drawing users in foundation's proactive nature,
craft-oriented nature of the continues the problematic by creating a desire to be "in" efficient operation, and
site. global navigation labels, the group and in the know. seriousness of purpose.
unfortunately.
BFI Comparative Analysis : Detailed Evaluations : Home Page 5

28. Usability
Make Grist Space Collective Surfrider Foundation
Professionals
Association
Home Page
Design 4.0 4.5 4.0
3.5 3.5
Interior
Page 5.0
Design 4.0 4.5 4.0
1.5
Search
Functionali
ty 4.5
3.5
2.5
1.0 1.0
Features
Offered 5.0 4.5 4.5
4.0
2.0
Navigation 5.0
4.0 4.0
3.0 3.0
Interaction
Design 5.0
4.0 4.0
1.5 1.0
BFI Comparative Analysis : Summary Evaluations (1 = poor, 5 = 11

29. BFI Comparative Analysis : SWOT Analysis for BFI.org 4
Strengths Weaknesses Opportunities Threats
• BFI has rich content • The BFI website does not • Sites such as Surfrider • Other sites' well engineered
addressing a wide range of fully promote R. Foundation have not yet page designs and
potential uses Buckminster Fuller's legacy fully developed interactive navigation mean that, in
by making his life, designs, functionality to encourage spite of the presence of
• Buckminster Fuller and his and ideas easily participation and a large advertising, users are more
ideas and design could be understandable by a wide potential user base may still likely to visit those sites and
enormously appealing for audience remain to be captivated/ thus spend less time at
educational uses captured BFI.org
• Information on the BFI
• BFI already has a dedicated website is not organized by • Sites such as Space • Several sites offer the site
community of fans of user-oriented topics Collective and Surfrider utilities users have come to
Buckminster Fuller the man Foundation provide poor expect and BFI risks losing
• Interactive functions are not search capabilities, and BFI potential users to sites
• The BFI Challenge has clearly described or could improve in these offering easier ways to get
created enormous goodwill distinguished from one areas to make its large pool involved or stay informed
and exposure for the another so not all users who of information more
organization might be interested in accessible and thus more • Other sites do a better job
participating are reached attractive of stating or making clear
• The Idea Index is unique their organization's and/or
and could be leveraged to • Separate registrations are • Sites such as Space website's purpose. Space
invite more discussion and required for BFI and the BFI Collective and Surfrider Collective's textual
interaction Challenge Foundation do not offer description may be
• Top level navigation is much in the way of site somewhat vague, but the
utilities to engage users, design of its home page
difficult to find, so users and BFI could work to clearly implies the nature of
may be confused about how improve in this area and the site's contents
to navigate the site increase user involvement
• Labels use specialized • BFI risks losing potential
• BFI hosts such a wealth of members to sites such as
vocabulary that new users historical and current Space Collective who have
may not understand information regarding already succeeded in
• Tagging functionality has Fuller's Ideas, Life, and generating active member
not been fully exploited to Designs that its redesign engagement
enable exploration and could establish BFI as the
discovery of site content authority on information
regarding the man
• The use of the blog format
results in posts and
comments that appear to be
orphaned and not actively
moderated: the site appears
to be a ghost town

30. Methods
First Thinking and Then Doing

31. GUERILLA USABILITY
effectiveness + efficiency + satisfaction
Guess(ability) vs. Learn(ability)

32. EFFECTIVENESS
Can a task be completed.

33. EFFECTIVENESS
measured in yes or no.

34. EFFICIENCY
how long to complete a task.

35. EFFICIENCY
measured in time.

36. SATISFACTION
physiological / social / psychological /
ideological

37. SATISFACTION
measured in affect.

38. Guessability vs. Learnability

39. Guerilla Testing
4 steps

40. 1. Decide what to test...
A whole site?
Part of a site?
A process?

41. 2. Decide how to test it...
What are the tasks to complete?
What are the probing questions?
What are the post-test questions?

42. 3. Test it...
Recruit users
Test Users

43. 4. Analyze results and make
recommendations
Quantitative
Qualitative
Severity

44. dare to be naive.

45. Thanks.
david@davidwalczyk.com