1. Usability Evaluation Plan
For Kijiji Taiwan
Prepared for: Sydney Chen / Head of Kijiji TW
Tang Hsun / Director of User Experience Design
Prepared by: wwwins consulting
Author: Ruby Kuo & Alex Chiu
Date: Sep. 20, 2006
Revision 2.0
Approvals: ______________________ _______________________
Name Name

2. Usability Evaluation Plan for Kijiji TW 2006/9/20
0.Introduction
This document describes the usability evaluation plan for Kijiji TW. The purpose of this study is
to discover significant usability issues by testing real user with current live web site. This plan
includes the following sections:
1. Purpose of the usability evaluation
2. Participants
3. Design of the usability evaluation
4. Data collection methodology
5. Deliverables
6. Schedule
7. Appendix - Script of test scenario (English version)
Background information of Kijiji TW
Kijiji Taiwan was launched in May, 2005. The site is a group of free, local, community
classifieds websites. It offers a convenient, fun, and easy way (no need to register as a member to
post or respond an ad) for people in the same city to meet, trade, share ideas, and help each other
out in areas such as goods, cars, services, housing, jobs, and personals. There are over a hundred
thousand ads now, and the average increase rate is about 30%. Some most popular categories
(count by the number of ad post) are like: [1]housing (租屋售屋), [2]service(個人服務/商家),
[3]flea market (跳蚤市場), and [4]jobs(當地求職求才). Moreover, Kijiji cooperates with urMap
to supply the map exploration and quest function to assist user navigate efficiently. As realized
the Kijiji TW target user profile can be described as below:
busy working adults age 20-40
Internet daily usage 2-8 hrs per day
Trust individuals & turn to internet community for help
Use internet to connect to their lives and friends
Proactive, independent SOHOs or small shops owners
Page 1

3. Usability Evaluation Plan for Kijiji TW 2006/9/20
1. Purpose of the Usability Evaluation
1.1. Concerns
This section outlines the specific concerns that refine from the objectives in RFP document from
Kijiji TW.
The particular questions to be addressed in this study include:
Overall
O1). Will users be able to use this website easily, efficiently and satisfyingly?
O2). Is the navigation concept clear? Will users always know where they are and easily find their
way around the website?
O3). Is the information architecture about classified ad appropriate? Will users be able to find the
information or function they need very easy and quickly?
O4). Will users be able to understand the category naming or labeling meaning? Are they
meeting users’ expectation?
O5). Are the website’s color, graphics and design pleasing and attractive?
O6). What’s the user’s preference of this website? What is the overall impression of it, and what
emotional responses towards it?
O7). Which contents (information, functions, and services) meet users’ needs? And what are their
wishes in the future?
O8). Will users regard this website useful for them? Will they return in the future on their own?
Or recommend it to other friends? If not, what are the reasons? Figure it out.
Specific task and function
S1). Do users understand and use the "Location" options accurately during this study?
S2). Are users able to leave “Comment” on an Ad, and do they regard it useful or confusing
(compared with replying via email to the Poster of the Ad)?
S3). How do users search for “Housing (租屋售屋)” on the site? Do they need additional tools
or information?
S4). How do users search for “Services(個人服務/商家)” on the site? Do they need additional
tools or information?
S5). Do users understand the concept of Wanted/Offer(尋找、徵求/出售、提供) on the site? Is
this clear? Is it useful?
S6). Do users use the Maps available on the site? If so, when will they use it, for what purpose
and how?
S7). Do users have any difficulties in selecting a category to post (or browse) an Ad? If so, what
are they?
Additional
A1). Will the concerns listed above get any different opinions (or findings) between existing and
new kijiji users? If so, what’s the difference?
A2). To understand how people commuting between two cities feel when they have to select
different subsites for viewing different regions’ ads.
A3). How do users feel about my Kijiji?
Page 2

4. Usability Evaluation Plan for Kijiji TW 2006/9/20
3.1.3. Performance and subjective preference evaluation
The performance evaluation consists of a series of tasks that are evaluated separately and
sequentially. The typically procedure is as follows:
• After the orientation, the participants will be asked to sit down at the computer. The
facilitator will give the participants the task scenario script.
• Every participant will be encouraged to work without guidance, and be asked “thinking
aloud” while performing the tasks of the test.
• The facilitator may ask the participant to verbalize his/her thoughts if the participant becomes
stuck, confused or in silence.
• The after-scenario questionnaires for collecting subjective preference data (when available).
3.1.4. Post-test stage
At this stage, each participant will be interviewed by the facilitator, and assignment will include
the following:
• Completion of a post-questionnaire in which the participants share their opinions on the
website’s usability, appearance and general impressions of it (such as, look and feel and
navigation design).
• Participant’s overall comments about his or her experience.
• Clarify and collect more insight or internal ideas of his/her manipulate behavior.
4. Data Collection Methodology
Data will be collected through the use of a “thinking aloud” protocol and videotaping. Observer’s
notes will also assist for data analysis (include oral data, facial expression, emotional expression,
manipulate behavior and body language).
Measures to be collected include the following:
Performance measure:
1. Time/percentage to complete each scenario session & task.
2. The number of non-complete scenario sessions & tasks.
3. The number of all correct/incorrect selections (in each scenario & task).
i、 The number of incorrect menu or icons choices (in each scenario & task).
ii、 The number of total clicks (in each scenario & task).
4. The number of times to use “search” function (and first/each use in which scenario).
5. The number of times to use “map” function (and first/each use in which scenario).
6. The number of positive/critical statements (in each scenario).
7. The number of indications of frustration/joy from the participant (in each scenario).
Subjective preference measure:
Page 7

5. Usability Evaluation Plan for Kijiji TW 2006/9/20
1. Degree of “ease of use” overall (and/or in each scenario).
2. Degree of “ease of learning” overall (and/or in each scenario).
3. Degree of “satisfaction” overall (and/or in each scenario).
4. Degree of “usefulness” overall (and/or in each scenario).
5. Confidence with their performance overall (and/or in each scenario).
6. Degree of awareness where they are and ability of way finding.
7. Preference of visual design overall (include color, graphics, look and feel)
8. The most attractive function, service or content.
9. The number of subjective opinions of the usability and aesthetics of the website expressed by
the participants.
10. Degree of “intention” that returns and recommend to friends.
5. Deliverables
1. PowerPoint presentation with summary, interpretations, recommendations, and
embedded video clips to illustrate key findings.
2. Video/audio recordings of each test sessions.
3. Final presentation via telephone conference.
4. Digital audio files of simultaneous translation of each test sessions.
6. Schedule
The usability evaluation will be conducted between 25 Sep. and 1 Oct. 2006. The final detail
schedule is dependent on approval of this evaluation plan by Kijiji TW and the selection of
qualified participants.
Page 8

6. Usability Evaluation Plan for Kijiji TW 2006/9/20
7. Appendix
A. Script of test scenario (English version)
Participant greeting and background questionnaire (5mins)
Hi, welcome, thanks for coming….. I am ____ (facilitator introduce itself )
To break the ice - based on the basic information we have to chat with participant, and to make them
comfortable…
Pre test-questionnaire – Give them the agreement and background questionnaire to fill out…
Orientation (5mins)
Give an introduction of this study – the purpose, process and evaluation time for this test.
Today we are asking you to serve as an evaluator of this Web site and to complete a set of
scenarios. Our goal is to see how easy or difficult you find the site to use. We will record your
reactions and opinions; so, we may ask you to clarify statements that you make from time to
time.
There are totally 8 scenarios that we are going to take later, we calculate that will need around
an hour to finish. And I’m here to assist and give you orientation in each task scenario. For
example, I will ask you perform a specific task in a website, and would like you to think a loud
while you look for the answer. Although you may ask question anytime, but during the task
session I will not be able to offer any suggestions or hints. Don’t worry if you can’t find the
answer every time. Remember, we are testing the effectiveness of website--this is not a test of
you!
When you are ready, than we can begin. And when you feel that you need to take a break
during this study, please say so.
Here are couples thing that I like you keep in mind:
This is not a test of you; you’re testing the site. So don’t worry about making mistakes.
There is no right or wrong answer. We really just want to know if we designed the site well for you.
If you ever feel that you are lost or cannot complete a scenario with the information that you have been
given, please let me know. I’ll ask you what you might do in a real-world setting and then either put you
on the right track or move you on to the next scenario.
We will be video recording this session for further study if needed. Your name will not be associated or
reported with data or findings from this evaluation. Please fill out the video release form.
Finally, as you use the site, please do so as you would at home or your office. I do ask that when looking
for information, you do so as quickly and as accurately as you can.
Performance and subjective preference evaluation (60~80 mins)
Scenario 1.
Warm-up session. To learn the prior user experience of group A and the first
impression of Kijiji Taiwan homepage from group B.
(to group A) (to group B)
1.1 Please show me how you use kijiji 1.1 I will show you a homepage of a website
usually. (Open kijiji Taiwan hompage) we are testing today. Please look at it for
5-10 seconds as you might did when
Items to note in task 1.1: visiting a website for the first time. Then
a) What tasks they perform most often? And how? describe what you have seen.
Items to note in task 1.1:
Page 9