1. SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
José
Luis
González,
Roberto
García,
Josep
Maria
BruneI,
Rosa
Gil
and
Juan
Manuel
Gimeno
Universitat
de
Lleida

2. Introduc;on
• Objec;ve:
• Define
a
model
to
evaluate
the
quality
of
applica;ons
based
on
Seman;c
Web
Technologies
• Focus
on:
• Quality
in
Use
• Seman3c
Web
Explora3on
Tools
(SWETs)
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
2

3. Quality
of
a
SoVware
Product
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
3

4. Semantic Datasets
295 datasets
http://linkeddata.orgUse
Extension
Model
for
Seman;c
Web
Explora;on
Tools
SWET-­‐QUM:
A
Quality
in
4

5. Fine for computers… but people?
C. Warren
(blogger)
I’m writing about “Films I Like”.
Can I reuse LinkedMDB?
M. Harper
(developer)
I’m developing a bird
watching application.
Can I reuse
DBPedia?
http://linkeddata.orgUse
Extension
Model
for
Seman;c
Web
Explora;on
Tools
SWET-­‐QUM:
A
Quality
in
5

6. Mo;va;on
• Seman3c
Web
HCI
Challenges:
• Lot
of
data
but...
available
as
RDF
data
dumps,
seman;c
query
services,
etc
• It
requires
Seman3c
Web
exper3se
• Exis;ng
tools
hardly
usable
for
lay
users
User
Interface
needed
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
6

7. SWET-­‐QUM
• Extension
of
ISO/IEC
25010:2011:
• Metrics
based
on
standards
+
• Metrics
specially
conceived
for
SWETs
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
7

8. Quality
in
Use
Model
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
8

9. Effec;veness
• Defini3on:
the
degree
to
which
users
can
achieve
the
seman2c
data
explora2on
tasks
with
precision
and
completeness
• Metrics:
• Task
success:
what
%
of
one
task
is
completed?
• Task
comple3on:
what
%
of
the
tasks
is
completed?
Extension
• UI
Component
Effec3veness:
what
%
of
the
UI
components,
relevant
for
the
task,
do
the
users
view?
• Input:
eye
tracking
record
of
screen
areas
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
9

10. Efficiency
• Defini3on:
the
degree
to
which
users
can
achieve
tasks
by
inves2ng
an
appropriate
amount
of
resources
in
rela2on
to
the
effec2veness
• Metrics:
• Task
3me:
how
long
does
it
take
to
complete
a
task?
• Total
3me:
how
long
does
it
take
to
complete
all
the
tasks?
• Task
efficiency:
task
success
/
task
;me
• Help
requests:
how
many
help
requests
has
the
user
asked?
Extension
• UI
Component
Efficiency:
what
%
of
the
ahen;on
of
the
user
is
captured
by
the
UI
components
relevant
for
the
task?
• Input:
eye
tracking
record
of
screen
areas
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
10

11. Sa;sfac;on
• Defini3on:
the
degree
to
which
users
are
sa2sfied
by
the
seman2c
data
explora2on
tool
• Metrics:
• Sa3sfac3on
ques3onnaire:
how
sa;sfied
is
the
user
with
specific
soVware
features?
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
11

12. Context
Coverage
• Defini3on:
the
degree
to
which
Seman2c
Web
tools
can
be
used
in
different
contexts
and
adapt
to
different
user
mental
models
• Metrics:
Extension
• Task
Flexibility:
what
%
of
the
task
can
be
achieved
by
alterna;ve
methods
of
interac;on?
• Layout
Flexibility:
what
is
the
average
number
of
interac;on
steps
required
to
reach
the
UI
Components?
• Input:
expert
analysis
of
the
tasks,
applica;on
domain
and
user
interface.
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
12

13. Case
Study
• Rhizomer:
tool
for
Seman;c
Data
explora;on
• Automa3c
and
generic
UI:
menus,
facets,
breadcrumbs…
• Methodology:
• User-­‐centered
design
development
process
• RITE
Method:
short
itera;ons,
more
evalua;ons,
small
groups
• SWET-­‐QUM
model
• Evalua3on
proces:
• Pre-­‐test
• Test
• Post-­‐test
• Reports
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
13

14. Case
Study
• Dataset:
LinkedMDB
(seman;c
version
of
IMDB.com)
• Tasks:
• Task
A:
“Find
three
films
where
Clint
Eastwood
is
director
and
also
actor”
• Task
B:
“Find
who
has
directed
more
films
in
countries
located
in
Oceania”
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
14

15. Results
• Effec3veness
and
efficiency:
Task
Task
Success
Task
Time
Help
Requests
Task
(min)
Efficiency
Task
A
100
%
1.69
0
%
68
%
Task
B
100
%
3.43
100
%
32
%
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
15

16. Results
• UI
Components
Effec3veness
and
Efficiency:
UI
Components
Fixa3on
Fixa3on
UI
Component
UI
Component
count
Dura3on
(s)
Effec3veness
Efficiency
Naviga3on
bar
14
4.1
100
%
4
%
Facets
161
91.92
100
%
90
%
Facets
“pivot
bu[on”
5
3.75
100
%
4
%
“Navigate
to”
Box
3
1.15
100
%
1
%
Breadcrumbs
7
1.78
100
%
2
%
Total
190
102.7
-­‐
100
%
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
16

17. Heat
Map
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
17

18. Results
• Context
coverage:
Task
A
Task
B
Task
Flexibility
100
%
It
is
possible
to
go
through
50
%
It
is
possible
to
go
throuh
actor,
director
or
film
director
or
film,
but
not
through
country
or
con;nent
Layout
Flexibility
1.5
Zoom
film
(0),
expand
1.83
Zoom
film
(0),
pivot
country
facet
(1),
search
box
(2),
(1),
expand
con;nent
(2),
select
person
(3),
expand
select
Oceania
(3),
pivot
film
facet
(1),
select
person
(2)
(2),
expand
director
(3)
Interac;on
Steps
6
6
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
18

19. Results
• Sa3sfac3on:
(from
1
to
5)
• S-­‐G1
The
task
was...
very
hard
-­‐
very
easy
• S-­‐G2
I
think
that
I
have
done
the
task...
not
correctly
at
all
-­‐
absolutely
correct
• S-­‐G3
The
interface
structure...
did
not
help
me
at
all
-­‐
did
help
me
very
much
• S-­‐G4
The
;me
to
complete
task
has
been...
long
-­‐
short
• S-­‐G5
To
achieve
the
task
I
have
had
to
be...
very
focused
–
not
focused
at
all
• S-­‐G6
The
task
was...
badly
defined,
I
did
not
understand
the
objec;ve
-­‐
Well
defined,
I
understood
the
objec;ve
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
19

20. Results
• Sa3sfac3on:
Task
A
Task
B
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
20

21. Conclusions
and
Future
Work
• Conclusions:
• Model
to
evaluate
the
Quality
in
Use
of
Seman;c
Web
Explora;on
Tools
• Standard
metrics
+
SW
metrics
• Applied
in
Rhizomer
à
maximise
the
quality
of
user
experience
• Future
Work:
• Explore
addi3onal
metrics
• Leverage
eye
tracking
• Apply
to
other
tools
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
21

22. Thanks for your attention
Josep Maria Brunetti
josepmbrunetti@diei.udl.cat
http://rhizomik.net
SWET-­‐QUM:
A
Quality
in
Use
Extension
Model
for
Seman;c
Web
Explora;on
Tools
22