The GReAT project aimed to develop an affordable computer-based gesture therapy called Gest to help people with severe aphasia practice gestures at home. Through participatory design sessions with people with aphasia, they created a prototype for testing in a pilot study. Preliminary results found Gest was successfully used independently in diverse home settings and enjoyed by users. However, the pilot study was still needed to determine if Gest improved gesture production or naming abilities and if effects were maintained after use.

1. Aphasia and Technology:
The GReAT Project
Abi Roper and Jane Marshall
On behalf of the GReAT Project Team
Division of Language and Communication Science
Department of Human Computer Interaction Design
City University London

2. Presentation Outline
•The Project
•Designing and Refining a Computer Gesture
Therapy - Gest
•Gest Demonstration
•Delivering a Computer Therapy
•Gest Pilot Study
•Preliminary Outcomes

3. Project Aims
• To develop an affordable, computer-based
technology that can be used in therapy at home to
help people with severe aphasia to gesture.
• To establish how to design effective/engaging
interactions for people with aphasia.
• To evaluate the efficacy of the technology within a
pilot therapy study

4. Project Structure
• Phase 1: Designing a prototype gesture
therapy using participatory design
methods.
• Phase 2: Testing and piloting the
prototype

5. Project Team
Human Computer Interaction Design & Language and Communication Science
Stephanie Wilson Sam Muscroft Julia Galliers Jane Marshall
Naomi Cocks Tim Pring Abi Roper

6. Phase 1
• Designing a prototype gesture
therapy using participatory
design methods.

7. Consultants
Justine Everson Gerald Hartup Carol Watson
Philip Pepper Emma Buswell

8. Consultants
•Role: to test and feedback about relevant technology.
•Person Specifications:
–Expressive aphasia language difficulties.
–Able to attend university once or twice a month for
participatory design sessions.
•Recruited through in house clinic and through links with
the Stroke Association Communication Support
Co-ordinators.
•Employed by City University London as Casual Staff
members.

9. Methods: Participatory Design
Sessions
•Participatory design – engaging end users in design
process
•Sessions explored offline gesture therapy, computer
gesture recognition, interaction within 3D worlds and
computer interfaces.
•Consultants took part in 9 sessions each
•Project team involved in each session
- 1 HCID Researcher
- 1 HCID Developer
- 1 Speech and Language Therapist Researcher
- 2 or 3 Consultants

10. Session Structure
1. Introduction to scheduled
activities
2. Round table gesture activity
3. Demonstration of Technology
4. Trial use of technology by one
consultant - followed by
interview at computer
5. Tea break
6. Trial use of technology by
remaining consultant(s)

11. Participatory Design
• Watch a video excerpt of the
design process in action on the
next slide

12. What did we learn from the
Sessions?
1. Consistency
2. Simplicity
3. Pace
4. Reliability
5. Rewards
6. Individual Differences
7. Potential of ‘gaming’.

13. The Prototype
OK
← →

14. Demonstration
•Watch a video Demo of the
Gest prototype in action on
the next slide

15. Using the Therapy at home
•How does this work at home?
Key differences between lab and
home –
User practising
independently, User intending to
practise daily. User practising in
non-lab conditions.

16. Things to consider when setting
up
•Lighting conditions
•Safety and permanence
(negotiate!)
•User comfort and access

17. Things to consider when training
•Develop the user’s confidence in the system.
(Be confident yourself)
Demonstrate:
1. Allow user to observe entirely
2. Allow user to observe and operate
interaction buttons
3. Allow user to operate alone but with
support as needed (confidence)

18. Things to consider when training
•Reinforce how to switch the computer
on and off several times.
•Make an appointment to come back in
one week to review.
•At review appointment, observe and re-
train difficult procedures.

19. Phase 2
• The Pilot Study

20. Questions
• Will practice with Gest improve participants’ production of
gestures &/or spoken words?
• Will improvements be specific to items that feature in the
programme?
• Will gains occur when Gest is used without ongoing
therapist support?
• Will gains be maintained after Gest is withdrawn?
• What are participants’ views about Gest?
• What are carers’ views about Gest?
(where relevant)
• Is Gest easy and enjoyable to use?

21. Participants
• 10 people with severe aphasia
– Consent to take part
– Fluent pre-stroke users of English
– Naming score <20%
– Able to recognise pictures
– No known dementia or other cognitive impairment

22. Consent
Screening
Phase 1 with weekly
Tests (1)
visits from therapist
3 Weeks
Practice
Phase 2 with no weekly
Tests (2)
visits from therapist
3 Weeks
Practice
Tests (3)
3 weeks
no tool
Total time commitment: about 14 weeks Tests (4)

23. Practice Phases
• Each last 3 weeks
• Each practise 15 gestures with the tool
• Phase 1: Weekly visits from therapist
• Phase 2: Initial but no weekly visits

24. Tests
• 60 items
– Gesture from picture
– Name from picture What is the
name of this?
How would
you gesture
Items: this?
30 practised with Gest
15 familiarised only
15 controls

25. Scoring Gestures
• Gesture tests are filmed
• 4 Scoring videos created
• Each video contains 60 gestures in random
order:
– 15 from test 1
– 15 from test 2
– 15 from test 3
– 15 from test 4

26. Scoring Gestures
• Scores
– Recognition Score
– Rating Score
• Scorers are ‘blind’ to the time of assessment

27. Usability Evaluations
• Observe participants using the tool
• Interview participants
• Interview carers (if relevant)
– When technology is installed
– After each practice phase

28. Usage Logs
• Record
– Number of sessions
– Length of sessions
– Levels of programme accessed
– Number of gestures recognised

29. Results so far
Usage

30. Mean Usage: 7 Participants
60
50
40
30
20
10
0
Days No of Time used Time per
available sessions (hrs) session
(mins)

31. Individual Usage: 3 participants
80
70
60
50
40
30
20
10
0
Days available No of Time used Time per
sessions (hrs) session (mins)

32. Usage x Recognition
80 300
70 250
60
50 200
40 150
30
20 100
10
0 50
Days No of Time used Time per 0
available sessions (hrs) session Recognition score
(mins)

33. Mean Usage over Phases
28 400
27 350
26 300
250
25 Supported
200
24 Independent
150
23 100
22 50
21 0
No of sessions Time spent (mins)

34. Usage: Levels
• Three participants use level 1 more than 2 & 3
• Two participants use all 3 levels and rate them
equally highly
• Two participants rate levels 2 & 3 more highly than 1
• Possibly contingent on navigation abilities

35. Usage Observations: Challenges
• Set up
– Lighting
– Positioning (e.g. wheelchairs)
– Security
• Glove
– Putting glove on the wrong hand
– Using the peg board (although often not
necessary)

36. Usage Observations: Challenges
• Starting and stopping
– Pressing key board buttons before menu has
appeared
– Not always pressing ‘off’ at end of session

37. Usage Observations: Challenges
• Navigation
– Variable use of OK, forward, back & menu buttons
– Variable navigation between levels
– Some unprincipled button pushing
Speed and competence may relate to prior
computer usage

38. Usage Observations: Challenges
• Gesture production
– Knowing when to gesture; waiting for 321 ping
– Knowing when the gesture has been recognised
– Variable use of cues; e.g. some adjust handshape
in response to glove image others do not

39. Usage Observations: Enjoyment
• All signal high enjoyment levels
– Thumbs up sign
– Drawn smiley face
• Positive reactions to level 2
– Game format
– Narrative context
– Environments

40. Usage Observations: Enjoyment
• Positive reactions to level 3
– Humour (spider, dentures)
– Stroke survivors as actors
– Presence of children

41. Other Observations
• Some target spoken words produced during
Gest use
• Spontaneous uses of practised gestures
(‘umbrella’ gestured when participant noticed
that it was raining outside; ‘child’ gesture
when talking about grandchild)

42. Results so far
‘Carer’ Comments

43. Independence of Use
• ‘She uses it all on her own, I don’t know how to
operate it’
• The first session I stayed with L, after that I’ve
helped only if she’s found something
particularly frustrating’
• All comment that the participant initiated use
of Gest

44. Enjoyment
• All say that the participant enjoyed Gest
• ‘he likes it when they clapped’
• ‘some of the gestures are particularly fitting
and she enjoyed rainbow’

45. Views about Technology
• ‘I was a technophobe and when they said
‘computer’ I thought it was going to cause
problems. I thought I wouldn’t understand
and he wouldn’t understand it. But it’s so
‘easy’

46. Reservations
• Carry over to real life (1 carer):
• ‘while she works on it here (points to
computer) it doesn’t necessarily translate’
• She wanted a hankie last night and didn’t
make a gesture’

47. Conclusions
• Gest was created through participative design
involving people with aphasia
– It offers 6 packages of hierarchical practice on 30 gestures
– It is accessible even to people with severe strokes
– It can be used successfully in diverse home settings
– It allows for flexible, self directed practice and is typically
intensively used
– It is enjoyable to use, with no reports of increased ‘carer
burden’

48. Conclusions
• But we do not know if
– Gest improves gesture production
– Gest improves spoken naming
– Effects generalise to unpractised targets
– Effects are maintained
• The results of the pilot study will give us
answers to these questions

49. Acknowledgements
The Research Councils UK Digital
Economy Programme
The Stroke Association
Consultants and their families
Participants and their families
Thank You
GReAT@city.ac.uk
www.soi.city.ac.uk/great