Introduces research methods that may be used for the study of socio-technical systems

1. Research methods for socio-
technical analysis
Ian Sommerville
Research methods, EngD course, 2012 Slide 1

2. Human and organisational
issues
• When designing, testing or evaluating critical systems we need to know
about how that system will be used in real settings
– What do users really do and why?
– What potential is there for error?
– How are errors avoided or mitigated?
• There are many useful generic insights into human factors
– For example the GEMS model
• But systems engineers will inevitably need to understand human factors
that are specific to the system they are constructing or maintaining
– Specifics emerge from: the location, the people, their work, the technologies
in use, etc.
• Process models are too abstract to tell us what we need to know about
what users do
Research methods, EngD course, 2012 Slide 2

3. Human and organisational
issues
• Because of the importance of human and organisational factors,
systems engineering draws from fields such as
– psychology
– sociology
– organisational studies
• But draws from their methods as much as their results
– Don‟t expect research done in other disciplines to provide all the necessary
answers for questions in systems engineering
– Don‟t expect that researchers from other disciplines will understand or be
interested in technology systems. Some are, many aren‟t
• Systems engineers have to
– Find out about human and organisational factors themselves
– Work closely with people from other disciplines
Research methods, EngD course, 2012 Slide 3

4. The devil is in the detail
• The fundamental principle of software design is
abstraction
– Find the commonalities between the entities in the system
and represent these as software elements
• This is based on the assumption that the common
features are the most important and that stakeholders
can adapt to them
• Social science perspective is rather different. The
focus is on the detail of a situation
– The key argument is that the details are very significant
indeed and that failure to pay attention to them leads to
problems
Research methods, EngD course, 2012 Slide 4

5. Studying socio-technical
systems
• Systems engineers need to find out about about human, social and
organisational issue that are specific to the system they are working on and to
the class of systems being developed
• There are several useful ways to do this
1. Experiments
2. Archive research / Case studies
3. Interviews and focus groups
4. Simulation
5. Fieldwork
• And several less useful ways (which are better than nothing)
1. Questionnaires and surveys
2. Imagining what users do
Research methods, EngD course, 2012 Slide 5

6. Case study analysis
• Retrospective analysis, looking back at a system or
incident (a case) that has occurred
• In-depth study, based on a detailed analysis of a set
of information sources
• Based around one or more research questions e.g.
– What were the factors in system X that led to catastrophic
failure?
– What management decisions contributed to the cost overrun
in system Y?
• Intention is to gain generally useful insights from
specific cases
Research methods, EngD course, 2012 Slide 6

7. Examples of case studies
• “Normal accidents‟
– The problem of nuclear reactor overheating at 3-mile Island
• Denver airport baggage system
• The London Ambulance Computer-aided Dispatch
System
• Airbus accident – Warsaw airport
• Case studies tend to focus on system failure rather
than system successes
Research methods, EngD course, 2012 Slide 7

8. Stages of case study analysis
• Determine and define the research questions
• Select the cases and determine data gathering and
analysis techniques
• Prepare and collect the data
• Collect data in the „field‟ (if appropriate to do so)
• Evaluate and analyse the data
• Prepare the report of the case study
Research methods, EngD course, 2012 Slide 8

9. Research questions
• Establish the focus of the study by forming questions about the
situation or the problem to be studied and so defining the study‟s
purpose
• The question or questions are then used to select material for
analysis
• Accident enquiries are focused case studies where the question
is decided in advance i.e.
– Why did accident X occur?
• Danger of bias in establishing the research question
– What management problems led to the collape of Lehmans?
– What were the systemic factors that led to the collapse of
Lehmans?
Research methods, EngD course, 2012 Slide 9

10. Data gathering and analysis
techniques
• Case study selection
– Often the most „typical‟ case study does not reveal the greatest
insights
– Choose case studies that are
• Extreme - representative of an exceptional situation e.g. an accident
• Critical - having strategic importance with respect to the general problem
• Data to be analysed
– Essential to use multiple sources of data
• Interviews
• Archives and reports
• Fieldwork
Research methods, EngD course, 2012 Slide 10

11. Data collection and analysis
• Typically, case studies generate large amounts of qualitative
data
– Transcripts of interviews
– Reports and associated notes
• Look for opportunities for triangulation
– Evidence from diverse sources that supports the same conclusions
• Field data is also important
– Look at the object of the case study as it is
– The system in operation
• Analyse the data in different ways and from different
perspectives
– Use quantitative data to corroborate qualitative data
Research methods, EngD course, 2012 Slide 11

12. Case study issues
• Generalisability
– Much of our learning is through case studies so we can make
generalisations from them. However, we have to be aware of
the possibility that specific factors may be dominant in any
one study and be careful about generalisation
– Case studies can be useful for finding „black swans‟ i.e.
contradicting received wisdom
• Hindsight bias
– Seeing events as more significant than they actually were
because there is a clear link between these events and the
research questions
• Obsolescence of results
Research methods, EngD course, 2012 Slide 12

13. Fieldwork - Get Out of the Office!
Research methods, EngD course, 2012 Slide 13

14. Fieldwork
• Generic name for a set of techniques which involve
collecting data directly from a worksite (the “field”),
rather than relying on reports from other people
• Encompasses a range of techniques, which have
much in common:
– Direct observation and discussion
– Ethnography
– Cultural and technology probes
Research methods, EngD course, 2012 Slide 14

15. Ethnography of work
• Situated observation of work with a view to
developing a detailed understanding of the way that
work is done and how the actors respond to other
actors and the context of the work
• The aim is to develop a „rich picture‟ of the work done
• Ethnomethodological ethnography
– Ethnography that is intended to describe what work is rather
than explain work in terms of some social theory
– Approach proposed in the 1980s at Xerox, developed at
Lancaster University in the 1990s.
– Work at Lancaster related ethnography explicitly to systems
design.
Research methods, EngD course, 2012 Slide 15

16. Ethnography in practice
• Looks at work in „real-time‟ rather than retrospectively
• The ethnographer „lives in the workplace‟ and
experiences it in the same way as the people
involved in the work
• Examines the processes and artefacts that take place
in the workplace
• The aim is to understand work as it really is rather
than work as defined or retrospectively described
Research methods, EngD course, 2012 Slide 16

17. Examples of ethnography
• The UK en route air traffic control system
• The London Underground control room
• Preparing and printing government documents
• Health systems
– Managers in the health service
– Breast cancer screening
– NHS Direct
Research methods, EngD course, 2012 Slide 17

18. Fieldwork Methods
• Get out of the office
– Talk to people about their work
– Observe practices
– Visit settings of use
• This can be part of an informal or formal study
– How formal or informal you are depends on:
• The formality of the results you wish to produce (Are you writing a study
to be read by others, or just trying to get a feel for a setting?)
• Your relationship to the people you research (Do they work in the same
organisation as you? Are they customers?)
• The setting (Some workplaces require different kinds of formalities, e.g.
steel mill workers vs. hospital workers)
Research methods, EngD course, 2012 Slide 18

19. Observe “ordinary practice”
• Don‟t wait for things to go wrong
– The routine and mundane are interesting (safe technology should be
mundane and routine!)
– Error is a judgement, are you in a position to judge what is an error?
• What is supposed to happen and what really happens are often
different
– People break the rules
– People have to go to extra effort to make the rules work
• How is the system used in practice?
• What extra things are necessary to make the system work?
• Look and listen, but don‟t judge. Users know their work better than you
do
• Observation is sometimes referred to as “Ethnography”
Research methods, EngD course, 2012 Slide 19

20. Fieldwork and systems
engineering
• “Traditional‟ ethnography is based on the
ethnographer spending prolonged periods in the
workplace
– Problematical when there are deadlines for procurement and
system specification
• Various techniques have been developed to address
this issue
– Concurrent fieldwork
– Evaluative fieldwork
– Quick and dirty fieldwork
Research methods, EngD course, 2012 Slide 20

21. Concurrent fieldwork
Ethnographic Systems
study Debriefing development
meetings
System prototype
Research methods, EngD course, 2012 Slide 21

22. Concurrent fieldwork
• An approach that can be used in conjunction with
iterative development or system prototyping
– The field worker can act as a „surrogate user‟ because of
their knowledge of the work and workplace
– However, unlike users, they can be more objective about the
system
• The debriefing meetings involve the fieldworkers and
system developers
• The developers set questions for the fieldworker
based on the prototype system
Research methods, EngD course, 2012 Slide 22

23. Evaluative fieldwork
Initial
specification or
design
Ethnographic
study Debriefing
meetings
Amended
specification or
design
Research methods, EngD course, 2012 Slide 23

24. Evaluative fieldwork
• An approach which is geared to „sanity checking‟
assumptions about a system or a system
specification
• The field worker reads and understands the
implications of the specification then makes
observations of the processes to be suppprted
• During the debriefing meetings, he or she discusses
the issues and problems that arose during the
fieldwork and the team discusses possible changes
to the specification to address these
Research methods, EngD course, 2012 Slide 24

25. Quick and dirty fieldwork
Outline
requirements
Quick and dirty Debriefing Requirements
fieldwork meetings engineering
More detailed
requirements
Research methods, EngD course, 2012 Slide 25

26. Quick and dirty fieldwork
• Short, focused studies to highlight the key issues of
concern in the ways in which work is done
• Aim of the work is to refine an initial specification
• Fieldwork is concentrated in a short period (1 day-1
week), with no iterations after debriefing meetings
Research methods, EngD course, 2012 Slide 26

27. Planning Fieldwork
• Plan
– But don‟t plan too much. Allow for the unexpected
– How long you need is contingent on what you are looking at and how long
people will put up with you
– Who you observe is also contingent on how long you have, and who will
have you. Don‟t always assume the „important‟ people are important (e.g.
secretaries can often play critical roles)
– Whether you make recordings or take notes depends on the setting, and
whether and how you want to write up examples afterwards
– Be ethical! This is not just about bureaucracy but about morality and
common sense
• Access Issues
– You may need to approach and negotiate with people: managers, staff,
trades-unions
Research methods, EngD course, 2012 Slide 27

28. Writing up results
• What you do after a field study depends on who the results are relevant to and
how formal the work was
– If it was sensitisation they may be nothing to do immediately, or you may
need to do a presentation or write a report and discuss it (Discussion is very
helpful)
• The value of the sensitisation should be relevant later - it will influence
subsequent decisions
– If the work was more formal and for wider dissemination (i.e. you work in a
large team, a long project and/or your work has wide relevance) you will
need to write more formal reports.
• First step should be to transcribe or write up what you have found
• Analysis can be done in a number of ways and is usually qualitative
rather than quantitative
– Field-studies often lead to useful contextual information and often negative
information rather than any functional or non-functional requirements.
Research methods, EngD course, 2012 Slide 28

29. Informal notes about lab-books
• She works over constant periods – maybe a day, maybe 10 minutes.
• Has done collaborations in past, but not now. Works for a „boss‟. Seems like
the usual working arrangement. She has two supervisors, one bioinformatician,
one biologist. Biologist doesn‟t mind about the methods (as in paper)
• The blasting etc will just be there in a paragraph. If you find errors you re-submit
correct versions of sequences.
• A lot of stuff looks similar as you work on it so a lab-book helps.
• When is it useful? When writing up. Very useful when writing up rather than
wading through files. When have collaborators for presentations or talks. When
you have a student you can just show them the process. Time- you can see if
someone published a suggestion before you went ahead and did it. When you
want to submit a sequence. When you have competitors. A collaborator did ask
the date once – I could go back. In bioinformatics I‟ve never had that.
Research methods, EngD course, 2012 Slide 29

30. Key points
• Case studies are a retrospective analysis technique that examine a
single case in depth
• Case studies rely on diverse information sources including (sometimes)
fieldwork
• Fieldwork is a way of finding out about human and organisational
factors relevant to a system
• Talk to users (stakeholders, developers, etc) and spend some time
observing what they do
• Shut up, look, listen. They are the experts, not you.
• Discuss your findings informally with other developers, and write up
your findings more formally if they are useful to a wide number of
people
Research methods, EngD course, 2012 Slide 30