Expert workshop report, Birmingham, February 2011 (C-SAP collections project)

Report on C‐SAP (Higher Education Academy Subject Centre for
Sociology, Anthropology and Politics) expert focus group and user
testing, February 2011
_________________________________________________________________________________

Isabelle Brent
C‐SAP, April 2011

This content is licensed under Creative Commons Attribution‐NonCommercial‐ShareAlike 2.0 UK:
England & Wales http://creativecommons.org/licenses/by‐nc‐sa/2.0/uk/

1

Contents

Expert Workshop Report ........................................................................................................................ 3
Trust .................................................................................................................................................... 3
Student Expectations .......................................................................................................................... 4
Resistance to Digital Materials ........................................................................................................... 4
Digital Resources as ‘Additional’ ......................................................................................................... 4
Personal Connection ........................................................................................................................... 4
Career Stage ........................................................................................................................................ 5
Preparation Rather Than Re‐Use? ...................................................................................................... 5
Cross‐disciplinary Materials ................................................................................................................ 6
Attitudes towards OERs ...................................................................................................................... 6
Google as Orientation ......................................................................................................................... 7
Individual Interviews Summary .............................................................................................................. 8
Dave Harris .......................................................................................................................................... 8
Sara Ryan ............................................................................................................................................ 9
.
Sean Moley.......................................................................................................................................... 9
Alan Bryman ...................................................................................................................................... 10
Antje Lindenmeyer ............................................................................................................................ 10
Kate Orton‐Johnson .......................................................................................................................... 10
User Testing Review .............................................................................................................................. 11
Relevance .......................................................................................................................................... 11
Interface Difficulties .......................................................................................................................... 11
An example of usability problems: ................................................................................................... 12
Barriers to Use in OER Sites .............................................................................................................. 14
Triton Blog ......................................................................................................................................... 16
Conclusion ............................................................................................................................................. 16


2

Expert Workshop Report

As part of the C-SAP Discovering Collections Project an expert workshop was held in
Birmingham on 24th February 2011 to investigate the discovery and use of digital and OERs
in research methods’ teaching. The workshop was attended by Alan Bryman, Dave Harris,
Sean Moley, Kate Orton-Johnson, Sara Ryan and Antje Lindenmeyer. The first session was a
focus group chaired by Graham Gibbs. Topics covered included what resources were used,
how trust was established in resources, disciplinary considerations and student engagement.

During the coffee break short individual interviews were filmed with the participants. The
rest of the workshop was devoted to a user testing session of Jorum, Xpert, Connexions,
MethodSpace and the Triton politics blog.

The major themes to emerge from both sessions are outlined below.

Trust
In general a preference was expressed for in-house materials that integrate with courses and
have been checked for quality. Reasons for this included trust issues and relevance.

Resources produced by other universities were cited – particularly good ones for methods
included Leicester for qualitative materials and Methods at Manchester.

In the user testing, preference was expressed for Google because of the quick recognition of
academic institutions through the URL in results pages. In contrast the affiliation of some
OER resources was not immediately clear which caused frustration.

In addition to university websites, book companion sites (such as those by Sage) were also
trusted to have relevant and accurate information.


3

Student Expectations
The importance of responding to student expectations were raised at various points during the
day which participants feel they must respond to in the contemporary political context
(several mentioned the National Student Survey).

Resistance to Digital Materials
One interesting theme to emerge from the workshop was an identified resistance to digital
learning resources amongst students who are not sure whether such materials count as being
‘properly academic’. In addition to this response there seemed to be a lot of confusion
amongst students as to what constituted ‘legitimate’ sources to refer to in assessed work and
how citations were made. It was suggested that assessment strategies have not really adapted
to include digital sources which accounts for student responses and universities should be re-
evaluating their assessment strategies.

Digital Resources as ‘Additional’
Participants also agreed that student expectations are informed by how digital resources are
presented. Most pointed students to online materials as additional or extension activities
which may communicate the notion that such materials are not really necessary. It was felt
that only at postgraduate level could such attitudes can be challenged in the context of
training researchers.

Personal Connection
During the user testing session one of the groups expressed a clear reservation about using an
OER teaching resource developed by someone else. A distinction was made between research
and teaching institutions and it was felt like students had an expectation in research-led
institutions that their teachers will use personal examples:

I think there is an element of students, particularly in the current climate of
thinking, ‘why are you just foisting me off with some other resource?’


4

This consideration was not just an expectation of students; all the group agreed that making
the connection between teaching and research was important and represented part of their
own professional identity:

there is kudos if you can have all your examples from things you have done

In addition the practical difficulties of using other materials for methods teaching were
referred to. For example, in teaching coding participants felt it would be hard to get access to
data which shows the various stages in coding. Their own data was trusted more because of
being able to refer to the different stages and having a detailed knowledge of the whole
process:

with your own data you have coded it and are familiar with it

Career Stage
All our participants were mid- or later career stage and all agreed that teaching resources
could be useful for lecturers starting out or moving into an area they were unfamiliar with:

If I was starting out as a lecturer, it would be good to see other people’s slides and
see how they do it

It could be helpful if you knew nothing and didn’t have lecturer guiding you

Preparation Rather Than Re‐Use?
The reservations about using teaching resources did not mean that participants felt such
materials were irrelevant for themselves. There was a general feeling that seeing other
people’s resources was positive in a more diffuse manner as part of a general educational
process. This perspective is indicated in the first quotation above where the emphasis is on
‘seeing how things are done’. Several participants said they would use the resources as part of
preparation and research but would construct their own teaching resource.


5

Cross‐disciplinary Materials
Participants did not see any difficulties using materials from other disciplines, however
finding examples that appealed to students from a wide range of backgrounds was a
challenge. Examples were given from health studies and education that students were able to
relate to on a personal level.

Attitudes towards OERs
A variety of opinions were expressed over OERs by participants reflecting their differing
exposure to the field. The majority had no experience of (consciously) using OERs. Most
participants take for granted the assumption that they can use online materials for teaching
purposes as long as it doesn’t go outside the institution. This places a major hurdle in the way
of accepting the value of OERs because the CC license is not seen as relevant.

Apart from the one participant who has already submitted materials to Jorum ere was no
enthusiasm about this aspect of OERs. Some participants cited intellectual property reasons;
not wanting to share a resource that has been worked hard on. More commonly the expressed
view was that participants could not guarantee that all elements of their resources were free
from copyright. This was cited as a reason why some institutions (such as Southampton) have
pulled back from making more materials open. A further institutional barrier was where the
university had copyright over teaching materials which made sharing impossible.

Above and beyond this, the OER sites that stressed the ease of which it was possible to
submit materials caused a general lack of trust amongst those unfamiliar with the concept of
submitting open materials. In particular the slogan ‘Creating content in Connexions is as easy
as 1, 2, 3’, caused both amusement but also mistrust of the site. One participant commented,
‘you could be anybody’. In discussion it was suggested that information about submitting
resources should not be on the homepage.


6

Evolving Attitudes
Despite these reservations the potential advantages of sharing materials was acknowledged.
During the workshop all agreed that a resource that had been quality checked would be useful
for methods. Doctoral training centres were suggested as providing a potential mechanism for
sharing resources. The challenge of how quality was to be assessed and by whom was
discussed.

Whilst not embracing OERs in the sense of wanting to make personal contributions to
repositories, attitudes did evolve both during the day and reflected in some post-workshop
feedback we received:

I came away thinking that I was made aware of a lot more online resources I was
using on a regular basis, but there were a lot more ‘teaching’ resources such as
presentations and slides that I could be using

This was a common sentiment. The habitual nature of online searching was acknowledged by
another participant that predisposes people to rely on ‘tried and trusted sites’. Another
participant suggested that OER could have a valuable role to play in providing methods
related examples and data-sets that will appeal to the diverse personal experiences students
bring to methods courses.

Google as Orientation
A key theme throughout the day was the pervasiveness of Google in determining how people
make sense of searching online. Phrases such as ‘like Google’ and ‘a kind of focussed
Google’ were common, particularly in the user testing.

Even when not mentioned, some of the difficulties described in the user testing below can be
understood in the context of how Google operates (such as assuming the left hand menu
represents a narrowing option rather than a new search, consistent use of font colour to
denote links and description of resource).


7

One participant also suggested that Google’s ability to target adverts has increased
expectations (as has Amazon) that results will be personalised. Now people expect similar
technologies in academic searches.

Individual Interviews Summary

During the break between the focus group and user testing the participants did individual
interviews that were filmed. Anna Gruszczynska asked what participants felt were the major
issues facing methods teaching and what elements they would like to see in an online
collection of research methods.

Dave Harris
Dave emphasised the need for methods to be embedded in the social sciences curriculum and
taught alongside substantive issues as opposed to being limited to a separate module.

Dave articulated the value of incorporating online resources in methods teaching as
supplementary materials but stressed the need for face-to-face teaching for communicating
core academic values.

The elements he identified as important to an online resource collection were defined in terms
of classification and stratification as well as a degree of agreement about suitability. However
he accepted that there needs to be diversity reflecting the different needs and preferences of
users (such as those preferring videos or PDFs).


8

Sara Ryan
Sara described the research group Oxford University she is part of at that uses qualitative
methods to collect people’s experiences of different health conditions (through filmed
interviews). The materials are freely available online and are used for methods courses at an
undergraduate and postgraduate level. This approach is helpful in communicating the
‘messiness’ of the research process and to get students to think critically about the research
process. Sara identified a difference in emphasis between the use of the resource teaching
different levels of students. With undergraduates more generic themes are pursued whereas
with postgraduates the interviews act as discussion points for issues such as validity and
transferability.

The elements Sara identified as important for an online resource collection were described as
well-organised, well referenced and well signposted. She argued that she wanted to have a
degree of trust in the materials but part of this was a subjective judgement process she must
engage in individually.

Sean Moley
Sean was speaking from the perspective of the NCRM who provide resources for advanced
methods teaching, some of which are delivered through online courses. He identified a lack
of resources available for advanced methods and described NCRM’s work to provide step-
by-step guides that takes learners through more advanced techniques.

The elements Sean described as important for an online resource collection was described as
a ‘collection of collections’ where providers of resources could present what they are
producing in a clear way that is easy to find. This he argued was better than needing to go to
the individual websites of different providers.


9

Alan Bryman
Alan highlighted the differences between teaching undergraduates and postgraduates with the
former being more difficult to persuade about the need for research methods teaching.
Resources should reflect the different needs and levels of undergraduate and graduate
students.

Alan described his own use of online resources as being private and supplementary to
enhance lectures rather than provide the core content.

The elements Alan described as important for an online resource collection was a gateway
model that would be an access point for different methods and analytical approaches.

Antje Lindenmeyer
Antje identified engagement with students as the most important issue facing methods
teaching, particularly in the health context within which she works. Antje described the
difficulties of teaching qualitative methods when judged in terms of quantitative criteria.

Kate Orton‐Johnson
[Recording problems made part of this interview inaudible]

Kate identified time as a key issue in using additional resources for courses. With the limited
time available to prepare, the number of resources that can be integrated is restricted.

The elements Kate described as important for an online resource was described as a sort of
portal model indexed by subject areas, interests, different methodological approaches and
ethical issues. The categorisation would be done in such a way so that users could draw on
generic or specific resources.


10

User Testing Review

In addition to the themes outlined in the previous section the user testing raised specific
issues that need to be considered separately.

The user testing looked at Jorum, Xpert, Connexions, MethodSpace and the Triton politics
blog. Participants were asked to find materials on qualitative interviewing in each of the OER
sites. The Triton blog was included as part of the reciprocal relations between the two
projects and to provide an example of an alternative approach to presenting OERs.

Four recurrent themes from user testing:

1. Lack of relevant results
2. Interface difficulties
3. Barriers to use
4. Difficulties downloading materials

Relevance
The user testing was coloured by the general lack of resources that participants could find on
the topic of qualitative interviewing. Trust is the sites was weakened by the appearance of
irrelevant results (such as ‘A history of ragtime music’, ‘Great unsolved mysteries in
Canadian history’) One participant commented, ‘how do you get to elementary algebra even
though you are looking for a resource on interviewing?’

Interface Difficulties
Part of the reason for not being able to locate relevant resources related to a lack of filtering
options. Advanced searches did not offer better options and it was felt that it ought to be
possible to narrow by alternative criteria to those offered such as:


11

 Subject
 Topics (within subject)
 Level (undergraduate/post-graduate)
 Resource type

Having to specify a subject or area from the outset in order to narrow down results was
disliked. For example, Merlot forces a choice between materials, learning exercises and
members that was considered unhelpful. Participants tried searching within the social science
area to see if more relevant materials would emerge but the process was cumbersome and
achieved no better results.

Two distinct approaches to searching for materials emerged. One participant (the only one to
have submitted OER materials) took what might be termed the ‘browsing’ approach, wanting
to explore different categories for the sake of it. However the rest of the participants were
focused on a particular search and wanted fast results to a specific query.

Within this latter approach the preferred search strategy was to perform and general search
and then to narrow down. Tag clouds were popular as well as the ability to select more than
one term. Participants tried searching within the social science area in Jorum to see if they
could find more appropriate results, however they did not like being restricted to this area and
found the number of stages cumbersome. The Boolean search features were not popular and
seemed to generate no better results.

An example of usability problems:
The Jorum website (which was liked on the grounds of aesthetics) demonstrates many of the
usability shortcomings. A screen shot is provided below1.

1
The Leicester e‐link button is specific to this particular computer and is not part of the reviewed websites


12

1. Misunderstanding/disliking the left hand menu
People initially thought the left menu was a means of narrowing down results rather than an
entirely fresh search. This is the natural area to look for such refining search techniques and
participants felt that there should have been options to narrow down the results at this stage.

2. Following the wrong link
Although the ‘Download Package’ button seems unambiguous, it was not the first choice
when participants attempted to locate the resource. The most common link participants tried


13

first was the ‘persistent link’ URL. The ‘web resource’ URL was also tried and confused
people because it appeared twice in results. The ‘Download Package’ was the third choice.
This may be because people expect links from specific features like URLs, particularly if a
colour is used to denote links. The Jorum site’s use of colour is inconsistent with regard to
links, for example sometimes the mid-teal colour is used to denote a clickable link (as in the
‘license’ information) but other times (as in the title) it isn’t.

3. Not scrolling down to the individual files
When the participants realised what the ‘Download package’ was they no longer looked for
information below it which provided the possibility in some cases to download individual
files.

Barriers to Use in OER Sites
Participants identified various ‘barriers’ in all of the OER sites that would put off all but the
most enthusiastic users. As one person commented,

There is so much on the internet, once you reach the barrier you just move on to
something that hasn’t got the barriers . . . I appreciate sites that are easy to use
and intuitive

Inadequate Information about Nature of Resource
Participants frequently asserted that they did not have enough clear information about what
the particular resource was before committing to download it. This information included:

 Clear short description of resource
 Author and institutional affiliation
 Description of what form resource takes (PowerPoint etc.)


14

Requiring Email Addresses
It was felt that having to put an email address in Jorum to get the resource was presenting
another hurdle:

You wouldn’t know how long it would take, will it come in a week’s time, will it end
up in spam when it comes?

Difficulties downloading materials
The time it took to conduct searches and to load materials caused frustration and confusion.
Users weren’t sure they had clicked the right option when there was no immediate response
and several said that they would give up.

Presentation/Accessibility
Participants commented on the ‘feel’ of the sites, preferring the presentation of Connexions
which is more like a standard website. Xpert was considered to be ‘uninspiring’, ‘boring’ and
‘flat’.

This issue goes beyond aesthetics and participants in some instances equated usability with
appearance. For example:

Participant 1: I don’t like the, it’s not a very useful interface

Facilitator: What’s not nice about it?

Participant 1: It’s ugly

The link between appearance and usability also relates to the organization of the information
on screen. Confusion was one of the central responses within the user testing session.
Reflecting towards the end of the session one participant commented:

I’m always struck by the mess of these things; it still looks messy, it is still hard to
find. There is still a lot of sifting to do. I don’t know how you would do it. I suppose


15

it is a combination of getting people to put better information on where stuff is
lodged in these repositories and then a better way of tidying it.

Concerns were also raised that the some of the sites (particularly Xpert and Jorum) used pale
fonts on white backgrounds that are difficult to read and might present accessibility issues for
those with a visual impairment.

YouTube videos were cited as useful resources for particular methods areas but again,
presentation was cited as a reason not to use them (such as bad sound quality).

Triton Blog
The Triton blog was presented at the end of the user testing as an alternative way of
presenting OERs tied to topical posts. The blog received a positive response; participants
liked the topicality of the posts and the tag cloud idea as a means of linking to resources. The
magazine-like presentation was also preferred to a ‘list of lists’.

However, there was an (inaccurate) assumption that because of the Oxford/Cambridge
affiliation that the OER resources generated through the Xpert widget had been reviewed by
the institutions.

Conclusion
The different activities that constituted the day workshop provided a wide range of valuable
information relating to methods teaching that will benefit the Collections Project. Since the
participants come from outside the OER community they provide an essential balance to the
perspective of those committed to OERs.

An unexpected (and possibly the most significant) issue to emerge was the reservation
relating to digital resources in general, not just OERs. For a variety of reasons digital
resources do not seem to be well integrated into higher education, particularly at


16

undergraduate level. Reasons seem to revolve around assessment which is still mostly
traditional, and student expectations. Without reform of how assessment is carried out the
value systems of students reflect this traditional approach where resources are valued as
‘academic’ according to a limited criteria. Problems citing digital resources compound this
difficulty as does the tendency of teaching staff to direct students to digital resources as
supplementary materials.

In addition to student attitudes towards digital materials, student expectations in general and
student engagement emerged as an important theme. It was recognised that methods can be
an unpopular and difficult subject to teach, particularly to students from diverse subjects and
backgrounds. This issue was more important than disciplinary considerations; in general
participants were happy to draw resources from different subjects to their own but described
the challenge of making them personally relevant to students.

Trust emerged as an important issue that lay behind the preference for in-house produced
materials. During the individual interviews several participants suggested some form of
gateway to methods resources was most appropriate, where materials would be vetted,
however during discussions the consensus was that with the volume of online materials
available now such a model is not sustainable.

The user testing indicated difficulties with all the OER sites used associated with inaccurate
search results and usability problems. This is reflected in one of the groups who unilaterally
decided to compare the results with a Google search and insisted that the Google search
produced better results. The significance of Google in orientating people’s approach to the
internet is important to acknowledge.

All but one of the participants was unwilling to submit materials to OER sites, however over
the course of the day attitudes towards OERs in general did change. It is clear that for most
participants the case for OERs has not been put persuasively within their own institution and
that were it, attitudes might change. This suggests that more progress might be made to
establish OERs into university teaching if it were part of the professional development of
teaching staff.


17

Expert workshop report, Birmingham, February 2011 (C-SAP collections project)

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (6)

Similar a Expert workshop report, Birmingham, February 2011 (C-SAP collections project)

Similar a Expert workshop report, Birmingham, February 2011 (C-SAP collections project) (20)

Más de CSAPSubjectCentre

Más de CSAPSubjectCentre (20)

Último

Último (20)

Expert workshop report, Birmingham, February 2011 (C-SAP collections project)