Research into how people find and share expertise can be traced back to the 1960s, with early studies focusing on knowledge workers such as engineers and scientists and the information sources they consult. However, in recent years there has been a growing recognition that the effectiveness of expertise retrieval systems is highly dependent on a number of contextual factors, where the emphasis is on how people search for expertise in the context of a specific task. These studies have typically been performed in an enterprise context, where the aim is to utilize human knowledge within an organization as efficiently as possible. This talk presents results of an Innovate-UK funded project investigating the use of complex search strategies in the workplace, with the aim of producing requirements for the design of next generation search tools.
2. April 2014: InnovateUK grant
Complex query formulation
Jan 2015: Smart Award
Proof of Market
3UXLabs - User Experience Research and Design - www.uxlabs.co.uk
3. Assess the potential markets
Qualitative research involving
end users from target sectors
Catch-22
Target sector <-> market
research
Who uses complex queries?
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 4
4. Usual suspects:
Patent agents
Lawyers
Unknown quantities:
Librarians?
Power users:
Media monitoring
Healthcare (systematic evidence review)
And…
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 5
5. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 6
6. Nested composite
Java AND (Design OR develop OR code OR Program) AND
("* Engineer" OR MTS OR "* Develop*" OR Scientist OR
technologist) AND (J2EE OR Struts OR Spring) AND
(Algorithm OR "Data Structure" OR PS OR Problem
Solving)
Enumerations of related terms
("looking for" OR "in search of" OR "open to" OR "new
job" OR "actively pursuing" OR "pursuing new" OR
"searching for" OR "new opportunity" OR "new
opportunities" OR "available for" OR "in transition"
OR unemployed OR "immediately available" OR
"currently seeking" OR "seeking new" OR "seeking a
new" OR "interested in")
Index field lookups
site:ca.linkedin.com "network engineer" "ccnp" "wan" "lan"
"vancouver" -intitle:"profiles" -inurl:"dir/ " -
inurl:job|jobs|jobs2
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 7
7.
8. Qualitative
Interviews with stakeholders
▪ 2-3 from each sector
Quantitative
Online survey
Baselines for comparison
▪ Patent search [Joho et al, 2010]
▪ Medical professional search [Geschwandtner et al, 2011]
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 9
9. 40 questions, 15 minutes
5 sections
Demographics:The background and professional
experience of the respondents
Search tasks:The types of search task that
respondents perform in their work
Query formulation:The approaches and techniques
used to construct queries
Evaluation: How they assess and evaluate the results
of their search tasks
Ideal search engine: Any other features additional to
those described above
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 10
10. Distributed via social media + SurveyMonkey
Qualifying question
“Is your primary job function to recruit and hire
professionals for your organization or for clients?”
09 June 2015 to 01 August 2015
416 responses, 69 qualified, 64 usable
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 11
11.
12. Clients external (48%), internal (34%), both
(17%)
patent searchers mostly internal (88%)
Job titles: recruiter (15%) HR Manager (8%)
and HR Generalist (7%)
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 13
0
2
4
6
8
10
12
14
16
18-24 25-31 32-38 39-45 46-52 53-59 60-65 Over 66
Numberofparticipants
0
5
10
15
20
25
30
35
40
45
50
Male Female
Numberofparticipants
13. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 14
0 5 10 15 20 25 30 35 40
Industry experience
Recruitment experience
14. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 15
0 10 20 30 40 50 60
Psychometric assessment
Other
Employer/brand consulting
Candidate selection
Candidate search
Number of participants
Services provided
0 10 20 30 40 50 60
Other
Commercial database
Open web
Proprietary internal database
Social networks
Job boards
Number of participants
Data sources
0
5
10
15
20
25
Always Often Sometimes Rarely Never N/A
Numberofparticipants
Use of previous examples
0 5 10 15 20 25 30 35
Nobody
Colleagues in my workgroup
Colleagues across my organisation
Public forums
Other
Number of participants
Query sharing
15. 3 hours to complete a search task of 5 queries, with
each query taking around 5 minutes to formulate.
iterative paradigm
▪ candidate search <-> candidate selection and evaluation
Shorter tasks (3 hours vs. 12 hours)
substantially longer than typical web search tasks
Fewer queries (5 vs. 15)
Same query formulation time (5 minutes)
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 16
Min Median Max
Search task completion time (hours) 0.06 3 30
Query formulation time (mins) 0.1 5 90
# queries submitted 1 5 50
16. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 17
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
Case sensitivity
Query translation
Truncation
Wildcard
Misspellings
Field operators
Weighting
Proximity
Abbreviations
Query expansion
Synonyms
Boolean
17. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 18
0 5 10 15 20 25 30 35 40
Manually, on paper
Text editor
Form based query builder
Visual query builder
Other
Number of participants
18. Time to assess relevance same as patent
searchers (5 minutes).
# results examined lower (30 vs. 100).
satisficing strategy
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 19
Min Median Max
Ideal # of results 1 33 1000
Number of results examined 1 30 100000
Time to assess relevance 1 5 50
19. Reflects Woudstra &Van den
Hooff (2008) selection criteria
topic of knowledge (job function)
physical proximity (location)
perspective (industry
sector/career level)
familiarity (previous contact with
recruiter)
availability
Different to medical searchers:
Type of source, date range,
language, etc.
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 20
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Previous contact with recruiter
Salary
Availability
Career level
Industry sector
Location
Job function
20. Different to healthcare
professionals
Woudstra &Van den
Hooff “source quality is
the most dominant factor
in the selection of human
information sources”
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 21
0 5 10 15 20 25 30 35 40
Other
Most recent item
Most trustworthy source
First item
Item which looks most relevant
Number of participants
21. UXLabs - User Experience Research and Design - www.uxlabs.co.uk 22
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50
Exporting search queries (histories)
Storing search results with an expiry date
Using facets/guided filtering
Search history
Saving custom lists from search results
Alerting function
Combining search queries
Combining multiple search results
Organising search queries
Recency of retrieved results
22.
23. Lengthy search sessions, different notions of
relevance, different sources searched separately,
specific domain knowledge
“The hardest part of creating a query is comprehending
new information and developing a mental model of the
ideal search result.”
Some of the most complex queries of any
community
‘Boolean Blackbelts’
Average time spent evaluating a typical result is
5 minutes (not 7 seconds!)
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 24
24. Selection of suitable terms is an ongoing challenge:
“The specific job is so new I cannot find terms used on
resumes to match”.
Satisficing strategies
“Generally speaking, it's a trade-off between time and
quality of results. [If] I can't identify the required information
in the available time... this is because the data is not present
in any of my go-to data sources, and working around that
limitation isn't the best use of the time”.
The search tasks are inherently interactive, multiple
iterations of query formulation and results evaluation
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 25
25. Unstudied community of information
professionals
New set of real world ‘people search’ scenarios
IR research assumes natural language queries
Creation of new test collections and tasks?
Publicly visible forum to compare and contrast
search strategies (Glen Cathey)
Translate IR research into practice
UXLabs - User Experience Research and Design - www.uxlabs.co.uk 26