Smarter search drives value to your business. Delivering search that matches users to the right content is what you care about. But organizations often get stuck getting there. It turns out that you need quite a number of very different ingredients to deliver tremendous search. It can make your head spin! To help you think through where your team is on its road to smarter search, Pugh introduces the maturity model used by OpenSource Connections and walks you through a very concrete method to inventory needed skills and translate that into search roles for your team. He shows how to measure your capabilities in key areas of search to drive better ROI from search.

1. www.opensourceconnections.com
How to Structure
Your Search Team
for Success
Enterprise Search & Discovery 2020 | Eric Pugh | @dep4b | www.opensourceconnections.com

2. www.opensourceconnections.com

3. www.opensourceconnections.com
Who am I?
● Co-founder of OpenSource Connections
● Apache Software Foundation member
● Committer on the Apache Solr project
● Steward of Quepid: A Query Relevance Workbench
● Creator of Chorus: An Open Source Search Stack for Ecommerce
● Member of The Search Network

4. www.opensourceconnections.com
Avi AgnesMartin Miles HelenCharlie
Chat
with
us!

5. www.opensourceconnections.com
How to Structure
Your Search Team
for Success
Enterprise Search & Discovery 2020 | Eric Pugh | @dep4b | www.opensourceconnections.com
“The Magic
Ingredient”

6. ESS 2020 | Eric Pugh | @dep4b
People are the
Magic Ingredient!
Sebastian from Tudock leading the
conversation last year....
6

7. ESS 2020 | Eric Pugh | @dep4b
People are the
Magic Ingredient!
The COVID version of group work.... ;-)
7

8. 8

9. www.opensourceconnections.com
Search Team Maturity Model
Business
Understand
User Needs
Search &
Discovery
Tech
Experiment
Driven
User
Experience
Enrichment
Advanced
Data
Inventory
Business
stakeholders
use real-time
KPIs
Producing
quality data
from analytics
Develops
custom plugins
Ops supports
A/B testing &
offline tests
Innovative
Discovery
(chatbots, etc)
NLP & Data
science team
Varied,
complex,
large-scale
data
Practitioner
Occasional
reporting
Some user
testing / basic
analytics
Complex
relevance
config; uses
plugins
Available, but
complex
experiments
UI supports
findability
Taxonomies /
Ontologies
Moderate data
complexity
Baseline
Business
impact not
measured
No query logs
or user testing
Stock or
moderately
tweak config
Test discovery
manually,
deployed
rarely
10 search links
on page
Minor
enrichment
(synonyms)
Very simple
data model

10. 10

11. www.opensourceconnections.com
Search Team Maturity Model
Business
Understand
User Needs
Search &
Discovery
Tech
Experiment
Driven
User
Experience
Enrichment
Advanced
Data
Inventory
Business
stakeholders
use real-time
KPIs
Producing
quality data
from analytics
Develops
custom plugins
Ops supports
A/B testing &
offline tests
Innovative
Discovery
(chatbots, etc)
NLP & Data
science team
Varied,
complex,
large-scale
data
Practitioner
Occasional
reporting
Some user
testing / basic
analytics
Complex
relevance
config; uses
plugins
Available, but
complex
experiments
UI supports
findability
Taxonomies /
Ontologies
Moderate data
complexity
Baseline
Business
impact not
measured
No query logs
or user testing
Stock or
moderately
tweak config
Test discovery
manually,
deployed
rarely
10 search links
on page
Minor
enrichment
(synonyms)
Very simple
data model

12. www.opensourceconnections.com
Critical Areas needed for Learning to Rank!
Business
Understand
User Needs
Search &
Discovery
Tech
Experiment
Driven
User
Experience
Enrichment
Advanced
Data
Inventory
Business
stakeholders
use real-time
KPIs
Producing
quality data
from analytics
Develops
custom plugins
Ops supports
A/B testing &
offline tests
Innovative
Discovery
(chatbots, etc)
NLP & Data
science team
Varied,
complex,
large-scale
data
Practitioner
Occasional
reporting
Some user
testing / basic
analytics
Complex
relevance
config; uses
plugins
Available, but
complex
experiments
UI supports
findability
Taxonomies /
Ontologies
Moderate data
complexity
Baseline
Business
impact not
measured
No query logs
or user testing
Stock or
moderately
tweak config
Test discovery
manually,
deployed
rarely
10 search links
on page
Minor
enrichment
(synonyms)
Very simple
data model

13. www.opensourceconnections.com
Critical areas for measuring search!
Business
Understand
User Needs
Search &
Discovery
Tech
Experiment
Driven
User
Experience
Enrichment
Advanced
Data
Inventory
Business
stakeholders
use real-time
KPIs
Producing
quality data
from analytics
Develops
custom plugins
Ops supports
A/B testing &
offline tests
Innovative
Discovery
(chatbots, etc)
NLP & Data
science team
Varied,
complex,
large-scale
data
Practitioner
Occasional
reporting
Some user
testing / basic
analytics
Complex
relevance
config; uses
plugins
Available, but
complex
experiments
UI supports
findability
Taxonomies /
Ontologies
Moderate data
complexity
Baseline
Business
impact not
measured
No query logs
or user testing
Stock or
moderately
tweak config
Test discovery
manually,
deployed
rarely
10 search links
on page
Minor
enrichment
(synonyms)
Very simple
data model

14. KMWorld/ESD 2020 | Eric Pugh | @dep4b
Maturity Model lays
out where you are and
where you want to be,
now do you have the
team to get there?
14

15. KMWorld/ESD 2020 | Eric Pugh | @dep4b
How it started...
Our non-quantitative understanding of teams...
Business Stakeholders - business sponsors of the search team. Who sits outside the search team and holds them accountable to meeting
goals/metrics/milestones?
Search Team Leadership - who is accountable to search's performance to the organization?
Analytics Developers - who wires up the UI and collects analytics on search's performance?
Relevance Engineers - who is responsible for manipulating a search algorithm to return content that matches user's intent?
QA / Qualitative Feedback - who tests search relevance (and other) quality issues and reports them to the team?
Expert User - who on the team is able to stand in for the user?
Search Ops Engineers - who manages scaling, deployment, and search infrastructure?
Search UI/UX devs - who develops the search UI?
Search UI/UX designers - visual designer of how the search UI is interacted with
Search API devs - developers that work on the API that fronts search
Taxonomists - someone tasked with linking concepts in a vocabulary (for building concept search)
Data Scientists - tasked with augmenting data / queries / user metadata to augment the ranking task. Often has an NLP role with a company

16. KMWorld/ESD 2020 | Eric Pugh | @dep4bStep 1: Brainstorming “The Work” of the Search Team 16

17. ESS 2020 | Eric Pugh | @dep4b
Step 2: Cluster the
Tasks
17

18. KMWorld/ESD 2020 | Eric Pugh | @dep4bStep 2: Clustering Tasks Together 18

19. KMWorld/ESD 2020 | Eric Pugh | @dep4bStep 3: Naming the Roles 19

20. KMWorld/ESD 2020 | Eric Pugh | @dep4b
Trying to Learn to Rank before they can
Walk!
20

21. KMWorld/ESD 2020 | Eric Pugh | @dep4b
“App and Ops Focused” Team Starting to
Think about Relevance
21

22. KMWorld/ESD 2020 | Eric Pugh | @dep4b
Roles of a
Search Team -
Raw Numbers
6 Teams worked
with
367
Responsibilities
written down
56 Roles
identified

23. KMWorld/ESD 2020 | Eric Pugh | @dep4b
Roles of a
Search Team -
Raw Numbers
Responsibilit
ies tied to
Role

24. KMWorld/ESD 2020 | Eric Pugh | @dep4b
Roles of a
Search Team
How it
ended...

25. www.opensourceconnections.com
Thank you! Stay safe, and let's
keep in touch!
Email: epugh@opensourceconnections.com
Twitter: @dep4b