BioSolr, funded by the BBSRC, is a collaboration between open source search experts Flax and the European Bioinformatics Institute (EBI), aiming to significantly advance the state of the art with regard to indexing and querying biomedical data with freely available open source software

1. OCTOBER 13-16, 2016 • AUSTIN, TX

2. Searching the Stuff of Life: BioSolr
Matt Pearce & Alan Woodward
Senior Developers, Flax – www.flax.co.uk

3. •Building open source search applications since 2001
•Independent, honest advice and analysis
•Expert design & development, Apache Solr committers
•UK Authorized Partner of
•Test-driven relevancy and performance tuning
•Custom training & mentoring
02

4. 02

5. 01
•The European Bioinformatics Institute
•Part of the European Molecular Biology Laboratory
•Based on the Wellcome Genome Campus in Hinxton, Cambridge, UK.
•Maintains the world’s most comprehensive range of freely available and up-to-date
molecular databases, serving millions of researchers – indexing over 1 billion items.
•BioSolr project involves two teams from EMBL-EBI:
•Protein Data Bank in Europe (PDBe)
•Samples, Phenotypes and Ontologies (SPOt)

6. 02
The genesis of BioSolr
•Grant Ingersoll visits the Wellcome Campus in July ’13
•Around 90 people attend
•Show of hands indicates 75% using Lucene/Solr
•Sameer Velankar of EMBL-EBI identifies grant funding
•Flax and EMBL-EBI apply successfully to the BBSRC

7. 03
BioSolr
•One year BBSRC funded project from September 2014
•“to significantly advance the state of the art with regard to indexing and querying biomedical data
with freely available open source software”
•Outputs:
•Workshops
•Papers & presentations
•Software (Open Source, of course!)
•Documentation
•Inputs: from the PDBe & SPOt teams

8. 01
BioSolr
•Tom Winch
•Working on site with Sameer Velankar & the PDBe team
•Facet.contains, Xjoin, Federated Search
•Matt Pearce
•Working on site with Tony Burdett & the SPOt team
•Indexing ontologies

9. 01
BioSolr & PDBe - Introduction
•Protein Data Bank (PDBe)
•facet.contains (autosuggest)
•https://issues.apache.org/jira/browse/SOLR-1387
•In Solr 5.1
•Xjoin (searching external sources)
•https://issues.apache.org/jira/browse/SOLR-7341
•Federated search

10. 01
Xjoin concepts
•The problem – you have data in an external data store which is not suitable
for indexing in Solr
•The data may be from a live source, for example.
•You need to match data from your search results against data from one or
more of these external sources for display or analysis.

11. 01
Xjoin implementation
•Xjoin is implemented as a Solr search component.
•There should be one configured instance per external source.
•XJoinResultsFactory interface defines the search behaviour:
•Communicates with the external source to carry out the query
•Configured in solrconfig.xml, along with presets
•Returns results as an XjoinResults object
•Results are keyed by a string ID, defined in the configuration
•User is required to provide the implementation of this interface for each
external source being used

12. 01
Xjoin configuration
•Configure the XJoinSearchComponent in solrconfig.xml with details of your
XJoinResultsFactory implementations
•Add the search component to the search request handler
•Needs to be in both first-components and last-components sections.

13. 01
Xjoin results handling
•Uses a query parser based on TermsQParserPlugin
•Allows the same methods as TermsQParserPlugin
•Enables the use of multiple external sources, joined using Boolean
operators.
•Results are returned in a separate block
•Similar to how highlights are returned.

14. 01
Federated search - introduction
•Problem: we need to search data sets split across multiple locations (even
different countries)
•Records may contain different fields.
•Similar to pre-SolrCloud distributed search

15. 01
Federated search challenges: result counts
•The same document may appear in more than one shard, so they need to be
aggregated.
•Also applies to facet counts.
One solution:
•Shards return all document IDs rather than the number found.
•Aggregator builds a set of unique documents from the returned results.
•Simple for small result sets but inefficient for large sets.
•Estimate the result count, using statistical methods:
•If two shards always return similar counts, overlap likely to be high;
•If they don’t, overlap will be small so dataset can be treated as independent and
number found added to total.

16. 01
Federated search challenges: merging document sets
•Problem: documents are not unique across data sets.
•Default behaviour is to use the first instance of a document, ignoring others.
•Datasets may contain different fields – we need all versions.
One solution:
•Use a custom MergeStrategy to build the ID list.
•Cannot use a Grouper – prevents result grouping in the query.
•Scoring is also a challenge!

17. 01
Federated search challenges: merging document data
•Problem: document data may not be the same across data sets.
A solution:
•Merge documents together into a single, composite document.
•Potentially use an aggregation schema describing merge process.
•Merge strategy must be capable of merging disparate field types.

18. 01
BioSolr & SPOt – Indexing ontologies
Washington, N. & Lewis, S. (2008) Ontologies: Scientific
Data Sharing Made Easy. Nature Education 1(3):5

19. 01
Indexing ontologies – the problem
•You have a collection of documents annotated with ontology references.
•You want to search both the documents and the associated ontology data.
•This may include associated nodes – “has location”, “is part of”, etc.
•Faceting the ontology references would be nice!
•(especially if the facets can be presented in a tree)

20. 01
Approach 1
Keep the data separate
documents
Documents
Indexer
Documents
Indexer
ontology
Ontology
Indexer

21. 01
Approach 1 - steps
•Index the documents with the node annotations but no further
detail.
•Index the ontology in its own core.
•Search the documents, then cross-match against the ontology.
•BUT requires multiple calls, doesn’t allow searching both
cores at the same time.

22. 01
Approach 2
Add some ontology data to your documents
Documents
Indexer Ontology
documents

23. 01
Approach 2 – step 1
•Index the documents with the node annotations.
•While indexing, look up the node references, with their labels
and synonyms.
•Easier to include the ontology references in your search.
•Can boost fields as required.
•Faster to search

24. 01
Approach 2 – step 2
•Expand the ontology data being stored.
•Include single (or multi) level parent and child nodes, with their
labels.
•Use dynamic fields to store additional relationships.
•Dynamic fields allow searches across specific relationship
types (“is part of”, “has location”, etc.)
•BUT requires additional Solr look-ups to be fully dynamic
•(using /admin/luke to look through the current schema for dynamic fields).

25. 01
Approach 2 – search screen
•General search box
•Options to include child and parent labels (one step
removed)
•Dynamically-generated additional relationship search
options

26. 01
Approach 2 – search results

27. 01
Approach 2 – final result
•We have developed an UpdateProcessor to do this as part of the update
chain.
•The user defines the ontology location (file, URL) and the field to
reference.
•Aims for convention over configuration for remaining properties.
•Field names and ontology annotation details all customisable.
•A similar plugin for ElasticSearch has also been developed.

28. 01
Aside: facet trees
•Additional search component to return facets from ontology references in
tree form.
•Extends FacetComponent
•Takes initial facets from results, searches hierarchical references to build
the tree during facet generation.
•Avoids multiple calls to Solr from client-side to build the tree.
•A second collection can be used to search the ontologies...
•… BUT it must be part of the same Solr instance

29. 01
Aside: facet trees - challenges
•Nodes with multiple parents may appear more
than once…
•… not yet found a solution for this!
•Default behaviour is to return entire tree.
•Not always useful – may need to drill through
several layers to get to useful entries.
•Solution: prune the tree!

30. 01
Aside: facet trees - pruning
•Multiple pruning options available:
•Simple pruning: remove layers with no useful
information.
•Datapoint pruning: return the highest counts at
top-level, remainder in “Other” section.

31. 01
Approach 3
•Search the ontology, and cross-match with the documents.
•Allow SPARQL queries over the ontology index.
•Enables complex searches over ontology relationships.
•(SPARQL is a semantic query language)

32. 01
Approach 3

33. 01
Adding Apache Jena
•We use Apache Jena to provide TDB-querying with SPARQL.
•Jena uses Solr to search specified text fields – set via configuration files.
•Uses its own Triple Store for other fields and relationship queries.
•Return the reference URI in the returned fields to cross-match with
documents.
•Use a filter query to choose the matched documents.
•Not that different from Xjoin!

34. 01
Generic ontology indexer
•Stand-alone application to index different ontologies
•Allows separate configuration for each ontology
•Plugins may be used:
•At item level, to external data some/all nodes;
•At ontology level, to push the ontology into MongoDB, etc.
•Cross-pollination with the EBI's Ontology Lookup Service site (currently in
beta)

35. 01
Conclusions – what have we achieved?
•Searching across multiple external datasets (xjoin)
•See SOLR-7341 (and please up-vote!)
•Searching across multiple Solr nodes across different campuses.
•Indexing ontologies – both with and without document data.
•Enriching documents with ontology data.
•(…and facet trees)

36. 01
Get Involved!
•Check out the github page: https://github.com/flaxsearch/BioSolr
•Vote for Xjoin: https://issues.apache.org/jira/browse/SOLR-7341

37. 01
Upcoming events
•7th December: SWAT4LS tutorial session in Cambridge, UK
•Will cover ontology indexing and search
•http://www.swat4ls.org/workshops/cambridge2015/
•3rd/4th February: Workshop at the EBI campus, Hinxton, UK
•http://www.ebi.ac.uk/pdbe/about/events/open-source-search-bioinformatics

38. 01
Thank you for listening!
Matt Pearce
matt@flax.co.uk
www.flax.co.uk/blog
+44 (0) 8700 118334
Twitter: @flaxsearch