Solr 4.0 dramatically improves scalability, performance, and flexibility. An overhauled Lucene underneath sports near real-time (NRT) capabilities allowing indexed documents to be rapidly visible and searchable. Lucene’s improvements also include pluggable scoring, much faster fuzzy and wildcard querying, and vastly improved memory usage. These Lucene improvements automatically make Solr much better, and Solr magnifies these advances with “SolrCloud.” SolrCloud enables highly available and fault tolerant clusters for large scale distributed indexing and searching. There are many other changes that will be surveyed as well. This talk will cover these improvements in detail, comparing and contrasting to previous versions of Solr.

1. Solr 4
Presented by Erik Hatcher
© Copyright 2012

2. About: Erik Hatcher
• “Lucene in Action”, co-author
- And also “Java Development with Ant”/”Ant in Action” co-author
• Open Source
- Apache Software Foundation: member, Lucene/Solr committer
and PMC
- Originator of “Blacklight”, a Solr-powered discovery interface
• LucidWorks
- Co-founder
- Recently renamed from Lucid Imagination
- Customer Support
© 2012 LucidWorks
2

3. Abstract
Solr 4.0 dramatically improves scalability, performance,
and flexibility. An overhauled Lucene underneath sports near
real-time (NRT) capabilities allowing indexed documents to
be rapidly visible and searchable. Lucene’s improvements
also include pluggable scoring, much faster fuzzy and
wildcard querying, and vastly improved memory usage.
These Lucene improvements automatically make Solr much
better, and Solr magnifies these advances with “SolrCloud.”
SolrCloud enables highly available and fault tolerant clusters
for large scale distributed indexing and searching. There are
many other changes that will be surveyed as well. This talk
will cover these improvements in detail, comparing and
contrasting to previous versions of Solr.
© 2012 LucidWorks
3

4. Lucene 4 Improvements
• Flexible index formats
• Pluggable scoring
• String -> BytesRef
• DWPT (Document Writer Per Thread)
- faster, more consistent indexing speed
• NRT (Near Real-Time)
• Spatial overhaul
• FST/FSA
- FuzzyQuery over 100x faster
- also reduces memory footprint for Terms index
• DocValues: aka column-stride fields
© 2012 LucidWorks
4

5. Flexible index formats
• For terms, postings lists, stored fields, term vectors, etc
• Several new posting list codecs
- Pulsing (inlines low doc freq)
- Block (packed int blocks)
- SimpleText (debugging, transparency)
- Bloom (experimental, also inlines low doc freq)
- Appending (for append-only filesystems such as HDFS)
- Memory (terms as FST)
© 2012 LucidWorks
5

6. Pluggable scoring
• Decoupled from traditional vector space (TF/IDF)
• Additional index statistics
- number of tokens for a term or field
- number of postings for a field
- number of documents with a posting for a field
• Several built-in alternatives:
- BM25
- DFR – divergence from randomness
- Information-based models
• “norms” are no longer limited to a single byte
- Similarity implementations can use any DocValues type to store
norms
© 2012 LucidWorks
6

7. String -> BytesRef
• How many bytes does a Java String require?
- BytesRef is now used to avoid this overhead
- Think of the internal structure as a big buffer with pointers
• Garbage collection much more efficient
- big blocks rather than zillions of small ones
• How much reduction? 10%? 20%?
- No. Way more than that
© 2012 LucidWorks
7

8. NRT: Near Real-Time
• Per-segment
- FieldCache needs to only load from new segments
• Soft commit
- Faster: does not fsync
- Can soft commit very rapidly, as low as every second
© 2012 LucidWorks
8

9. Lucene 4: there’s more
• AutomatonQuery
- term matching a provided finite-state automaton
• Term offsets
- optionally encoded into the postings lists and can be retrieved
per-position
• DirectSpellChecker
- finds possible corrections directly against the main search index
without requiring a separate index
• DWPT
- Flushing new segment is now concurrent w/ indexing
© 2012 LucidWorks
9

10. Indexing performance (Wikipedia 4KB docs)
• http://people.apache.org/~mikemccand/lucenebench/
indexing.html
© 2012 LucidWorks
10

11. QPS (primary key lookup)
• http://people.apache.org/~mikemccand/lucenebench/
PKLookup.html
© 2012 LucidWorks
11

12. FuzzyQuery
• http://people.apache.org/~mikemccand/lucenebench/
Fuzzy2.html
© 2012 LucidWorks
12

13. Solr 4 Highlights
• SolrJ streaming response
• Pivot facets
• New relevancy function queries
- termfreq, tf, docfreq, idf norm, maxdoc, numdocs, exists, if, and, or,
xor, not, def, and true and false constants
• DirectSpellChecker support
• Improved document response: DocTransformer, function
calculations
• Pseudo-join
• New admin UI: Including SolrCloud cluster visualizations
• Transaction log
• Several new update processors, including a “script” one
• Spatial overhaul
• Content-type savvy /update handler
• SolrCloud
© 2012 LucidWorks
13

14. Per-segment faceting improvement
• Field-cache, per segment
- Test index: 10M documents, 18 segments, single valued field
• facet.method=fcs
• Result set=100 docs, 100,000 unique terms
- static index fc=3ms fcs=244 ms
- quickly changing index fc=1388 ms, fcs=267 ms
• Result set=1,000,000 docs, 100 unique terms
- static index fc=26 ms fcs=34 ms
- quickly changing index fc=741 ms, fcs=94 ms
• Data from Yonik’s Lucene Revolution 2011 faceting talk
© 2012 LucidWorks
14

15. Solr 3.x scalability
• Capabilities:
- Replication
- Distributed search
• Limitations:
- Documents only available after (expensive) “hard” commit,
replication, and warming delays
- Configuration labor intensive, manually maintained and
coordinated
- Manual sharding: no automatic distributed indexing
- Failure recovery difficult if master goes down
© 2012 LucidWorks
15

16. SolrCloud: Solr 4’s scalability
• Sharded leaders and replicas
• ZooKeeper used for cluster management
• Distributed indexing
- Automatically distributes updates to appropriate shard
- Facilitates Near Real-Time (NRT) searching
• Distributed search
- Automatically distributes to nodes of each shard
• Robust, automatic update recovery
• Real-time /get
- Leverages transaction log
• No single point of failure
• Large scale NRT using soft commits
© 2012 LucidWorks
16

17. SolrCloud details
• “Leaders” and “replicas”
- Leaders are automatically elected
• Leaders are just a replica with some coordination
responsibilities for the associated replicas
• If a leader goes down, one of the associated replicas is
elected as the new leader
• New nodes are automatically assigned a shard and
role, and replicate/recover as needed
• CloudSolrServer
• Replication in Solr 4
- Used for new and recovering replicas
- Or for traditional master/slave configuration
© 2012 LucidWorks
17

18. NoSQL
• Update durability
- A transaction log ensures that even uncommitted documents are
never lost.
• Real-time Get
- The ability to quickly retrieve the latest version of a document,
without the need to commit or open a new searcher
• Versioning and Optimistic Locking
- combined with real-time get, this allows read-update-write
functionality that ensures no conflicting changes were made
concurrently by other clients.
• Atomic updates
- the ability to add, remove, change, and increment fields of an
existing document without having to send in the complete
document again.
© 2012 LucidWorks
18

19. Some numbers
• On a Wikipedia index (11M documents)
- Time to perform the first query with sorting (no warmup queries)
Solr 3x: 13 seconds, Solr 4: 6 seconds.
- Memory consumption Solr 3x: 1,040M, Solr 4: 366M. Yes,
almost a 2/3 reduction in memory use. And that’s the entire
program size, not counting memory used to just start Solr and
Jetty running.
- Number of objects on the heap. Solr 3x: 19.4M, Solr 4: 80K. No,
that’s not a typo. There are over two orders of magnitude fewer
objects on the heap in trunk!
• From an Erick Erickson blog entry (see Links slide)
© 2012 LucidWorks
19

20. Links
• Lucene/Solr: lucene.apache.org
• “Lucene in Action”: www.manning.com/lucene
• Blacklight
- projectblacklight.org
- Examples: search.lib.virginia.edu and searchworks.stanford.edu
• SearchHub.org
- Community/public content
- http://searchhub.org/dev/2012/04/06/memory-comparisons-
between-solr-3x-and-trunk/
© 2012 LucidWorks
20

21. About LucidWorks
• LucidWorks Search
- Lucene/Solr 4 powered
- On-premise or hosted (Amazon EC2 and Azure)
- Rich connector framework for SharePoint, web crawling, etc
- Built-in security support
• LucidWorks Big Data
- Scalable classification, machine learning, analytics
• Lucene/Solr commercial support
• Consulting
• Training
• http://www.lucidworks.com
© 2012 LucidWorks
21