Lucene @ Yelp provides various search services using Lucene including business search, phone search, list search, review search, and auto-completion. The services were originally too slow due to seeking across hard disks for the large index. The solution was to shard the index across multiple machines ("federation") and have a coordinator retrieve and combine results. Lucy is used for indexing and searching individual shards efficiently. Statistical modeling and simulations showed fewer hits needed to be retrieved from each shard to obtain the overall top results compared to naive approaches.

1. Lucene @ Yelp
Sudarshan Gaikaiwari

2. Bio
1. Over a decade of experience in information retrieval
2. Used IR techniques at Symantec's DLP group
3. Search Engineer at Yelp

3. Outline
1. Overview of search services at Yelp
2. Federation Motivation
3. Lucy Indexing
4. Lucy Searching
5. Efficiently Retrieving top k hits

4. The services we provide

5. Lucy: business search

6. Lucy also powers phone search

7. Cathy: she 'talks' a lot

8. Listsearch: it searches lists....

9. Reviewsearch: it searches reviews....

10. DYM: did you really mean that?

11. Suggest: auto completion

12. Federation Motivation

13. Problem
Search is too slow

14. Hard Disk Seek Latency
Disk seek 10,000,000 ns
Source Software Engineering Advice from
Building Large-Scale Distributed Systems
Jeffery Dean

15. RAM read latency
Main memory
reference
100 ns

16. Pinning Index in RAM
● vmtouch
● mlock
● http://hoytech.com/vmtouch/

17. Problem
Index is too large fit in memory on a single machine

18. Geographical sharding

19. Geographical Sharding drawbacks
1. Cumbersome manual process to determine shard boundary
2. No guarantee that a boundary can be found.

20. Federation
1. Split index across multiple machines
2. Shard on business id
3. TF-IDF scores from different machines should be
comparable

21. Mapping businesses to shards
1. Assigning businesses to shards
shard = shardlist[hash(business_id) % len(shardlist)]
Problems
1. Involves re-indexing all the businesses if we want to add a
new shard

22. Virtual Nodes

23. Advantages
1. Flexibility (move vbuckets from one shard to another)
2. Split hot spot shards

24. Lucy Master Slave Architecture
Separate indexing (masters)
A master for each shard of a service
Searching (slaves)
A slave for every replica of a service

25. Lucy Indexing

30. Lucy Searching

32. Federator: Combining results across
shards
1. Once we distribute an index across shards we need a
component which will search all these shards and combine
their results.
2. Written in Python (runs inside a python web process).
3. Uses Tornado IO loop to send requests to all shards.
4. The transfer protocol for the requests in JSON RPC

33. Lucy Server

36. Tokens to Business Attributes

37. Executing queries
1. Gather the top results for a query
2. Collect attribute statitics for attributes like places, categories

38. Lucene
1. Efficiently executes queries over the index
2. Provides how relevant the business is to the words in the
query (word score)
3. Upgrading lucene to 2.9/3.1 is WIP

40. Successive geobounds relaxation

41. Successive geobounds relaxation

42. Federation

43. Efficiently Retrieving top k hits
1. When user moves through multiple pages the number of
hits to be returned increases
num hits = start + count
2. So if we need to retrieve 500 hits the naive way would be to
retrieve 500 hits from each shard and then sort them

44. Distribution of hits in shards

46. Probability a hit is in a shard

47. Binomial Distribution
Probability (r of top k hits) are in a particular shard
Mean
Variance

48. Formula
Std Deviation
Formula

49. Simulation
Formula Hits selected from each Results Missed (%)
shard
k = 100
p = 0.2
24 0.017
32 0.0001407
44 0.00000

50. Simulation Graph

51. Results
1. ~ 50% savings over 100 hits (44 hits requested from each
shard)
2. 77% savings over 1000 hits (228 hits requested from each
shard)

52. Future work
1. In memory index
2. Move towards real time search

53. Come Join Us!

54. Thank You
smg@yelp.com