Presented by David Giffin, Software Engineer, Etsy - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Search at Etsy poses significant challenges. Our marketplace is filled with millions of unique, short-lived items and people trying to find them over 13 million times a day. In this session we'll discuss many of the solutions we've engineered to meet these challenges including, the evolution of indexing at Etsy, how HBase and Hadoop have taken indexing from hours to minutes, how and why we use bittorrent for Solr replication, how we track search performance, our approach to shave crucial milliseconds off every search, and an overview of our continuous deployment strategy, web / search config integration and A/B testing and analytics.

1. Solr, Lucene & Hadoop
@
Thursday, May 10, 12

2. david@etsy.com
4 Years Lucene and Solr @ Etsy
Thursday, May 10, 12

3. History of Search @ Etsy
Hadoop + HBase Indexing
(in development)
Replication
Thursday, May 10, 12

4. About
Us
Thursday, May 10, 12

5. Thursday, May 10, 12

6. Thursday, May 10, 12

7. Thursday, May 10, 12

8. 13MM Listings
39MM Unique Visitors
880K Shops / 150 Countries
100+ Engineers
Thursday, May 10, 12

9. Architecture
Overview
Thursday, May 10, 12

10. Overview
Search Web Database
+n slaves +n webs +n db shards
Memcached
+n caches
Thursday, May 10, 12

11. Thrift
Search Web
slave web
query = hats for cats
slave web
result = 402, 283, 837
+n slaves +n webs
Thursday, May 10, 12

12. Hydration
Database
shard
shard
Web
web
+n shards
web
Memcached
+n webs
cache
cache
+n caches
Thursday, May 10, 12

13. The Results
Thursday, May 10, 12

14. History of Search
at Etsy
Thursday, May 10, 12

15. History of Search
2007
•1 Million Listings
•A Single “Master” Postgres Database
•PHP > Twisted > Stored Proc > TSearch
•18 “Baby” Postgres Databases
•Baby Replicator
Thursday, May 10, 12

16. History of Search
2008
•2 Million Listings
•A Single “Master” Postgres Database
•PHP > Solr
•4 Solr Slaves + 2 Masters
•Baby Replicator + DIH for Reindexing
Thursday, May 10, 12

17. History of Search
2009
•4 Million Listings
•A Single “Master” Postgres Database
•PHP > Solr
•6 Solr Slaves + 2 Masters
•Webs >ActiveMQ > Solr
Thursday, May 10, 12

18. History of Search
2010
•7 Million Listings
•A Single “Master” Postgres Database
•PHP > Thrift > Solr
•10 Solr Slaves + 1 Master
•Custom Import Handler
Thursday, May 10, 12

19. History of Search
2011
•10 Million Listings
•“Master” Postgres Database + DB SHARDS!
•PHP > Thrift > Solr
•24 Solr Slaves + 1 Master
•Custom Import Handler
Thursday, May 10, 12

20. Future of Search
2012
•?? Million Listings
•MORE DB SHARDS!
•PHP > Thrift > Solr
•?? Solr Slaves + 1 Master
•HBase + Hadoop Indexers
Thursday, May 10, 12

21. What Did We Learn?
Thursday, May 10, 12

22. Lucene + Solr > TSearch
http://www.depesz.com/2010/10/17/why-im-not-fan-of-tsearch-2/
Thursday, May 10, 12

23. Love Lucene + Solr Trunk!
Thursday, May 10, 12

24. Run, Don’t Walk...
Thursday, May 10, 12

25. Deployinator
Fork it: https://github.com/etsy/deployinator
Thursday, May 10, 12

26. Smoker
Thursday, May 10, 12

27. StatsD, Graph Everything!
Fork it: https://github.com/etsy/statsd
Thursday, May 10, 12

28. Thursday, May 10, 12

29. 95th Percentile
Thursday, May 10, 12

30. start · build_query · perform_search · receive_search_ads · search_side_response ·
create_event_logger · set_tpl_vars · tpl_render · receive_search_ads_post_render
Thursday, May 10, 12

31. Solr Top Level Cache > Memcached
Thursday, May 10, 12

32. etsy-index.properties
$ cat /search/data/person/index/etsy-index.properties
#Tue Mar 27 13:05:51 EDT 2012
max_update_time=2012-03-27T17:05:51.955Z
Thursday, May 10, 12

33. Check Index Size
Don’t Install if < 50% Current Size
Thursday, May 10, 12

34. Check if Index is Too Old
Don’t Update if > 10 Days Old
Thursday, May 10, 12

35. What Did We Learn?
Store Nothing
Thursday, May 10, 12

36. Keep Denormalized Data
Thursday, May 10, 12

37. DB Shard
PHP JSON Search
DB Shard Denormalizer Database
DB Shard
Thursday, May 10, 12

38. Full Apply
Install
Reindex Incremental
Thursday, May 10, 12

39. Full Apply Apply
Install
Reindex Incremental Incremental
Thursday, May 10, 12

40. r
Database
exe
Ind
Thursday, May 10, 12

41. HBase + Hadoop
Indexing
Thursday, May 10, 12

42. HBase + Hadoop Indexing
Why HBase?
Thursday, May 10, 12

43. HBase + Hadoop Indexing
DB Shard
PHP JSON
DB Shard Denormalizer HBase
DB Shard
Thursday, May 10, 12

44. HBase + Hadoop Indexing
listings_denormalized
{NAME => 'listings_denormalized', FAMILIES =>
[{NAME => 'listing_data', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', COMPRESSION =>
'SNAPPY', VERSIONS => '1', TTL => '-1', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}
Thursday, May 10, 12

45. HBase + Hadoop Indexing
listings_denormalized_modified_index
{NAME => 'listings_denormalized_modified_index',
FAMILIES => [{NAME => 'pks', BLOOMFILTER =>
'ROW', REPLICATION_SCOPE => '0', COMPRESSION
=> 'SNAPPY', VERSIONS => '1', TTL => '-1', BLOCKSIZE
=> '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}
Thursday, May 10, 12

46. HBase + Hadoop Indexing
SOLR-1301
https://issues.apache.org/jira/browse/SOLR-1301
Thursday, May 10, 12

47. HBase + Hadoop Indexing
Solr
Disk
•Solr Document Converter
Output Format
•Solr Requires Posix Disk
HDFS •Index Copied Back to HDFS
Thursday, May 10, 12

48. HBase + Hadoop Indexing
•Not Great with Multi-Core Configs
•Added Solr Multi-Core Support
• Solr Config Issues
•Added ENV support for Configs
•Uses “new” style Hadoop API
•Added Support for both Old and New
Thursday, May 10, 12

49. HBase + Hadoop Indexing
SolrInputDocumentWritable
public class SolrInputDocumentWritable extends SolrInputDocument
implements org.apache.hadoop.io.Writable {
Thursday, May 10, 12

50. HBase + Hadoop Indexing
Oozie
Thursday, May 10, 12

51. HBase + Hadoop Indexing
Oozie + HBase?
Thursday, May 10, 12

52. HBase + Hadoop Indexing
ScanStringGenerator
http://blog.ozbuyucusu.com/2011/07/21/using-hbase-tablemapper-via-oozie-workflow/
Thursday, May 10, 12

53. HBase + Hadoop Indexing
Hadoop Indexer
Oozie Start
Map HBase Copy
Reduce HDFS Merge
Solr
Disk Install
Output
Thursday, May 10, 12

54. HBase + Hadoop Indexing
IndexerActionMain
Thursday, May 10, 12

55. HBase + Hadoop Indexing
Deployinator
Thursday, May 10, 12

56. HBase + Hadoop Indexing
IndexCompare
Thursday, May 10, 12

57. HBase + Hadoop Indexing
$ ./compare
ERROR: please provide two index directories
example: ./compare -p 0.1 -i user_id ./index ./index-1332867952588
options:
-p --percent= percent of the index to check
-i --id= primary key id field in the index
-h --hash= comparison or hash field in the index
<index> <index>
Thursday, May 10, 12

58. HBase + Hadoop Indexing
$ ./compare
/search/data/person/index-1332867952588/
/search/data/person/index-1335378487672
id field: user_id
hash field: hash
percentage: 0.0010
files: /search/data/person/index-1332867952588/ /search/
data/person/index-1335378487672
/search/data/person/index-1332867952588 contains 1515512 docs
/search/data/person/index-1335378487672 contains 14837972 docs
1516 of 1516 documents are the same
Thursday, May 10, 12

59. HBase + Hadoop Indexing
Copy and Merge
Thursday, May 10, 12

60. HBase + Hadoop Indexing
Open Source
Thursday, May 10, 12

61. Replication
Thursday, May 10, 12

62. Replication
Thursday, May 10, 12

63. Replication
Slaves
Master
+n slaves
Thursday, May 10, 12

64. Thursday, May 10, 12

65. BitTorrent
Replication
Thursday, May 10, 12

66. Bit Torrent
Using BitTornado:
Thursday, May 10, 12

67. Replication
Bit Torrent + Solr
Thursday, May 10, 12

68. Replication
Bit Torrent + Solr
Thursday, May 10, 12

69. Thursday, May 10, 12

70. Thursday, May 10, 12

71. Replication
Fork of TTorent: https://github.com/etsy/ttorrent
Multi-File Support
Large File Support
Fork BitTorrent: Comming Soon
Thursday, May 10, 12

72. Need a job?
Thursday, May 10, 12

73. Thursday, May 10, 12

74. Thanks!
Thursday, May 10, 12

75. david@etsy.com
Thursday, May 10, 12