2. Solr Compute Cloud – An Elastic
Solr Infrastructure
Nitin Sharma
- Member of technical staff, BloomReach
- nitin.sharma@bloomreach.com
3. Abstract
Scaling search platforms is an extremely hard problem
• Serving hundreds of millions of documents
• Low latency
• High throughput workloads
• Optimized cost.
At BloomReach, we have implemented SC2, an elastic Solr infrastructure for big data applications
that:
• Supports heterogeneous workloads while hosted in the cloud.
• Dynamically grows/shrinks search servers
• Application and Pipeline level isolation, NRT search and indexing.
• Offers latency guarantees and application-specific performance tuning.
• Provides high-availability features like cluster replacement, cross-data center support, disaster
recovery etc.
4. About Us
BloomReach
BloomReach has developed a personalized discovery platform that features applications that analyze
big data to makes our customers’ digital content more discoverable, relevant and profitable.
Myself
I work on search platform scaling for BloomReach’s big data. My relevant experience and background
includes scaling real-time services for latency sensitive applications and building performance and search-
quality metrics infrastructure for personalization platforms.
6. BloomReach’s Applications
Organic
Search
Contentunderstanding
What it does
Content optimization,
management and measurement
Benefit
Enhanced discoverability and
customer acquisition in organic search
What it does
Personalized onsite search and
navigation across devices
Benefit
Relevant and consistent onsite
experiences for new and known users
What it does
Merchandising tool that understa
nds products and identifies oppo
rtunities
Benefit
Prioritize and optimize
online merchandising
SNAP
Compass
7. Agenda
• BloomReach search use cases and architecture
• Old architecture and issues
• Scaling challenges
• Elastic SolrCloud architecture and benefits
• Lessons learned
8. BloomReach Search Use Cases
1. Front-end (serving) queries – Uptime and Latency sensitive
2. Batch search pipelines – Throughput sensitive
3. Time bound indexing requirements – Customer Specific
4. Time bound Solr config updates
9. BloomReach Search Architecture
Solr
Cluster
Zookeeper Ensemble Map Reduce
Pipelines (Reads)
Indexing Pipelines
Pipeline 1
Pipeline 2
Pipeline n
Indexing 1
Indexing 2
Indexing n
Heavy Load
Moderate Load
Light Load
Legend
Public API
Search Traffic
Search Traffic
10. Throughput Issues…
Solr
Cluster
Zookeeper Ensemble
Pipeline 1
Pipeline 2
Pipeline n
Indexing 1
Indexing 2
Indexing n
Public API
Search Traffic
● Heterogeneous read
workload
● Same collection - different
pipelines, different query
patterns, different schedule
● Cache tuning is virtually
impossible
● Larger pipeline starving the
small ones
● Machine utilization
determines throughput and
stability of a pipeline at any
point
● No isolation among jobs
11. Stability and Uptime Issues…
Solr
Cluster
Zookeeper Ensemble
Pipeline 1
Pipeline 2
Pipeline n
Indexing 1
Indexing 2
Indexing n
Public API
Search Traffic
● Bad clients – bring down
the cluster/degrade
performance
● Bad queries (with heavy
load) – render nodes
unresponsive
● Garbage collection issues
● ZK stability issues (as we
scale collections)
● CPU /Load Issues
● Higher number of
concurrent pipelines,
higher number of issues
12. Indexing Issues…
Solr
Cluster
Zookeeper Ensemble
Pipeline 1
Pipeline 2
Pipeline n
Indexing 1
Indexing 2
Indexing n
Public API
Search Traffic
● Commit frequencies vary
with indexer types
● Indexer run during another
pipeline – performance
● Indexer client leaks
● Too many stored fields
● Non-batch updates
13. Rethinking…
• Shared cluster for pipelines does not scale.
• Guaranteeing an uptime of 99.99+ - non trivial
• Every job runs great in isolation. When you put them together, they fail.
• Running index-heavy load and read-heavy load - cluster performance issues.
• Any direct access to production cluster – cluster stability (client leaks, bad queries etc.).
What if every pipeline had its own cluster?
14. Solr Compute Cloud (SC2)
• Elastic Infrastructure – Provision Solr Clusters on demand, on-the-fly.
• Create, Use, Terminate Model - Create a temporary cluster with necessary data, use it and throw it away.
• Technologies behind SC2 (built in House)
Cluster Management API - Dynamic cluster provisioning and resource allocation.
Solr HAFT – High availability and data management library for SolrCloud.
• Isolation - Pipelines get their own cluster. One cannot disrupt another.
• Dynamic Scaling – Every pipeline can state its own replication requirements.
• Production Safeguard - No direct access. Safeguards from bad clients/access patterns.
• Cost Saving – Provision for the average; withstand peak with elastic growth.
15. Solr Compute Cloud
Solr
Cluster
Zookeeper Ensemble
Pipeline 1
Solr
Compute
Cloud
API
Solr Cluster
Collection A
Replicas: 6
1. Read pipeline requests
collection and desired
replicas from SC2 API.
2. SC2 API provisions
cluster dynamically with
needed setup (and
streams Solr data).
3. SC2 calls HAFT service to
replicate data from
production to provisioned
cluster.
4. Pipeline uses this cluster
to run job.
1
4
Request: {Collection: A, Replica: 6}
2
Solr
HAFT
Service
3
3
Read
Replicate
16. Solr Compute Cloud…
Solr
Cluster
Zookeeper Ensemble
Pipeline 1
Solr
Compute
Cloud
API
Solr Cluster
Collection A
Replicas: 6
1. Pipeline finishes running
the job.
2. Pipeline calls SC2 API to
terminate the cluster.
3. SC2 terminates the
cluster.
2
Terminate: {Cluster}
3
Solr
HAFT
Service
1
17. Solr Compute Cloud – Read Pipeline View
Zookeeper Ensemble
Pipeline 1
Solr
Compute
Cloud
API
Solr Cluster
Collection A
Replicas: 6
Request: {Collection: A, Replica: 6}
Pipeline 2
Solr Cluster
Collection B
Replicas: 2
Request: {Collection: B, Replica: 2}
Pipeline n
Solr Cluster
Collection C
Replicas: 1
Request: {Collection: C, Replica: 1}
Solr
HAFT
Service
Production
Solr Cluster
18. Solr Compute Cloud – Indexing
Production
Solr Cluster
Zookeeper Ensemble
Indexing
Solr
Compute
Cloud
API
Solr Cluster
Collection A
Replicas: 6
1. Read pipeline requests
collection and desired
replicas from SC2 API.
2. SC2 API provisions
cluster dynamically with
needed setup (and
streams Solr data).
1. Indexer uses this cluster
to index the data.
2. Indexer calls HAFT
service to replicate the
index from dynamic
cluster to production.
3. HAFT service reads data
from dynamic cluster and
replicates to production
Solr.
1
3
Request: {Collection: A, Replica: 2}
2
Replicate
Solr HAFT Service
4
5
Read
19. Solr Compute Cloud – Global View
Zookeeper Ensemble
Solr
Compute
Cloud
API
Solr HAFT Service
Production
Solr Cluster
Indexing Pipelines 1
Elastic Clusters
Read Pipelines 1
Read Pipelines n
Indexing Pipelines n
Provision: {Cluster}
Terminate: {Cluster}
Replicate Index
Replicate Index
Run Job
20. Solr Compute Cloud API
1. API to provision clusters on demand.
2. Dynamic cluster and resource allocation (includes cost optimization)
3. Track request state, cluster performance and cost.
4. Terminate long-running, runaway clusters.
21. Solr HAFT Service
1. High availability and fault tolerance
2. Home-grown technology
3. Open Source - (Work in progress)
4. Features
• One push disaster recovery
• High availability operations
• Replace node
• Add replicas
• Repair collection
• Collection versioning
• Cluster backup operations
• Dynamic replica creation
• Cluster clone
• Cluster swap
• Cluster state reconstruction
22. Solr HAFT Service
Clone Alias
Clone Collections
Custom Commit Node Replacement
Node Repair
Clone Cluster
Collection Versioning
Black Box Recording
Lucene Segment
Optimize
Index Management Actions
High Availability Actions
Cluster Backup Operations
Solr Metadata
Zookeeper
Metadata
Verification Monitoring
Solr HAFT Service – Functional View
Dynamic Replica
Creation
Cluster Clone
Cluster Swap
Cluster State
Reconstruction
23. Disaster Recovery in New Architecture
Old
Production
Solr
Cluster
Zookeeper Ensemble
New
Solr
Cluster
Zookeeper Ensemble
Solr HAFT Service
Push
Button
Recovery
Brave Soul on Pager Duty
1
2
DNS
3
1. Guy on Pager clicks the
recovery button
2. Solr HAFT Service
triggers
Cluster Setup
State Reconstruction
Cluster Clone
Cluster Swap
3. Production DNS – New
Cluster
24. SC2 vs Non-SC2 (Stability Features)
Property Non-SC2 SC2
Linear Scalability for Heterogeneous
Workload
Pipeline Level Isolation
Dynamic Collection Scaling
Prevention from Bad Clients
Pipeline Specific Performance
No Direct Access to Production Cluster
Can Sleep at night?
25. SC2 vs Non-SC2 (Availability Features)
Property Non-SC2 SC2
Cross Data-Center Support
Cluster Cloning
Collection Versioning
One-Push Disaster Recovery
Repair API for Nodes/Collections
Node Replacement
26. Lessons Learned
1. Solr is a search platform. Do not use it as a database (for scans and lookups).
Evaluate your stored fields.
2. Understand access patterns, QPS and queries in detail. Be careful when tuning
caches.
3. Have access control for large-scale jobs that directly talk to your cluster. (Internal
DDOS attacks are hard to track.)
4. Instrument every piece of infrastructure and collect metrics.
5. Build automated disaster recovery (You will need it. )
GM. Thanks for making it to the session
I am Nitin…
The Talk is about SC2 which was built inside bloomreach to scale to our search use cases.
If things go as planned, we should have a few mins for questions. If not I will be more than happy to talk offline. Please post your questions in the activity feed
Typical Search Platforms have low latency requirements as we scale # of collections and # of documents
Performance, Availability, Scalability and Stability
Job Level Isolation with latency and SLA guarantees
Diaster Recovery Platform
This presentation will describe an innovative implementation of scaling Solr in an elastic fashion..
Bloomreach is a big data based marketing platform
We offer products that make our customer’s digital content more relevant
What kind of search use cases does bloomreach have?
A year worth of work and home grown technologies. Staying at a high level
Glad to go over offline.
Latency Sensitive Frontend Applications (Incoming customer queries through api)
Huge Batch (Map reduce based jobs) jobs constant analyzing and figuring out the relevant content
Time Sensitive index reads and writes
Picture is worth a 1000 words
Search Traffic – Different set of queries (utilizing all sorts of features faceting, sorting , custom ranking)
ETL pipelines, Product based pipelines
Pipelines Running at different schedules, different QPS and different throughput requirements.
Indexer (Partial and Full indexing)
Lets go over what kind of issues we encountered with this setup
Red Circle indicates the issues.
Multiple Products, Mulitple read pipleines. They run at different schedules. One can starve the other. One can screw up the latencies of the other.
The setup is very static
Un even load. Mem/CPU Usage varies and so does the machine utilization
OOM
PermGen issues (#Collections)
Old Gen Issues
High CPU
Bad Queries
Even Sharding
Unused features
Highlighting
Random jars
Bad clients – bring down the cluster/degrade performance (Any misbehaving job shoud Fail instead of bringing cluster down?)
Bad queries (with heavy load) – render nodes unresponsive
Perm Gen , Old Gen , Young Gen (Parallel GC vs CMS)
ZK txn log and data dirs in different drives.
We have developed multiple custom components. They might cause exceptions during querytime
Auto commits did help to a certain extent but not too much
No job level isolation
Client Leaks
No Rate limiting
Summary of issues
Ok how do we achieve that?
Introducing SC2
Spend More time…
Linearly Scalable with Dynamic Scaling
No direct access to production. Any misbehaving job “Fails” instead of bringing cluster down?
Cost Saving:
Instead of from , you replicate to production
Access to production is only through replication and not through any other means