1. Copyright © 2014 Intridea Inc. All rights reserved.
Elasticsearch with Rails
July 10, 2014
Tom Zeng
Director of Engineering
tom@intridea.com
@tomzeng
www.linkedin.com/in/tomzeng

2. What is Elasticsearch
!
Elasticsearch is a “flexible and powerful open source, distributed real-time
search and analytics engine for the cloud”
More than just full text search, it has powerful analytics capability, and can
be used as a NoSQL data store
Easy to setup, easy to use, easy to scale, easy to maintain
Suitable for projects of any size (large and small, cloud or non-cloud) that
need full text search and/or analytics, it’s our preferred search engine for
Rails apps

3. Elasticsearch Quick Live Demo
Use curl to add some data (local Elasticsearch instance at port 9200)
!
curl -X DELETE "http://localhost:9200/todos" (clean up the index and start from scratch)
!
curl -X POST "http://localhost:9200/todos/task/1" -d '{"title" : "Learn Elasticsearch",
"due_date" : "20140710T00:00:00", "done" : true, "tags" : ["seach","backend"]}'
!
curl -X POST "http://localhost:9200/todos/task/2" -d '{"title" : "Learn D3 and Backbone",
"due_date" : "20140720T00:00:00", "done" : true, "tags" :
["frontend","javascript","visualization"]}'
!
curl -X POST "http://localhost:9200/todos/task/3" -d '{"title" : "Learn Rails 4",
"due_date" : "20140830T00:00:00", "done" : false, "tags" : ["backend","ruby","rails"]}'
!
curl -X POST "http://localhost:9200/todos/task/4" -d '{"title" : "Learn Backbone
Marionette", "due_date" : "20140715T00:00:00", "done" : true, "tags" :
["frontend","javascript"]}'
!

4. Elasticsearch Quick Live Demo
Use curl to query data
curl http://localhost:9200/todos/task/1 (use as K/V store)
curl http://localhost:9200/todos/_search?pretty&q=done:false
curl http://localhost:9200/todos/_search?pretty&q=tags:backend
curl http://localhost:9200/todos/_search?pretty&q=title:back*
curl -X POST "http://localhost:9200/todos/task/_search?pretty" -d '
{
query : {
range : { due_date : { from : "20140701", to : "20140715" } }
}
}'
!

5. Elasticsearch Quick Live Demo

6. What is Elasticsearch
http://www.elasticsearch.org

7. What is Elasticsearch
http://www.elasticsearch.org

8. What is Elasticsearch
http://www.elasticsearch.org

9. Who uses Elasticsearch
http://www.elasticsearch.org

10. Who uses Elasticsearch - Github
Sort
HighlightFacets
Filters
Fulltext Search
Pagination

11. Elasticsearch Key Concepts
Cluster – A cluster consists of one or more nodes which share the same cluster name. Each
cluster has a single master node which is chosen automatically by the cluster and which can be
replaced if the current master node fails.
Node – A node is a running instance of elasticsearch which belongs to a cluster. Multiple
nodes can be started on a single server for testing purposes, but usually you should have one
node per server.
At startup, a node will use unicast (or multicast, if specified) to discover an existing cluster with
the same cluster name and will try to join that cluster.
Index – An index is like a ‘database’ in a relational database. It has a mapping which
defines multiple types.
An index is a logical namespace which maps to one or more primary shards and can have zero
or more replica shards.
Type – A type is like a ‘table’ in a relational database. Each type has a list of fields that can be
specified for documents of that type. The mapping defines how each field in the document is
analyzed. http://www.elasticsearch.org/guide/reference/glossary

12. Elasticsearch Key Concepts
Document – A document is a JSON document which is stored in elasticsearch. It is like a
row in a table in a relational database. Each document is stored in an index and has a type and
an id.
A document is a JSON object (also known in other languages as a hash / hashmap / associative
array) which contains zero or more fields, or key-value pairs. The original JSON document that is
indexed will be stored in the _source field, which is returned by default when getting or
searching for a document.
Field – A document contains a list of fields, or key-value pairs. The value can be a simple
(scalar) value (eg a string, integer, date), or a nested structure like an array or an object. A field
is similar to a column in a table in a relational database.
The mapping for each field has a field ‘type’ (not to be confused with document type) which
indicates the type of data that can be stored in that field, eg integer, string, object. The mapping
also allows you to define (amongst other things) how the value for a field should be analyzed.
Mapping – A mapping is like a ‘schema definition’ in a relational database. Each index has
a mapping, which defines each type within the index, plus a number of index-wide settings. A
mapping can either be defined explicitly, or it will be generated automatically when a document
is indexed http://www.elasticsearch.org/guide/reference/glossary

13. Elasticsearch Key Concepts
Shard – A shard is a single Lucene instance. It is a low-level “worker” unit which is managed
automatically by elasticsearch. An index is a logical namespace which points to primary and
replica shards.
Elasticsearch distributes shards amongst all nodes in the cluster, and can move shards
automatically from one node to another in the case of node failure, or the addition of new
nodes.
Primary Shard – Each document is stored in a single primary shard. When you index a
document, it is indexed first on the primary shard, then on all replicas of the primary shard. By
default, an index has 5 primary shards. You can specify fewer or more primary shards to scale
the number of documents that your index can handle.
Replica Shard – Each primary shard can have zero or more replicas. A replica is a copy of
the primary shard, and has two purposes: 1) increase failover: a replica shard can be promoted
to a primary shard if the primary fails. 2) increase performance: get and search requests can be
handled by primary or replica shards.
!
!
http://www.elasticsearch.org/guide/reference/glossary

14. Elasticsearch Key Concepts
!
Elasticsearch SQL
!
Index => Database
Type => Table
Document => Row
Field => Column
Mapping => Schema
Shard => Partition
!
!

15. Elasticsearch Installation
OS X
brew install elasticsearch
Ubuntu
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb
sudo dpkg -i elasticsearch-1.1.1.deb
Centos
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.noarch.rpm
sudo yum install elasticsearch-1.1.1.noarch.rpm
!

16. Elasticsearch Status Check

17. Elasticsearch Cluster Status

18. Elasticsearch Monitoring
elasticsearch-head - https://github.com/mobz/elasticsearch-head
!
!
!
!
Marvel - http://www.elasticsearch.org/guide/en/marvel/current/#_marvel_8217_s_dashboards
Paramedic - https://github.com/karmi/elasticsearch-paramedic
Bigdesk - https://github.com/lukas-vlcek/bigdesk/
!

19. Elasticsearch APIs

20. Elasticsearch API Examples
Use curl to run the query and facet APIs
curl -X POST "http://localhost:9200/todos/_search?pretty=true" -d '
{
"query" : { "query_string" : {"query" : "Learn*"} },
"facets" : {
"tags" : { "terms" : {"field" : "tags"} }
}
}
'
Facets – todos tagged with keywords
javascript: 2
frontend: 2
backend: 2
visualization: 1
!
!

21. Elasticsearch Query DSL
http://www.elasticsearch.org

22. Elasticsearch Query DSL Examples
http://www.elasticsearch.org

23. Elasticsearch Query DSL Examples

24. Elasticsearch Query DSL Examples
http://www.elasticsearch.org

25. Elasticsearch Plugins and Rivers
!
Use plugins to extend Elasticsearch functionality
elasticsearch-head, paramedic, bigdesk are all plugins
!
Rivers are pluggable services that pull and index data into
Elasticsearch
Rivers are available for mongodb, couchdb, rabitmq, twitter, wikipedia,
mysql, and etc
!
!

26. Elasticsearch and Hadoop
Create an external Hive table using ES query q=china

27. Elasticsearch and Hadoop
External Hive table data – wiki articles that reference the word 'china'

28. Elasticsearch and Rails
Well supported with the following gems:
!
elasticsearch-rails https://github.com/elasticsearch/elasticsearch-rails
!
elasticsearch-ruby https://github.com/elasticsearch/elasticsearch-ruby
!
searchkick https://github.com/ankane/searchkick
!
tire (retire) https://github.com/karmi/retire
!
!

29. Elasticsearch and Rails/Ruby

30. Elasticsearch vs Solr
!
Feature Parity between Elasticsearch & Solr http://solr-vs-elasticsearch.com/
!
Elasticsearch is easier to use and maintain
!
Built from ground up for scale (for all features)
!
Solr - not all features are available in Solr Cloud
!
!
!
!

31. Advanced Features of Elasticsearch
!
Fuzzy and Proximity Search
!
Autocomplete (term, phrase, completion, and context suggesters)
!
Suggest API
!
Geospatial Search (point, bounding box, polygon)
!
Plugins to extend functionality
!
Scripting in JavaScript, Python, Groovy, and Java
!
!

32. Advanced Features of Elasticsearch
!
Aggregation (more dimensions than Facets)
!
Related Image Search using LIRE (search similar images based on criteria)
!
Percolator (index queries & match on data - useful for event alert, i.e. back in stock)
!
Re-scoring on query results
!
Polymorphic Search
!
!
!

33. Elasticsearch the ELK Stack
!
Combining the massively
popular Elasticsearch,
Logstash and Kibana
!
End-to-end stack that
delivers actionable
insights in real-time from
almost any type of
structured and
unstructured data source
!
!
!

34. Spree + Elasticsearch - you do e-commerce?
!
Elasticsearch compliments or replaces Spree’s built-in search (AR with the ransack gem)
!
Two existing gems spree_elasticsearch and spree_elastic
!
spree_elasticsearch - replace built-in search - i.e. Product.search, etc
spree_elastic - complement built-in search - i.e. Product.elasticsearch
!
Both using model Decorators to add the ES search capabilities
!
Can build on top of one of them, or use the elasticsearch_rails directly
!
!

35. Elasticsearch and Spree - spree_elastic gem

36. Elasticsearch and Spree - spree_elasticsearch gem

37. Elasticsearch Resources
!
http://www.elasticsearch.org/overview/
http://www.elasticsearch.org/guide/
https://github.com/elasticsearch/elasticsearch-hadoop
https://github.com/mobz/elasticsearch-head
http://railscasts.com/episodes/306-elasticsearch-part-1
http://railscasts.com/episodes/307-elasticsearch-part-2
!
!
!

38. ! 🌎 #
Copyright © 2014 Intridea Inc. All rights reserved.
BY WORKING REMOTELY
9,816 Hours Saved
ACROSS 4 COUNTRIES
31 Employees
FOUNDED & STARTED IN 2007
Washington D.C.
We Make.
Designers, developers and project managers; this is who we are.
Building, breaking and solving; this is what we do.

39. Gracias
Merci
ありがとう
Danke
谢谢
Thank You
Copyright © 2014 Intridea Inc. All rights reserved.