by Darin Briskman, Technical Evangelist, AWS Elasticsearch is the most popular open-source search and analytics engine - it's easy to use, but not always easy to configure an manage. Learn about Amazon's fully managed service that provides easier deployment, operation, and scale for Elasticsearch. Level: 200

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introduction to
Amazon Elasticsearch Service
Darin Briskman
AWS Technical Evangelist
briskman@amazon.com

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-time, distributed, search & analytics
engine:
• Built on top of Apache Lucene
• Schema free
• Developer friendly RESTful API

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is the ELK stack?
Logstash – simple tool for transforming and streaming data into ES
Elasticsearch – distributed search engine based on Lucene
Kibana – Easy to use tool for visualization of data in Elasticsearch

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elasticsearch for trends and patterns
Aggregations:
• Buckets (like GROUP BY in SQL)
• Metrics (like COUNT, SUM, MAX etc)

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elasticsearch for full-text search
• Core search is provided through Apache Lucene
• Elasticsearch handles JSON structure, including nesting
• Aggregations to provide faceting++
• Supports core search features
• Suggestions
• Highlights
• Boolean expressions
• Adjustable ranking
• Fuzzy search and more

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operating Elasticsearch is time-consuming
“Elasticsearch allows us to easily and quickly build bleeding edge big data
and analytics applications using the ELK stack. By offering direct access
to the Elasticsearch API while offloading administrative tasks, Amazon
Elasticsearch Service gives us the manageability, flexibility and control we
need ”
Sean Curtis,
SVP Engineering at Major League
Baseball Advanced Engineering

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Leading enterprises trust Amazon Elasticsearch
Service for their search and analytics applications
Media &
Entertainment
Online
Services
Technology Other

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Adobe Developer Platform (Adobe I/O)
P R O B L E M
• Cost effective monitor
for XL amount of log
data
• Over 200,000 API calls
per second at peak -
destinations, response
times, bandwidth
• Integrate seamlessly
with other components
of AWS eco-system.
S O L U T I O N
• Log data is routed with
Amazon Kinesis to
Amazon Elasticsearch
Service, then displayed
using AES Kibana
• Adobe team can easily
see traffic patterns and
error rates, quickly
identifying anomalies and
potential challenges
B E N E F I T S
• Management and
operational simplicity
• Flexibility to try out
different cluster config
during dev and test
Amazon
Kinesis
Streams
Spark Streaming
Amazon
Elasticsearch
Service
Data
Sources
1

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
McGraw Hill Education
P R O B L E M
• Supporting a wide catalog
across multiple services in
multiple jurisdictions
• Over 100 million learning
events each month
• Tests, quizzes, learning
modules begun / completed
/ abandoned
S O L U T I O N
• Search and analyze test
results, student/teacher
interaction, teacher
effectiveness, student
progress
• Analytics of applications
and infrastructure are now
integrated to understand
operations in real time
B E N E F I T S
• Confidence to scale
throughout the school year.
From 0 to 32TB in 9 months
• Focus on their business, not
their infrastructure

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elasticsearch fundamentals

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building blocks - documents
• A document is the core entity of search – file,
database, log line, data structure etc., represented
with JSON
• Documents are referenced by an index
• Documents contain field-value pairs
• Documents are distributed across shards and
replicas
• You choose which fields in the document to index for
search, which to store, and with each query, which to
search

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building blocks - shards
• A shard is an instance of Lucene, using compute, storage and other
system resources
• Elasticsearch deploys shards elastically, and dynamically to the
instances in the cluster, providing the means for horizontal scale
• An index always has at least one shard and may have any number
• Shards are primary (exactly one) or replica
(dynamic)
• Elasticsearch routes requests to the applicable
shards

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building blocks - replicas
• A replica is a copy of a shard
• You can replicate a single shard multiple times for
availability or scale
• Replicas are never allocated on the same instance as
the original shard
• Replicas allow you to scale throughput as searches
can leverage the replicas in parallel
• The number of replicas can be changed
dynamically

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building blocks - index
• An index is a collection of documents that has
fields that can be searched
• You access an index through any participating
instance via a restful interface
• The index spans across instances by
leveraging shards and replicas

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building blocks - instance
• For AES, an instance is an Elasticsearch
node installed on EC2
• An instance can work with other AES
instances to facilitate distribution of work
or it can stand by itself
• In a cluster, multiple instances allow you
to scale the compute and storage that
holds your indexes
• Instances hold shards and replicas that
are referenced by the index

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Index and mappings/types
Index:
Product
Type:
cellphone
ID, make, color, etc.
ID, make, color, etc.
ID, make, color, etc.
ID, make, color, etc.
ID, make, color, etc.
Type:
review
ID, author, date, title, text
ID, author, date, title, text
ID, author, date, title, text
ID, author, date, title, text
ID, author, date, title, text

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Documents are distributed to shards randomly
• Document IDs are hashed
• You can control this behavior
Shard 1
Shard 2
Shard 3
Primary
shards
Index
Shard 1
Shard 2
Shard 3
Replica
shards
_bulk
API

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elasticsearch Service

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Easy
Elasticsearch
cluster creation
and scaling

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES domain overview
Amazon Route
53
Elastic Load
Balancing
IAM
CloudWatch
Elasticsearch API
CloudTrail

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Route
53
Elastic Load
Balancing
IAM
CloudWatch
Elasticsearch API
CloudTrail
Amazon ES domain overview
Instances under management

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IAM
CloudWatchCloudTrail
Elasticsearch API
Amazon Route
53
Elastic Load
Balancing
Amazon ES domain overview
Single endpoint, REST API

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatchCloudTrail
Elasticsearch API
Amazon Route
53
Elastic Load
Balancing
IAM
Amazon ES domain overview
IAM integration

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elasticsearch API
Amazon Route
53
Elastic Load
Balancing
IAM
CloudWatchCloudTrail
Amazon ES domain overview
CloudWatch/CloudTrail for monitoring

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI commands
add-tags
create-elasticsearch-domain
delete-elasticsearch-domain
describe-elasticsearch-domain
describe-elasticsearch-domain-
config
describe-elasticsearch-domains
list-domain-names
list-tags
remove-tags
update-elasticsearch-domain-config
aws es create-elasticsearch-domain --domain-name my-domain
--elasticsearch-cluster-config
InstanceType=m4.xlarge.elasticsearch,InstanceCount=4
--ebs-options
EBSEnabled=true,VolumeType=gp2,VolumeSize=512

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster reconfiguration
• Console or single API call
• You can change any parameters of the domain: Instance
types and counts, storage options, dedicated master
configuration, zone-awareness, etc.
• Amazon ES non-disruptively makes the changes
requested
• Expands the cluster with new instances
• Elasticsearch replicates data
• Old instances are dropped

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dedicated
masters
improve
stability

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster with no dedicated masters
Amazon ES cluster
1
3
3
1
Instance 1,
Master
2
1
1
2
Instance 2
3
2
2
3
Instance 3

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster with dedicated masters
Amazon ES cluster
1
3
3
1
Instance 1
2
1
1
2
Instance 2
3
2
2
3
Instance 3Dedicated master instances
Data instances: queries and updates
Adding dedicated
masters improves
stability by removing
the master function
from data instances
Amazon ES provides
1-click deployment

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Master node recommendations
Number of data nodes Master node instance type
< 10 m3.medium+
< 20 m4.large+
<= 50 c4.xlarge+
50-100 c4.2xlarge+
Always use an odd number of masters, >= 3

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Zone awareness
for high
availability

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster with zone awareness
Amazon ES cluster
Availability Zone 1 Availability Zone 2
1
3
Instance 1
2
1 2
Instance 2
3
2
1
Instance 3
2
1
Instance 4
3
3
Cluster instances are split evenly between two zones
Primary and replica shards go to different zones

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scale for your
workload

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Instance type recommendations
Instance Workload
T2 Entry point. Dev and test. OK for dedicated masters.
M3, M4 Equal read and write volumes.
R3, R4 Read-heavy or workloads with high memory demands (e.g.,
aggregations).
C4 High concurrency/indexing workloads
I2 Up to 1.6 TB of SSD instance storage.

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secure access
to your domain

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster Security
AWS manages
security of the
cluster
You manage
access to the
domain via
policies

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security through IAM A&AC
• IP-based policies limit access to CIDR blocks for
anonymous requests
• Principal-based policies limit access to particular IAM
users or roles, requiring AWS SigV4 signing
• Policies
• Can control HTTP method allowed to create differential
access for e.g. queries and updates
• Can specify resources at the index level

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example IAM Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:123456789012:user/susan"
},
"Action": [ "es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost",
"es:CreateElasticsearchDomain",
"es:ListDomainNames" ],
"Resource":
"arn:aws:es:us-east-1:###:domain/logs-domain/<index>/*"
} ] }

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Security Responsibilities
• Creation of a service VPC that allows limited access to
the cluster through a configurable access policy
• Application of security patches on the instances
• DDOS protection for the DNS name associated with the
domain via Route53
• Facades the transport protocol with an ELB (HTTP:80)
• Hides ports 9200 and 9300
• Built on top of AWS secure networking

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security Patches
• There is no defined maintenance window as patches
are applied using blue/green methodologies
• Critical patches applied immediately
• Routine patching typically applied during customer
activities such as cluster resizing or policy changes

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load data

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Logstash
REST
CWL Agent
EC2 Instances
Amazon
Kinesis
Amazon
RDS
Amazon
DynamoDB
Amazon
SQS
Queue
Logstash
Cluster
Amazon
Elasticsearch
Service
Amazon
CloudWatch
AWS
Lambda
AWS
CloudTrail
Access Logs
Amazon
VPC Flow
Logs
Amazon S3
bucket
AWS IoT
Amazon Kinesis
Firehose
Integration with the AWS
ecosystem

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitor and audit
CloudWatch CloudTrail Elasticsearch APIs

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Durability
• Read Replicas + Zone Awareness
• Automatic Daily snapshots
• Manual Index Snapshots

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Built-in Kibana

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application overview
Logstash indexer
Amazon
Elasticsearch
Service
Application instances/
Logstash forwarders

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pay for what you
use

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pay for compute and storage you use
With Amazon Elasticsearch Service, you pay only for the
compute and storage resources you use. AWS Free Tier for
qualifying customers.

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Easy to Use
Deploy a production-ready Elasticsearch
cluster in minutes
Simplifies time-consuming management
tasks such as software patching, failure
recovery, backups, and monitoring
Open
Get direct access to the Elasticsearch
open-source API
Fully compatible with the open source
Elasticsearch API, for all code and
applications
Secure
Secure Elasticsearch clusters with AWS
Identity and Access Management (IAM)
policies with fine-grained access control
access for users and endpoints
Automatically applies security patches
without disruption, keeping Elasticsearch
environments secure
Available
Provides high availability using Zone
Awareness, which replicates data between
two Availability Zones
Monitors the health of clusters and
automatically replaces failed instances,
without service disruption
AWS Integrated
Integrates with Amazon Kinesis Firehose,
AWS IOT, and Amazon CloudWatch Logs for
seamless data ingestion
AWS CloudTrail for auditing, AWS Identity
and Access Management (IAM) for
security, and AWS CloudFormation for
cloud orchestration
Scalable
Scale clusters from a single instance up to
100 instances
Configure clusters to meet performance
requirements by selecting from a range of
instance types and storage options
including SSD-powered EBS volumes
Amazon Elasticsearch Service Benefits

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Takeaways
1. Elasticsearch is a tool for full-text search, analysis, and
visualization of time series data that helps you get the
most out of your growing data set
2. Amazon Elasticsearch Service makes it easy to deploy
and manage an Elasticsearch cluster in the AWS cloud
3. Amazon Elasticsearch Service is a drop-in replacement
for your existing Elasticsearch cluster

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!