Speaker: Peter Dachnowicz - Sr. Technical Account Manager, AWS Enterprise Support

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Using Search with a Database
Peter Dachnowicz
pdach@amazon.com
Sr Technical Account Manager

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elasticsearch Service
Fully-managed service that is easy to deploy, manage, and scale
+
Fully managed

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Amazon Elasticsearch Service
Supports Open-Source
APIs and Tools
Drop-in replacement with no need
to learn new APIs or skills
Easy to Use
Deploy a production-ready
Elasticsearch cluster in minutes
Scalable
Resize your cluster with a few
clicks or a single API call
Secure
Deploy into your VPC and
restrict access using security
groups and IAM policies
Highly Available
Replicate across Availability Zones,
with monitoring and
automated self-healing
Tightly Integrated with
Other AWS Services
Seamless data ingestion, security,
auditing and orchestration

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service architecture
AWS SDK
AWS CLI
AWS
CloudFormation
Elasticsearch
data nodes
Elasticsearch
master nodes
Elastic Load
Balancing
IAM
Amazon
CloudWatch
AWS
CloudTrail
Amazon Elasticsearch Service domain

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elasticsearch Service VPC Support
• Private networking between
your VPC and Amazon
Elasticsearch Service
• Traffic does not traverse
the public internet
• Use IAM policies and security
groups for authentication
and access control
VPC subnet
security group
VPC subnet
Availability
Zone A
IAM
Amazon Elasticsearch Service
Data Master
Availability
Zone B
security group
IAM
Data
Master

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Lambda architectures
S3 Events
Amazon S3 AWS Lambda
Function
Amazon Elasticsearch Service
Amazon
Kinesis
Files
Amazon
DynamoDB
Table
DynamoDB
streams
AWS Lambda
Function
Data
Producers
AWS Lambda
Function

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Firehose delivery architecture
Amazon
Elasticsearch Service
Transformed
records
Data transformation
function
• For public access domains
• Easily transform data
• Serverless with built-in batching,
index rollover, error handling
Data source
Source records
Firehose
delivery stream
Source records
Delivery failure
Transformation failure
Source records

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Encrypt your data
• Encrypted data at rest on
Amazon ES instances
• Both EBS and ephemeral store
• Encrypted automatic snapshots

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The front end

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application search

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Each document
has a set of fields
Key Idea

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elasticsearch works with
structured JSON containing
fields and values

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elasticsearch creates an index for each field
Doc
Fields Analysis
Field indices
Term 1
Term 2
Term 3
Term 4
Term 5
Term 6
Term 7
Name Value
Name Value
Name Value
Name Value
Name Value
Name Value
1,
4,
8,
12,
30,
42,
58,
100
...
Posting lists

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data is stored in indexes, distributed across shards
Index
Shards Instances
ID
Field: value
Field: value
Field: value
Field: value
Amazon Elasticsearch Service domain • Shards are
primary or replica
• Primary shard count
can’t be changed
• Elasticsearch
distributes shards to
instances elastically
• Primary and replica
are distributed to
different instances

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How many instances?
Index size is approximately source size
if you are deploying an index replica
Instance count based on storage requirements
• Either local storage or up to 1.5 TB of Amazon
Elastic Block Store (EBS) per instance

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Which instance type?
Instance Max storage* Workload
T2 3.5TB You want to do dev/QA
M3, M4 150TB Your data and queries are “average”
R3, R4 150TB
You have higher request volumes, larger documents,
or are using aggregations heavily
C4 150TB You need to support high concurrency
I2, I3 1.5PB You have XL storage needs

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elasticsearch Service Customers
Software & Internet Education Technology BioTech and Pharma
Media and EntertainmentFinancial Services Social Media
Telecommunications Travel & Transportation Real Estate
Logistics & Operations Publishing Other