In this session you will learn the best practices and various use cases for performing data analytics at scale with Amazon EMR. We will introduce you to Amazon EMR design patterns and share how to use big data analytics to provide business insights. Speakers: Dr Haydn Read, Head of Infrastructure Programmes, Auckland City Council & Brad Staszcuk, Solutions Architect, AWS

1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Brad Staszcuk
Solution Architect – Amazon Web Services
Dr Haydn Read
Head of Infrastructure Programmes - Auckland Council
Accelerate Analytics At Scale With
Amazon EMR

2. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
What Is Amazon EMR?
Low cost
Pay per-second
Open-source variety
Latest versions of software
Secure
Easy to enable options
Flexible
Full customisation and control
Easy to use
Launch a cluster in minutes
Managed
Spend less time monitoring
Flink

3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Open-Source Applications
Amazon EMR
service

4. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Use The AWS Glue Data Catalog
• Support for Spark,
Hive and Presto
• Auto-generate
schema and
partitions
• Managed table
updates

5. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Hbase For Random Access At Massive Scale
HDFS
(data node)
Local Disk
HBase region
server
YARN node
manager

6. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Tips To Lower Your Costs

7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Transient Or Long-Running Clusters
• Shut your clusters down when you don’t need them
• Use Amazon Linux AMI with preinstalled customisations for faster startup
• Use Auto Scaling to minimise costs for long-running clusters

8. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon EC2 Spot And Instance Fleets
• EMR will select optimal EC2 AZ
• Provision across instance types
• Switch to on-demand

9. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Use Auto Scaling
Scaling options
Threshold
CloudWatch or custom metric

10. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Auto Scaling
• EMR scales-in at YARN task completion
• Selectively removes nodes with no running tasks
• yarn.resourcemanager.decommissioning.timeout
• Default timeout is one hour
• Spark scale-in contributions
• Spark specific blacklisting of tasks
• Unregistering cached data and shuffle blocks
• Advanced error handling

11. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Tips To Secure Your Cluster

12. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Encryption
• Spark
• Tez
• MapReduce
• Presto
• HBase
• Hive
• Pig

13. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Authentication
LDAP
HiveServer2
Presto Coordinator
Spark Thrift Server
Hue Server
Zeppelin Server
AWS credentials
Amazon EMR Step (EMR API)
EC2 key pair
SSH as “hadoop”

14. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
New – Authentication With Kerberos
Microsoft
Active Directory
KDC
Users
YARN RMDoAs
Service principals for
all cluster nodes
Master Node

15. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Authorisation
• Storage-based
• EMRFS/S3
• HDFS
• HiveServer2 and Presto (SQL-based)
• HBase
• YARN queues
• Fine-grained access control by cluster tag (IAM)
• Apache Ranger on edge node (using AWS CloudFormation)

16. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Configure AWS IAM Roles For EMRFS Requests
Context
User: aduser
Group: analyst
IAM role: analytics_prod
Can map IAM roles to user, group, or S3 prefix
Context
User: aduser2
Group: dev
IAM role: analytics_dev

17. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Introducing…
Dr Haydn Read, Auckland Council

18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Thank You
craigar@amazon.com
https://aws.amazon.com/emr/