One of the benefits of having a data lake is that the same data can be consumed by multi-tenant groups—an efficient way to share a persistent Amazon EMR cluster. The same business data can be safely used for many different analytics and data processing needs. In this builders session, we discuss steps to make an Amazon EMR cluster multi-tenant for analytics, best practices for a multi-tenant cluster, and solving common challenges. We also address security and governance aspects of a multi-tenant Amazon EMR cluster.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
One Data Lake, Many Uses: Enable Multi-
Tenant Analytics with Amazon EMR
Bruno Faria
A N T 3 4 4

2. Users
Marketing
Finance
Data
science
Amazon S3 | Amazon Glacier | AWS Glue
Data lake
on AWS
Multi-tenancy motivation
Engineering

3. Multi-tenant analytics pipeline
COLLECT STORE
PROCESS/
ANALYZE
CONSUME
Authentication | Authorization | Tenant isolation | Monitoring | Metering | . . .
Tenant 1
Tenant 2
Tenant 3

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User isolation
• Authentication
(User
identification)
Data isolation
• Authorization
• Access
rights/privileges
to resources
• Coarse-grained
• Fine-grained
Resource
isolation
• Queues

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Terminology
• Silo mode
• Tenant analytics (data + processing) is fully isolated from other tenants
• Constructs logically “unique”
• Shared mode
• Tenants share all analytic resources

7. Silo scenario . . .
Wrangle/manipulate data
Explore/analyze data
Run machine learning models
Processed data
Query results
Deploy
Data engineer
Data analyst
Data scientist
EMR cluster
EMR cluster
EMR cluster
Data engineer bucket
Data analyst bucket
Data scientist bucket

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-tenancy in Amazon EMR
• Silo mode
• Each tenant gets their own Amazon EMR cluster with specific tools
for processing/analyzing
• Data stored in tenant’s Amazon Simple Storage Service (Amazon
S3) bucket or HDFS on the cluster
• Hive metastore on the cluster or externally on Amazon Relational
Database Service (Amazon RDS)
• Pros
• Complete isolation
• Easy to measure usage and resources
• Can be cost effective when using Spot instances
• Cons
• Cannot share data across clusters (especially when using HDFS)
• Can be expensive

9. Shared scenario . . . /Data engineer bucket
/Data scientist bucket
/Data analyst bucket
Explore/analyze data
Query results / processed data
Data engineer
Data analyst
Data scientist
EMR cluster
Edge node for Amazon EMR
Edge node for Amazon EMR
Edge node for Amazon EMR

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LDAP
HiveServer2
Presto coordinator
Spark Thrift server
Hue server
Zeppelin server
AWS credentials
EMR Step (EMR API)
Amazon Elastic Compute Cloud
(Amazon EC2) key pair
SSH as “hadoop”
Authentication

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Network authentication protocol
• Eliminates the need for
transmission of passwords
across network
• Removes potential threat of an
attacker sniffing the network
Authentication using Kerberos

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Storage-based
• EMRFS/Amazon S3
• HDFS
• HiveServer2 and Presto (SQL-based)
• HBase
• Fine-grained access control by cluster tag (AWS
Identity and Access Management (IAM))
• Apache Ranger on Amazon EC2 instance (using
AWS CloudFormation)
Authorization

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
YARN
• Queues
• Share cluster among multiple tenants
• Applications assigned to queues
• ‘root’ – Parent of all queues
• Queues correspond to departments, users, or priorities

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Root
Marketing
Engineering
Data
science
Sales
YARN queue example
{
"Classification": "capacity-scheduler",
"Properties": {
"yarn.scheduler.capacity.maximum-am-resource-percent": "0.6",
"yarn.scheduler.capacity.resource-calculator":
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator",
"yarn.scheduler.capacity.root.queues": "default,engineering,datascience,marketing",
"yarn.scheduler.capacity.root.default.capacity": "10",
"yarn.scheduler.capacity.root.default.user-limit-factor": "2",
"yarn.scheduler.capacity.root.default.maximum-capacity": "40",
"yarn.scheduler.capacity.root.engineering.capacity": "45",
"yarn.scheduler.capacity.root.datascience.capacity": "30",
"yarn.scheduler.capacity.root.marketing.capacity": "15",
"yarn.scheduler.capacity.root.engineering.user-limit-factor": "2",
"yarn.scheduler.capacity.root.datascience.user-limit-factor": "2",
"yarn.scheduler.capacity.root.marketing.user-limit-factor": "2",
"yarn.scheduler.capacity.root.engineering.maximum-capacity": "75",
"yarn.scheduler.capacity.root.datascience.maximum-capacity": "55",
"yarn.scheduler.capacity.root.marketing.maximum-capacity": "50",
"yarn.scheduler.capacity.root.engineering.state": "RUNNING",
"yarn.scheduler.capacity.root.datascience.state": "RUNNING",
"yarn.scheduler.capacity.root.marketing.state": "RUNNING",
"yarn.scheduler.capacity.root.engineering.acl_submit_applications": "*",
"yarn.scheduler.capacity.root.datascience.acl_submit_applications": "*",
"yarn.scheduler.capacity.root.marketing.acl_submit_applications": "*"
}
},
{
"Classification": "yarn-site",
"Properties": {
"yarn.acl.enable": "true",
"yarn.resourcemanager.scheduler.class":
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler"
}
}

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
YARN scheduler
• Nested queues
• Queue weights –
Control fair share of
apps in the queue
• Manage queue access
through ACLs
root
Engineering
Dev
Data
engineering
Marketing
Advertising Analytics

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.