Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Amazon Elasticsearch Service Security Deep Dive - AWS Online Tech Talks

Learning Objectives:
- Understand how Amazon ES security works
- Learn how to use Amazon VPC endpoints with Amazon ES
- Take away best practices for security and access control for your Amazon ES domains

  • Sé el primero en comentar

Amazon Elasticsearch Service Security Deep Dive - AWS Online Tech Talks

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Elasticsearch Service Security Deep Dive November 9, 2017 Jon Handler AWS Principal Solutions Architect or @_searchgeek
  2. 2. Get started at Introduction • Amazon Elasticsearch Service supported public endpoints. Access control used IAM and/or IP-based policies • October 17, we added VPC support • You can still use IAM for fine-grained access, but VPC support enhances security and simplifies communication with Amazon ES
  3. 3. Get started at Data Analysis InsightsActions Evolution Your Busines s What is Elasticsearch all about? Let’s start with the data • Derive insights from high-volume, unstructured or semi- structured data • at scale • securely • and cost-effectively • without diverting focus from your mission Your data drives your business
  4. 4. Get started at Some uses for your data IT operations monitoring Security information and event management Application monitoring and intelligence IoT/TSDB monitoring and data analysis Search for your application Ad targeting
  5. 5. Get started at How Amazon ES helps Log Analytics Analyze un-structured and semi-structured logs generated by websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital marketing, operational intelligence, fraud detection, ad tech, gaming, and IoT. Full Text Search Provide a highly performant, rich search and navigation experience over a diverse set of documents with support for features including text matching, faceting, filtering, fuzzy search, auto complete, and highlighting. Distributed Search Engine Power search for your application with an easy to use, highly performant JSON document-oriented platform that can store and retrieve billions of documents, with integrated replication across Availability Zones. Real-Time Application Monitoring Capture activity logs across your customer-facing applications and websites by indexing data for analysis in near real-time (less than one second), visualize it, and perform statistical aggregations to identify root cause and fix issues. Click-Stream Analytics Deliver real-time metrics on digital content and enable authors and marketers to connect with their customers. Stream billions of small messages into Elasticsearch where you can aggregate, filter, and process the data to provide content performance dashboards.
  6. 6. Get started at Scalability AWS IntegrationSecurity & Monitoring Availability & Data Protection Ease of use & TCO Dev Platform & Tools • Fully-managed – automated failure management, software patching and maintenance • Fast deployments • Support for search templates • Zone awareness to automatically replicate data across two Availability Zones • Automatic failure recovery without service disruption • Automated backups • Authentication via Amazon IAM • Index level access control • Auditing via Amazon CloudTrail • Monitoring and Alerting via Amazon CloudWatch • Seamless scalability without availability issues • Wide variety of instance types and storage options • Up to 100 node cluster support • Programmatic scale-up/ scale- down support based on CloudWatch metrics • Data ingestion - Amazon Kinesis Firehose, Amazon IOT, Amazon CloudWatch Logs • Amazon CloudFormation support • Auditing and Monitoring - Amazon CloudTrail, Amazon CloudWatch • Amazon IAM based authentication • Elasticsearch open-source APIs, supports versions through ES 5.5 • Logstash support data for ingestion and transformation • Kibana support for visualization • Support for 11 plugins covering areas such as extended Unicode support, Phonetic analysis etc. Amazon ES benefits
  7. 7. Get started at FT – Financial Times P R O B L E M • Which stories do our readers care about? What’s hot? • Required a custom clickstream analytics solution. • Need a solution that delivers analytics in real-time. • Did not have a team to manage analytics infrastructure. S O L U T I O N • Streaming user data to Elasticsearch Service for analysis. Created their own custom dashboards for editors and journalists – Lantern. • Lantern - ”shines a light” on reader activity for the editors and journalists at the FT. • Critical tool for making editorial decisions. Daily editorial meetings B E N E F I T S • Reliability - Lantern is used throughout the day by journalists and editors. Relying on Amazon to manage their systems for maximum uptime. • Cost savings - Able to easily tune their cluster to meet their specifics needs without much management overhead
  8. 8. Security Model for Amazon ES
  9. 9. Get started at Service architecture AWS SDK AWS CLI AWS CloudFormation Elasticsearch data nodes Elasticsearch master nodes Elastic Load Balancing AWS IAM CloudWatchCloudTrail Amazon Elasticsearch Service domain Internet or your VPC AWS Console
  10. 10. Get started at AWS Security Responsibilities • Creation of a service VPC that allows limited access to the instances in your domain • Application of security patches on the instances • DDOS protection for the DNS name associated with the domain via Route53 • Hides ports 9200 and 9300 • Built on top of AWS secure networking
  11. 11. Get started at • Decide on your authentication strategy • IAM • VPC • Identify key roles and build IAM policies or create networking infrastructure to support their interaction with Amazon ES • Apply and maintain IAM policies and security groups to AWS resources Customer Security Responsibilities
  12. 12. Using IAM for Authentication
  13. 13. Get started at Access via the internet
  14. 14. Get started at Use a template or write your own We strongly recommend against using an “open access” policy
  15. 15. Get started at IAM policy application and resolution role Elastic IP address role policy policy policy Users have roles Or policies Instances have roles with policies and IPs Amazon ES domains have policies • IAM authenticates based on all applicable identification and all policies are in play User-based policies Resource-based policy
  16. 16. Access Policy Application & Resolution • Deny ALWAYS wins over competing policy types • If you do not explicitly state a policy, deny is default Allowed in a resource-based policy Denied in a resource-based policy Neither allowed nor denied in a resource-based policy Allowed in an identity-based policy Allow Deny Allow Denied in an identity-based policy Deny Deny Deny Neither allowed nor denied in an identity-based policy Allow Deny Deny
  17. 17. Get started at Policy skeleton { "Version": "2012-10-17", "Statement": [ { "Effect":... "Principal": ... "Action": [...], "Resource": ..., "Condition": ... } ] } • Effect: Allow or Deny • Principal: AWS account ID • Action • HTTP verbs • Service actions • Resource: Amazon ES domain/index • Condition: IP Address
  18. 18. Get started at Baseline IP-based access Amazon ESIAM Elastic IP address • Accessing from a known IP address • The domain’s policy controls the Actions that are allowed/denied • Access is anonymous Policy
  19. 19. Get started at • IP-based control, resource-based policy. All users, all Actions, all indexes { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "*” }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:12345678910:domain/test/*” "Condition": { "IpAddress": { "aws:SourceIp": [”"] } } } Access policy for IP-based access
  20. 20. Get started at Identity-based access Amazon ESIAM • Accessing with an identity, authenticated with access/secret key • Policy resolution as described • Access is via signed request Policy AWS SigV4 signing Lambda AWS SigV4 signing
  21. 21. Get started at • Requests must be signed. User-name-1 can run all actions against all indices { "Sid": "", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::12345678910:user/user-name-1” ] }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:12345678910:domain/test/*” } Access policy
  22. 22. Get started at Application access Application Amazon ES - search Amazon ES - monitor User Amazon RDSUpdater Kibana Proxy Administrator DevOps/IT
  23. 23. Get started at Create roles for access to Amazon ES Administrator IT/DevOps Application Updater Proxy Type User-based User-based Resource-based Resource-based Resource-based Actions • es:CreateElasticsearchDomain • es:Describe* • es:DeleteElasticsearchDomain • es:ListDomainNames • es:AddTags • es:ListTags • es:RemoveTags • es:Update* • es:ESHttpGet • es:ESHttpPut • es:ESHttpDelete • es:ESHttpPost • es:Describe* • es:ListDomainNames • es:AddTags • es:ListTags • es:RemoveTags • es:Update* es:ESHttpGet es:ESHttpPost es:ESHttpGet Resources Amazon ES search Amazon ES monitor Amazon ES search Amazon ES monitor Amazon ES search Amazon ES search Amazon ES Monitor IPs No No EIP for the application instance EIP for the updater instance EIP for the proxy
  24. 24. Amazon Elasticsearch Service in your VPC
  25. 25. Get started at Amazon ES architecture with VPC Availability Zone A Availability Zone B VPC subnet security group VPC subnet security group Amazon Elasticsearch Service Data Master Data Master IAM IAM
  26. 26. Get started at Set up for VPC access • Select your VPC • Select a subnet with sufficient IP space and ENIs for 3x your data instances • Select a security group to apply to the Amazon ES ENIs
  27. 27. Get started at Simple VPC access Availability Zone A Subnet A security group Amazon Elasticsearch Service Data Master Application Instance(s) Amazon RDS Internet gateway Amazon Route 53
  28. 28. Get started at Simple VPC access • Internet gateway provides access for application users, search, and monitoring traffic within the subnet • Security group has normal inbound/outbound rules • Because the IPs are within the security group, SigV4 signing is not required
  29. 29. Get started at Application search within VPC Availability Zone A Availability Zone B VPC subnet security group VPC subnet security group Amazon Elasticsearch Service Data Master Data Master IAMApplication Application IAM Internet gateway
  30. 30. Get started at Application search within the VPC • With Zone Awareness enabled, the domain is in 2 subnets • IAM provides additional security for IP-based or signed requests
  31. 31. Get started at Logging infrastructure in your VPC security group ELB Logstash Indexers Amazon Elasticsearch Service Domain Internet gateway
  32. 32. Get started at Logging infrastructure in your VPC • Logstash colocated with the infrastructure you are monitoring • Use an ELB across an autoscaled group of indexers to batch and forward to Amazon Elasticsearch Service • Use a reverse proxy in the VPC to forward Kibana traffic to Amazon ES
  33. 33. Get started at Conclusions • Amazon Elasticsearch Service now supports access control using VPC security groups • With IAM, you can further narrow access for particular users • All traffic remains within your VPC
  34. 34. Find out more: AWS Centralized Logging: Elasticsearch at the AWS Database Blog: Or ask your Solutions Architect! Amazon Elasticsearch Service