Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Collecting AWS Logs & Introducing Splunk New S3 Compatible Storage (SmartStore)

248 visualizaciones

Publicado el

Two presentations at the January Splunk User Group in Edinburgh. Presenters were Harry McLaren and Tomasz Dziwok.

Topics covered are collecting AWS based logs at scale with Splunk and what the new object-based storage feature is within Splunk Enterprise (SmartStore).

Publicado en: Datos y análisis
  • Sé el primero en comentar

Collecting AWS Logs & Introducing Splunk New S3 Compatible Storage (SmartStore)

  1. 1. © 2019 SPLUNK INC.© 2019 SPLUNK INC. Splunk User Group Edinburgh
  2. 2. © 2019 SPLUNK INC. Harry McLaren ● Managing Consultant at ECS Security ● Member of SplunkTrust (MVP) ● Leader of the Splunk User Group Edinburgh
  3. 3. © 2019 SPLUNK INC. Introduction to ECS Security Splunk Partner - UK – Security Consultancy & Managed SOC Provider – Splunk Revolution Award & Splunk Partner of the Year
  4. 4. © 2019 SPLUNK INC. Agenda • Housekeeping: Event Overview & House Rules • AWS Log Collection at Scale (Tomasz Dziwok) • An Overview of SmartStore (Harry McLaren) • Configuration Monitoring TA for Splunk (Tomasz Dziwok)
  5. 5. © 2019 SPLUNK INC. Splunk [Official] User Group “The overall goal is to create an authentic, ongoing user group experience for our users, where they contribute and get involved” ● Technical Discussions ● Sharing Environment ● Build Trust ● No Sales!
  6. 6. © 2019 SPLUNK INC. AWS Log Collection at Scale Tomasz Dziwok
  7. 7. © 2019 SPLUNK INC. Amazon Web Services Log Collection
  8. 8. © 2019 SPLUNK INC. ▶ Advantages • HA & Distributed • Highly Versitile ▶ Drawbacks • Not really real-time (requires pull) • Uses S3 Bucket as a "buffer" • Rather complex ▶ Recommened for: • Acces Logs (ELB, CloudFront, S3) • Config • CloudTrail ▶ Usable for: • Any data that can be logged to S3 SQS S3-Based Inputs ▶ Data is sent to S3 as files ▶ SNS Queue is notified about new file ▶ SNS Notifies SQS ▶ Splunk HF polls SQS • If succesful; Splunk pulls the S3 file
  9. 9. © 2019 SPLUNK INC. ▶ Advantages • HA & Distributed • Real-time (push based) • Very Scalable ▶ Drawbacks • Few supported sources • One firehose required per sourcetype • Not possible to configure exclusively in AWS Web Console ▶ Recommened for: • VPC Flow Logs • GuardDuty Events ▶ Usable for: • Other Events of interest (e.g. EC2 events) Kinesis Data Firehose Inputs ( not kinesis stream ) ▶ VPC Flow: • Sent to Cloud Watch Logs • Forwarded to Kinesis Firehose • Pre-Processed in Lambda • Sent to Splunk HEC ▶ Cloud Watch Events • Forwarded to firehose • Sent to Splunk HEC
  10. 10. © 2019 SPLUNK INC. ▶ Advantages • Can collect from sources that are otherwise not collectable at all • Very Easy to configure ▶ Drawbacks • Not HA • Not Distributed • Pull Required ▶ Recommened for: • Inspector • Config Rules • CloudWatch • "Description" • Billing ▶ Usable for: • Virtually anything available in AWS AWS API-Based Inputs ▶ A Splunk HF polls AWS API ▶ The response is indexed ▶ There is no Step 3
  11. 11. © 2019 SPLUNK INC. AWS Permissions the most challanging part of the proccess NO! YES!
  12. 12. © 2019 SPLUNK INC. AWS Permissions : Splunk https://docs.splunk.com/Documentation/AddOns/released/AWS/ConfigureAWSpermissions MAYBE?
  13. 13. © 2019 SPLUNK INC. Linking Arbitrary S3 Bucket to SNS
  14. 14. © 2019 SPLUNK INC. ▶ Permissions ▶ Check data flow in AWS • S3 Bucket • Traffic charts ▶ CA Validity ▶ Permissions AWS/Splunk Debugging
  15. 15. © 2019 SPLUNK INC. ▶ Splunk _internal • index=_internal source!="/opt/splunk/var/log/splunk/splunkd_ui_access.log" err* OR warn* OR crit* ▶ AWS Policy Simulator • https://policysim.aws.amazon.com/home/index.jsp AWS/Splunk Debugging
  16. 16. © 2019 SPLUNK INC. An Overview of SmartStore Harry McLaren
  17. 17. © 2019 SPLUNK INC. ➢ Independently scale up/down compute (CPUs) and data storage based on business demands ➢ Automatically evaluates users’ data access patterns (via app- aware cache) – placing actively accessed data in local storage for real-time analytics; inactive data moved to low-cost, remote storage (any S3-compatible environment) Splunk SmartStore Maintain Performance & Availability (Lower Storage Cost) Search Indexers Storage
  18. 18. © 2019 SPLUNK INC. Splunk SmartStore Codename: S2 - Available in Splunk Enterprise 7.2+
  19. 19. © 2019 SPLUNK INC. ▶ Local Storage • Hot buckets are always on local storage [homePath] − No change from classic architecture ▶ Remote Storage • Buckets are copied to the [remotePath] when they roll from Hot  Warm - Remote storage must provide data protection - Splunk does not provide resiliency for buckets in remote storage ▶ Cache Manager • Recently read buckets are also on local cache − New indexer functionality (not a new role) − Each indexer has a cache manager that operates independently − Retrieves buckets from the remote store when needed − Evicts buckets from the local cache [homePath] Storage Architecture
  20. 20. © 2019 SPLUNK INC. 1. Data arrives and is written to a Hot bucket − This occurs using the standard indexing pipeline 2. The bucket rolls to warm 3. Bucket is registered with the cache manager 4. Cache manager uploads the bucket to the remote store 5. Bucket remains local and searchable until evicted by the cache manager Getting Data In Non-Clustered Deployments Remote Storage [remotePath] Hot/Cache Storage [homePath] 1 2 3 4 5
  21. 21. © 2019 SPLUNK INC. 1. Data arrives and is written to a Hot bucket 2. Hot bucket streams to cluster peer(s) according to RF 3. Replication completes and the buckets roll to warm 4. Buckets are registered with their cache managers 5. Cache manager on source peer uploads the bucket to the remote store 6. Source peer notifies replication peers that the bucket was uploaded successfully 7. Replication peers delete their local copy of the bucket and retain a stub 8. Cached copy remains on the source peer until evicted by the local cache manager Getting Data In Clustered Deployments Remote Storage Hot/Cache Storage 1 23 4 5 Hot/Cache Storage 1 6 78
  22. 22. © 2019 SPLUNK INC. Storage Configuration indexes.conf, server.conf, limits.conf ▶ Volume Configuration ▶ Enabled Per-Index − You can have a mix of SmartStore and classic indexes on the same indexer ▶ Index Configuration Conf File indexes.conf Parameters [volume:<volume_name>] storageType = remote path = <scheme>://<remote-location-specifier> remote.s3.endpoint = <URL of S3 API> remote.s3.secret_key = remote.s3.access_key = Conf File indexes.conf Parameters [<index_name>] homePath = coldPath = <path required, but not used> remotePath = <volume_name>/<index_name> maxGlobalDataSizeMB = frozenTimePeriodInSecs =
  23. 23. © 2019 SPLUNK INC. Some assertions about Splunk searches… ▶ Typically over near-term data − Research has shown that 97% of searches look back 24hrs or less By default, the cache manager will attempt to cache buckets with recent events ▶ Typically have spatial and temporal locality − If I find an event at a specific time or in a log, I will likely run additional searches against data at that time or in that log By default, the cache manager will attempt to cache recently accessed buckets Searching with SmartStore
  24. 24. © 2019 SPLUNK INC. 1. Search request is received 2. Indexer generates a list of relevant buckets to be searched 3. Search process is spawned 4. Spawned process reads the bucket list 5. Hot buckets are searched in the same manner as “classic” search Searching with SmartStore Hot Buckets Remote Storage Hot/Cache Storage 2 3 4 5 1 HOT
  25. 25. © 2019 SPLUNK INC. 1. Search process “opens” the bucket with the cache manager (Indexer) 2. Cache manager tells the search process that the bucket is local and available for search 3. Search process searches the bucket 4. Search process ”closes” the bucket with the cache manager Searching with SmartStore Cached Buckets Remote Storage Hot/Cache Storage 2 3 4 1 CACHED
  26. 26. © 2019 SPLUNK INC. 1. Search process “opens” the bucket with the Cache manager, but it isn’t in cache 2. Search process waits 3. Cache manager fetches the bucket from the remote store 4. Cache manager tells the search process that the bucket is local and available for search 5. Search process searches the bucket 6. Search process ”closes” the bucket with the cache manager 7. Bucket remains in cache until evicted by the cache manager Searching with SmartStore Remote Buckets Remote Storage Hot/Cache Storage 2 3 4 1 5 6 7 II CACHED
  27. 27. © 2019 SPLUNK INC. Architecture Small Clustered Deployment with SmartStore
  28. 28. © 2019 SPLUNK INC. ▶ Splunk Docs • About SmartStore: https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/AboutSmartStore • SmartStore Architecture: https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/SmartStorearchitecture • How Indexing Works in SmartStore: https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/SmartStoreindexing • How Search Works in SmartStore: https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/SmartStoresearching Resources
  29. 29. © 2019 SPLUNK INC. Configuration Monitoring TA for Splunk Tomasz Dziwok
  30. 30. © 2019 SPLUNK INC. Quick Review: What is it?
  31. 31. © 2019 SPLUNK INC. ▶ Open Source ▶ https://gitlab.com/ecs_ public_projects/splunk/ ta-confversion ▶ SplunkBase ▶ https://splunkbase.splu nk.com/app/4364/ ▶ Continued Development ▶ Merge Requests Welcome ▶ Version 1.1 to release by end of week Now Available
  32. 32. © 2019 SPLUNK INC. Get Involved! ● Splunk User Group Edinburgh – https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html – https://www.linkedin.com/groups/12013212 ● Splunk’s Slack Group – Register via http://splunk-usergroups.signup.team/ – Channel: #edinburgh ● Present & Share at the User Group? Connect: ‣ Harry McLaren | harry.mclaren@ecssecurity.co.uk | @cyberharibu | harrymclaren.co.uk

×