SlideShare a Scribd company logo
1 of 49
Download to read offline
Unmeltable Infrastructure at Scale:
Using Apache Kafka, Twitter Storm, and
ElasticSearch on AWS
Jim Nisbet

Philip O’Toole

CTO and VP of Engineering, Loggly

Lead Developer, Infrastructure, Loggly

November 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
What Loggly Does
• Log Management as a service
– Near real-time indexing of events

• Distributed architecture, built on AWS
• Initial production services in 2010
– Loggly Generation 2 released in Sept 2013

• Thousands of customers
Agenda for this Presentation
•
•
•
•
•

A bit about logging
Lessons learned from our first generation
How we leverage AWS services
Our use of Kafka, Storm, ElasticSearch
What worked well for us and what did not
Log Management
• Everyone starts with …
– A bunch of log files (syslog, application specific)
– On a bunch of machines

• Management consists of doing the simple stuff
– Rotate files, compress and delete
– Information is there but awkward to find specific events
– Weird log retention policies evolve over time
“…how can I make this someone else’s problem!”

“…hmmm, our logs are getting a bit bloated”

Log Volume

Self-Inflicted Pain

“…let’s spend time managing our log capacity”
Best Practices in Log Management
• Use existing logging infrastructure
– Real time syslog forwarding is built in
– Application log file watching

• Store logs externally
– Accessible when there is a system failure

• Log messages in machine parsable format
– JSON encoding when logging structured information
– Key-value pairs
From the Trenches…
• Managing Applications vs. Managing Logs
– Do not make this is an either/or proposition!

If you get a disk space alert, first login…
% sudo rm –rf /var/log/apache2/*

Admit it, we’ve all seen this kind of thing!
You Have Logs...
2013-10-25T18:35:43.387+0000: 441.482: [GC [PSYoungGen: 2430541K->268617K(2484544K)] 7687523K->5660738K(8076992K), 0.3266870
secs] [Times: user=1.05 sys=0.17, real=0.33 secs]
2013-10-25T18:35:43.714+0000: 441.809: [Full GC [PSYoungGen: 268617K->0K(2484544K)] [ParOldGen: 5392121K->354965K(5592448K)]
5660738K->354965K(8076992K) [PSPermGen: 44444K->44395K(83968K)], 0.9225290 secs] [Times: user=2.22 sys=0.26, real=0.92 secs]

• In this case, JVM garbage collection logs
enabled with…
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
Yes, you need to search these logs
But you also need to to spot trends
Loggly Offers Logging as a Service
Loggly First Generation
• Logging as a service
– Near real-time searchable logs

• Thousands of customers
– Transmission rates from 10 events/sec to 100k events/sec
– When customers systems are busy they send more logs
– Log traffic has distinct bursts; bursts can last for several hours

• Amazon EC2 deployment
– We used EC2 Instance storage

• SOLR Cloud
– Full power of Lucene search
– Tens of thousands of shards (with special ‘sleep shard’ logic)

• ZeroMQ for message queue
First Generation Lessons Learned
• Event ingestion too tightly coupled to indexing
– Manual re-indexing for temporary SOLR issues

• Multiple Indexing strategies needed
– 4 orders of magnitude difference between our high volume users
and our low volume users (10 eps vs. 100,000+ eps)
– Too much system overhead for low volume users
– Difficult to support changing indexing strategies for a customer
Big Data Infrastructure Solutions
We are not alone…
• Our challenges
–
–
–
–
–

Massive incoming event stream
Fundamentally multi-tenant
Scalable framework for analysis
Near real-time indexing
Time series index management

Scalability

Real
Time

Analytics

Multi
tenant

SaaS
Apache Kafka
• Overview
–
–
–
–

An Apache project initially developed at LinkedIn
Distributed publish-subscribe messaging system
Specifically designed for real time activity streams
Does not follow JMS Standards nor uses JMS APIs

• Key Features
–
–
–
–

Persistent messaging
High throughput, low overhead
Uses ZooKeeper for forming a cluster of nodes
Supports both queue and topic semantics
Message Queue Performance

http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
Storm Framework
• Storm (open sourced by Twitter)
– Open sourced September 2011
– Now an Apache Software Foundation project
• Currently Incubator Status

• Framework is for stream processing
–
–
–
–

Distributed
Fault tolerant
Computation
Fail-fast components
Storm Logical View

Example Topology
Bolt
Spout

Bolt

Bolt

Bolt
Spouts emit source stream

Bolts perform stream processing

Storm terminology
• Streams, Spouts, Bolts, Tasks, Workers, Stream Groups and Topologies
Storm Resource View
ElasticSearch
• Open source
– Commercial support available from ElasticSearch.com
– Growing open-source community

•
•
•
•
•

Distributed search engine
Fully exposes Lucene search functionality
Built for clustering from the ground-up
High availability
Multi-tenancy
ElasticSearch In Action
• Add/delete nodes dynamically
• Add indices with REST API
• Indices and Nodes have attributes
– Indices automatically moved to best Nodes

• Indices can be sharded
• Supports bulk insertion of events
• Plugins for monitoring cluster
Our Second Generation
Generation 2 – The Challenge
• Always accept log data
– Never make a customer’s incident worse

• Never drop log data
– A single log message could be critical

• True Elasticity
Perfect Match For Real Time Log Events
• Apache Kafka
– Extremely high-performance pub-sub persistent queue

• Consumer tracks their location in queue
– A good fit for our use cases

• Multiple Kafka brokers
– Good match for AWS
• Multiple brokers per region
• Availability Zone separation
Real Time Event Processing
• Twitter Storm
– Scalable real-time computation system

• Storm used as a “pull” system
– Provisioned for average load, not peak load
– Input from Kafka queue
• Worker nodes can be scaled dynamically

• Elasticity is key
– Another good match for AWS
• Able to scale workers up and down dynamically
Log Event Ingestion
Kafka
Queue
Loggly Collector Performance
• C++ multi-threaded
• Boost ASIO framework
• Each Collector can
handle 250k+ events
per second
– Per m2.2xlarge instance

1 x EC2 m2.2xlarge Collector
instance (300 byte average event
size).
Processing Events
Storm Event Processing

Classification

Kafka
Queue

Kafka
Queue

Rate
Determination
Summary
Statistics

Acme
S3 Bucket
Event Pipeline in Summary
• Storm provides Complex Event Processing
– Where we run much of our secret-sauce

• Kafka contains both raw and processed Events
• Snapshot the last day of Kafka events to S3
Elastic Search Clusters

From Kafka

Multi-Tiered
Elastic Cluster
Loggly and Index Management
• Indices are time-series data
– Separated by customer
– Represent slices of time
• Higher volume index will have shorter time slice

• Multi-tier architecture for efficient indexing
– Multiple indexing tiers mapped to different AWS instance types

• Efficient use of AWS resources
Staging Pre-Production System
Load Balancing

Kafka
Queue
Kafka enables Staging Architecture
• Kafka Broker doesn’t care if there are
multiple consumers
• Staging system runs pre-production code
• Pub-sub allows us to randomly index a
fraction of our production load
• A highly-effective pre-production system
AWS Deployment Details
AWS Deployment Instances – Collection

c1.xlarge

m2.2xlarge

•
•
•

•
•

Compute-optimized
High-traffic ingestion points
Disk not important

Memory-optimized
Disk buffer caching

4K Provisioned IOPs EBS
•
•
•

Ensures consistent IO
No noisy-neighbors
Persistent storage
AWS Deployment Instances – Processing

ZooKeeper

c1.xlarge
•
•
•

Compute-optimized
CPU-intensive
processing
Network IO

m1.xlarge
•
•
•

General-purpose
Configuration
Management
AWS Deployment Instances – Indexing

cc2.8xlarge
• 4K Provisioned IOPs EBS

m2.4xlarge
• 4K Provisioned IOPs EBS
A Few False Starts
Elastic Load Balancing in front of Collector
Had Limitations
• Initial testing used Elastic Load Balancing for incoming events:

• Elastic Load Balancing doesn’t allow forwarding port 514 (syslog)
• Elastic Load Balancing doesn’t support forwarding UDP
• Event traffic can burst and hit Elastic Load Balancing performance
limits
Amazon Route 53 DNS Round Robin a Win
• DNS Round Robin is pretty basic load balancing
– Not a bump in the wire

• Take advantage of AWS failover health checks
– When a collector goes out of service, it will be out of the DNS rotation

• Round Robin across multiple regions, AZs
– Latency based resolution optimizes inbound traffic
Our First Plan for Log Events
• Cassandra
– Highly scalable key-value store
– Impressive write performance a good match for us
– Apache project plus commercial support with DataStax

• Use Cassandra for both our Event Queue and
Persistent Store
– Our strategy was to get the raw event in to Cassandra
– …then perform workflow processing on events
Design meets Reality
• Cassandra not designed to be a message
queue
• Hard to track Events received out-of-order

• Multi-tenancy requires handling data bursts
– Collectors still needed to be able to buffer to disk
– Added complexity and became a point of failure
Big Wins
• Leveraging AWS services
–
–
–
–

Multi-region, multi-AZ
Provisioned IOPS for availability and scale
Amazon Route 53 DNS support with latency resolution
Easy to increase and decrease Storm resources

• Leveraging Open Source infrastructure
– Apache Kafka
– Twitter Storm
– ElasticSearch

• Pre-production “Staging” system
The Means to an End
Feedback
• Questions?

Jim Nisbet (niz@loggly.com)
CTO and VP of Engineering, Loggly

Philip O’Toole (philip@loggly.com)
Lead Developer, Infrastructure, Loggly

Follow us @loggly!
Please give us your feedback on this
presentation

ARC303
As a thank you, we will select prize
winners daily for completed surveys!

More Related Content

What's hot

Monitoring Flink with Prometheus
Monitoring Flink with PrometheusMonitoring Flink with Prometheus
Monitoring Flink with PrometheusMaximilian Bode
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka StreamsGuozhang Wang
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignMichael Noll
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using KafkaKnoldus Inc.
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producerconfluent
 
Elasticsearch in Netflix
Elasticsearch in NetflixElasticsearch in Netflix
Elasticsearch in NetflixDanny Yuan
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large ScaleVerverica
 
Introduction to Apache Flink
Introduction to Apache FlinkIntroduction to Apache Flink
Introduction to Apache Flinkmxmxm
 
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETStream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETconfluent
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent
 
APACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsAPACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsKetan Gote
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkFlink Forward
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward
 
One sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkOne sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkFlink Forward
 
ksqlDB로 시작하는 스트림 프로세싱
ksqlDB로 시작하는 스트림 프로세싱ksqlDB로 시작하는 스트림 프로세싱
ksqlDB로 시작하는 스트림 프로세싱confluent
 
Integrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect FrameworkIntegrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect Frameworkconfluent
 

What's hot (20)

Monitoring Flink with Prometheus
Monitoring Flink with PrometheusMonitoring Flink with Prometheus
Monitoring Flink with Prometheus
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Elasticsearch in Netflix
Elasticsearch in NetflixElasticsearch in Netflix
Elasticsearch in Netflix
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
 
Introduction to Apache Flink
Introduction to Apache FlinkIntroduction to Apache Flink
Introduction to Apache Flink
 
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETStream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NET
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database System
 
APACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsAPACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka Streams
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
 
One sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkOne sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async Sink
 
ksqlDB로 시작하는 스트림 프로세싱
ksqlDB로 시작하는 스트림 프로세싱ksqlDB로 시작하는 스트림 프로세싱
ksqlDB로 시작하는 스트림 프로세싱
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Integrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect FrameworkIntegrating Apache Kafka and Elastic Using the Connect Framework
Integrating Apache Kafka and Elastic Using the Connect Framework
 

Viewers also liked

Real time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache SparkReal time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache SparkRahul Jain
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman
 
An Introduction to Elastic Search.
An Introduction to Elastic Search.An Introduction to Elastic Search.
An Introduction to Elastic Search.Jurriaan Persyn
 
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...Michael Noll
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
 
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...SolarWinds Loggly
 
Apache Kafka at LinkedIn
Apache Kafka at LinkedInApache Kafka at LinkedIn
Apache Kafka at LinkedInGuozhang Wang
 
Being Ready for Apache Kafka - Apache: Big Data Europe 2015
Being Ready for Apache Kafka - Apache: Big Data Europe 2015Being Ready for Apache Kafka - Apache: Big Data Europe 2015
Being Ready for Apache Kafka - Apache: Big Data Europe 2015Michael Noll
 
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...Amazon Web Services
 
Real-time streaming and data pipelines with Apache Kafka
Real-time streaming and data pipelines with Apache KafkaReal-time streaming and data pipelines with Apache Kafka
Real-time streaming and data pipelines with Apache KafkaJoe Stein
 
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...DataWorks Summit/Hadoop Summit
 
Building a Replicated Logging System with Apache Kafka
Building a Replicated Logging System with Apache KafkaBuilding a Replicated Logging System with Apache Kafka
Building a Replicated Logging System with Apache KafkaGuozhang Wang
 
Apache Kafka lessons learned @PAYBACK
Apache Kafka lessons learned @PAYBACKApache Kafka lessons learned @PAYBACK
Apache Kafka lessons learned @PAYBACKMaxim Shelest
 
Kafka At Scale in the Cloud
Kafka At Scale in the CloudKafka At Scale in the Cloud
Kafka At Scale in the Cloudconfluent
 
Hadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureHadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureP. Taylor Goetz
 

Viewers also liked (20)

Real time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache SparkReal time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache Spark
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
An Introduction to Elastic Search.
An Introduction to Elastic Search.An Introduction to Elastic Search.
An Introduction to Elastic Search.
 
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
 
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
Why @Loggly Loves Apache Kafka, and How We Use Its Unbreakable Messaging for ...
 
Apache Kafka at LinkedIn
Apache Kafka at LinkedInApache Kafka at LinkedIn
Apache Kafka at LinkedIn
 
Being Ready for Apache Kafka - Apache: Big Data Europe 2015
Being Ready for Apache Kafka - Apache: Big Data Europe 2015Being Ready for Apache Kafka - Apache: Big Data Europe 2015
Being Ready for Apache Kafka - Apache: Big Data Europe 2015
 
Elastic Search
Elastic SearchElastic Search
Elastic Search
 
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
 
Real-time streaming and data pipelines with Apache Kafka
Real-time streaming and data pipelines with Apache KafkaReal-time streaming and data pipelines with Apache Kafka
Real-time streaming and data pipelines with Apache Kafka
 
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
 
Building a Replicated Logging System with Apache Kafka
Building a Replicated Logging System with Apache KafkaBuilding a Replicated Logging System with Apache Kafka
Building a Replicated Logging System with Apache Kafka
 
(Elastic)search in big data
(Elastic)search in big data(Elastic)search in big data
(Elastic)search in big data
 
Apache Kafka lessons learned @PAYBACK
Apache Kafka lessons learned @PAYBACKApache Kafka lessons learned @PAYBACK
Apache Kafka lessons learned @PAYBACK
 
Netflix Data Pipeline With Kafka
Netflix Data Pipeline With KafkaNetflix Data Pipeline With Kafka
Netflix Data Pipeline With Kafka
 
Kafka At Scale in the Cloud
Kafka At Scale in the CloudKafka At Scale in the Cloud
Kafka At Scale in the Cloud
 
Hadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureHadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm Architecture
 

Similar to Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013

AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly SolarWinds Loggly
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analyticsamesar0
 
Extending Spark Streaming to Support Complex Event Processing
Extending Spark Streaming to Support Complex Event ProcessingExtending Spark Streaming to Support Complex Event Processing
Extending Spark Streaming to Support Complex Event ProcessingOh Chan Kwon
 
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per SecondAmazon Web Services
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudyJohn Adams
 
Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...
Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...
Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...Amazon Web Services
 
Netflix Keystone—Cloud scale event processing pipeline
Netflix Keystone—Cloud scale event processing pipelineNetflix Keystone—Cloud scale event processing pipeline
Netflix Keystone—Cloud scale event processing pipelineMonal Daxini
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...
TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...
TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...Trivadis
 
Flink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasFlink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasMonal Daxini
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithMarkus Eisele
 
Scality S3 Server: Node js Meetup Presentation
Scality S3 Server: Node js Meetup PresentationScality S3 Server: Node js Meetup Presentation
Scality S3 Server: Node js Meetup PresentationScality
 
Performance architecture for cloud connect
Performance architecture for cloud connectPerformance architecture for cloud connect
Performance architecture for cloud connectAdrian Cockcroft
 
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudRick Bilodeau
 
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc.
 
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
 
Flying Server-less on the Cloud with AWS Lambda
Flying Server-less on the Cloud with AWS LambdaFlying Server-less on the Cloud with AWS Lambda
Flying Server-less on the Cloud with AWS LambdaSerkan Özal
 

Similar to Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013 (20)

AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analytics
 
Extending Spark Streaming to Support Complex Event Processing
Extending Spark Streaming to Support Complex Event ProcessingExtending Spark Streaming to Support Complex Event Processing
Extending Spark Streaming to Support Complex Event Processing
 
Spark cep
Spark cepSpark cep
Spark cep
 
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudy
 
Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...
Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...
Amazon RDS for MySQL – Diagnostics, Security, and Data Migration (DAT302) | A...
 
Netflix Keystone—Cloud scale event processing pipeline
Netflix Keystone—Cloud scale event processing pipelineNetflix Keystone—Cloud scale event processing pipeline
Netflix Keystone—Cloud scale event processing pipeline
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience Sharing
 
TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...
TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...
TechEvent 2019: Oracle Databases as Managed Service at AWS, Yes it works!; Al...
 
Flink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasFlink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paas
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolith
 
Create cloud service on AWS
Create cloud service on AWSCreate cloud service on AWS
Create cloud service on AWS
 
Scality S3 Server: Node js Meetup Presentation
Scality S3 Server: Node js Meetup PresentationScality S3 Server: Node js Meetup Presentation
Scality S3 Server: Node js Meetup Presentation
 
Performance architecture for cloud connect
Performance architecture for cloud connectPerformance architecture for cloud connect
Performance architecture for cloud connect
 
AWS glue technical enablement training
AWS glue technical enablement trainingAWS glue technical enablement training
AWS glue technical enablement training
 
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
 
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
 
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !
 
Flying Server-less on the Cloud with AWS Lambda
Flying Server-less on the Cloud with AWS LambdaFlying Server-less on the Cloud with AWS Lambda
Flying Server-less on the Cloud with AWS Lambda
 

More from Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Recently uploaded

IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 

Recently uploaded (20)

IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 

Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013

  • 1. Unmeltable Infrastructure at Scale: Using Apache Kafka, Twitter Storm, and ElasticSearch on AWS Jim Nisbet Philip O’Toole CTO and VP of Engineering, Loggly Lead Developer, Infrastructure, Loggly November 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 2. What Loggly Does • Log Management as a service – Near real-time indexing of events • Distributed architecture, built on AWS • Initial production services in 2010 – Loggly Generation 2 released in Sept 2013 • Thousands of customers
  • 3. Agenda for this Presentation • • • • • A bit about logging Lessons learned from our first generation How we leverage AWS services Our use of Kafka, Storm, ElasticSearch What worked well for us and what did not
  • 4. Log Management • Everyone starts with … – A bunch of log files (syslog, application specific) – On a bunch of machines • Management consists of doing the simple stuff – Rotate files, compress and delete – Information is there but awkward to find specific events – Weird log retention policies evolve over time
  • 5. “…how can I make this someone else’s problem!” “…hmmm, our logs are getting a bit bloated” Log Volume Self-Inflicted Pain “…let’s spend time managing our log capacity”
  • 6. Best Practices in Log Management • Use existing logging infrastructure – Real time syslog forwarding is built in – Application log file watching • Store logs externally – Accessible when there is a system failure • Log messages in machine parsable format – JSON encoding when logging structured information – Key-value pairs
  • 7. From the Trenches… • Managing Applications vs. Managing Logs – Do not make this is an either/or proposition! If you get a disk space alert, first login… % sudo rm –rf /var/log/apache2/* Admit it, we’ve all seen this kind of thing!
  • 8. You Have Logs... 2013-10-25T18:35:43.387+0000: 441.482: [GC [PSYoungGen: 2430541K->268617K(2484544K)] 7687523K->5660738K(8076992K), 0.3266870 secs] [Times: user=1.05 sys=0.17, real=0.33 secs] 2013-10-25T18:35:43.714+0000: 441.809: [Full GC [PSYoungGen: 268617K->0K(2484544K)] [ParOldGen: 5392121K->354965K(5592448K)] 5660738K->354965K(8076992K) [PSPermGen: 44444K->44395K(83968K)], 0.9225290 secs] [Times: user=2.22 sys=0.26, real=0.92 secs] • In this case, JVM garbage collection logs enabled with… -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
  • 9. Yes, you need to search these logs
  • 10. But you also need to to spot trends
  • 11. Loggly Offers Logging as a Service
  • 12. Loggly First Generation • Logging as a service – Near real-time searchable logs • Thousands of customers – Transmission rates from 10 events/sec to 100k events/sec – When customers systems are busy they send more logs – Log traffic has distinct bursts; bursts can last for several hours • Amazon EC2 deployment – We used EC2 Instance storage • SOLR Cloud – Full power of Lucene search – Tens of thousands of shards (with special ‘sleep shard’ logic) • ZeroMQ for message queue
  • 13. First Generation Lessons Learned • Event ingestion too tightly coupled to indexing – Manual re-indexing for temporary SOLR issues • Multiple Indexing strategies needed – 4 orders of magnitude difference between our high volume users and our low volume users (10 eps vs. 100,000+ eps) – Too much system overhead for low volume users – Difficult to support changing indexing strategies for a customer
  • 14. Big Data Infrastructure Solutions We are not alone… • Our challenges – – – – – Massive incoming event stream Fundamentally multi-tenant Scalable framework for analysis Near real-time indexing Time series index management Scalability Real Time Analytics Multi tenant SaaS
  • 15. Apache Kafka • Overview – – – – An Apache project initially developed at LinkedIn Distributed publish-subscribe messaging system Specifically designed for real time activity streams Does not follow JMS Standards nor uses JMS APIs • Key Features – – – – Persistent messaging High throughput, low overhead Uses ZooKeeper for forming a cluster of nodes Supports both queue and topic semantics
  • 17. Storm Framework • Storm (open sourced by Twitter) – Open sourced September 2011 – Now an Apache Software Foundation project • Currently Incubator Status • Framework is for stream processing – – – – Distributed Fault tolerant Computation Fail-fast components
  • 18. Storm Logical View Example Topology Bolt Spout Bolt Bolt Bolt Spouts emit source stream Bolts perform stream processing Storm terminology • Streams, Spouts, Bolts, Tasks, Workers, Stream Groups and Topologies
  • 20. ElasticSearch • Open source – Commercial support available from ElasticSearch.com – Growing open-source community • • • • • Distributed search engine Fully exposes Lucene search functionality Built for clustering from the ground-up High availability Multi-tenancy
  • 21. ElasticSearch In Action • Add/delete nodes dynamically • Add indices with REST API • Indices and Nodes have attributes – Indices automatically moved to best Nodes • Indices can be sharded • Supports bulk insertion of events • Plugins for monitoring cluster
  • 23. Generation 2 – The Challenge • Always accept log data – Never make a customer’s incident worse • Never drop log data – A single log message could be critical • True Elasticity
  • 24. Perfect Match For Real Time Log Events • Apache Kafka – Extremely high-performance pub-sub persistent queue • Consumer tracks their location in queue – A good fit for our use cases • Multiple Kafka brokers – Good match for AWS • Multiple brokers per region • Availability Zone separation
  • 25. Real Time Event Processing • Twitter Storm – Scalable real-time computation system • Storm used as a “pull” system – Provisioned for average load, not peak load – Input from Kafka queue • Worker nodes can be scaled dynamically • Elasticity is key – Another good match for AWS • Able to scale workers up and down dynamically
  • 28. Loggly Collector Performance • C++ multi-threaded • Boost ASIO framework • Each Collector can handle 250k+ events per second – Per m2.2xlarge instance 1 x EC2 m2.2xlarge Collector instance (300 byte average event size).
  • 31. Event Pipeline in Summary • Storm provides Complex Event Processing – Where we run much of our secret-sauce • Kafka contains both raw and processed Events • Snapshot the last day of Kafka events to S3
  • 32. Elastic Search Clusters From Kafka Multi-Tiered Elastic Cluster
  • 33. Loggly and Index Management • Indices are time-series data – Separated by customer – Represent slices of time • Higher volume index will have shorter time slice • Multi-tier architecture for efficient indexing – Multiple indexing tiers mapped to different AWS instance types • Efficient use of AWS resources
  • 36. Kafka enables Staging Architecture • Kafka Broker doesn’t care if there are multiple consumers • Staging system runs pre-production code • Pub-sub allows us to randomly index a fraction of our production load • A highly-effective pre-production system
  • 38. AWS Deployment Instances – Collection c1.xlarge m2.2xlarge • • • • • Compute-optimized High-traffic ingestion points Disk not important Memory-optimized Disk buffer caching 4K Provisioned IOPs EBS • • • Ensures consistent IO No noisy-neighbors Persistent storage
  • 39. AWS Deployment Instances – Processing ZooKeeper c1.xlarge • • • Compute-optimized CPU-intensive processing Network IO m1.xlarge • • • General-purpose Configuration Management
  • 40. AWS Deployment Instances – Indexing cc2.8xlarge • 4K Provisioned IOPs EBS m2.4xlarge • 4K Provisioned IOPs EBS
  • 41. A Few False Starts
  • 42. Elastic Load Balancing in front of Collector Had Limitations • Initial testing used Elastic Load Balancing for incoming events: • Elastic Load Balancing doesn’t allow forwarding port 514 (syslog) • Elastic Load Balancing doesn’t support forwarding UDP • Event traffic can burst and hit Elastic Load Balancing performance limits
  • 43. Amazon Route 53 DNS Round Robin a Win • DNS Round Robin is pretty basic load balancing – Not a bump in the wire • Take advantage of AWS failover health checks – When a collector goes out of service, it will be out of the DNS rotation • Round Robin across multiple regions, AZs – Latency based resolution optimizes inbound traffic
  • 44. Our First Plan for Log Events • Cassandra – Highly scalable key-value store – Impressive write performance a good match for us – Apache project plus commercial support with DataStax • Use Cassandra for both our Event Queue and Persistent Store – Our strategy was to get the raw event in to Cassandra – …then perform workflow processing on events
  • 45. Design meets Reality • Cassandra not designed to be a message queue • Hard to track Events received out-of-order • Multi-tenancy requires handling data bursts – Collectors still needed to be able to buffer to disk – Added complexity and became a point of failure
  • 46. Big Wins • Leveraging AWS services – – – – Multi-region, multi-AZ Provisioned IOPS for availability and scale Amazon Route 53 DNS support with latency resolution Easy to increase and decrease Storm resources • Leveraging Open Source infrastructure – Apache Kafka – Twitter Storm – ElasticSearch • Pre-production “Staging” system
  • 47. The Means to an End
  • 48. Feedback • Questions? Jim Nisbet (niz@loggly.com) CTO and VP of Engineering, Loggly Philip O’Toole (philip@loggly.com) Lead Developer, Infrastructure, Loggly Follow us @loggly!
  • 49. Please give us your feedback on this presentation ARC303 As a thank you, we will select prize winners daily for completed surveys!