Nyc storm meetup_robdoherty

•

6 recomendaciones•2,271 vistas

Presentation at NYC Storm Meetup #1 on the Kafka-Storm implementation used in production at Outbrain Engage to track thousands of web traffic pings per second.

Tecnología Empresariales

Storm at
Rob Doherty
Senior Backend Engineer
rdoherty@outbrain.com
@robdoherty2

Before Storm
● Custom distributed processing system
● Python and ZMQ
● Advantages:
○ Simple components
○ Well-understood
● Disadvantages:
○ Did not scale
○ Batch-processing

Kafka + Storm
● Kafka: high-throughput distributed messaging
● Storm: distributed, real-time computation

Why Kafka?
● Need method to buffer clicks into “stream”
● Kafka + Storm common pattern for click tracking

Why Storm?
● “Real time” (15s latency requirements)
● Fault tolerance
● Easy to manage parallelism
● Stream grouping
● Active community
● Open-source project

Nginx Servers
Kafka Cluster
Storm Topology
Elastic Load Balancer
Customer
Traffic
AWS
MongoDB
Redis
Algo
API
Architecture

Kafka Cluster
● 40 Producers (8 m1.large instances)
○ Python brod
● 4 Brokers (4 m1.large instances)
10k Clicks per second (peak)
14B Clicks per month
Kafka v0.7.2

Storm Topology
● 40 Supervisors (c1.xlarge instances)
● 35 Bolts, 1 Kafka spout
● 250+ Executors (worker threads)
160k+ tuples executed per second
Storm v0.82
Leiningen v1.7

Customer
Traffic
Kafka Spout Aggregate 15s
Aggregate 5m
Position
Customer
Social
Arrangement
Front Page
@Handle
Storm Topology

Challenges
● Shell Bolts
● Anchor Bolts/Replaying Stream
● Acking Tuples
● Monitoring

Monitoring
● Scribe Logging
● Munin + Nagios
● JMX-JMXTrans + Ganglia
● Storm UI
● Thrift interface into Nimbus + D3

Future Plans
● Load testing
● Break topology into smaller pieces
● Move from AWS to private data center

Thank you
Rob Doherty
Senior Backend Engineer
rdoherty@outbrain.com
@robdoherty2

Más contenido relacionado

La actualidad más candente

Pywps

Jachym Cepicky

What's new with serverless on google cloud

GDG Cloud Bengaluru

InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...

Caner Ünal

Infrastructure as Code with Terraform: Koombea TechTalks

Koombea

Breaking the RpiDocker challenge

Nicolas De Loof

Jumbo the Hadoop cluster bootstrapper

Gauthier LEONARD

Clojure News Feed Performance Testing

Dynamical Software, Inc.

In this workshop we will set up a streaming framework which will process realtime data of traffic sensors installed within the Belgian road system. Starting with the intake of the data, you will learn best practices and the recommended approach to split the information into events in a way that won’t come back to haunt you. With some basic stream operations (count, filter, … ) you will get to know the data and experience how easy it is to get things done with Spring Boot & Spring Cloud Stream. But since simple data processing is not enough to fulfill all your streaming needs, we will also let you experience the power of windows. After this workshop, tumbling, sliding and session windows hold no more mysteries and you will be a true streaming wizard.

Stream Processing Live Traffic Data with Kafka Streams

Tim Ysewyn

Kafka y python

Paradigma Digital

Kafka practical experience

Rico Chen

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Flink) and in-house technologies have helped Uber scale.

Kafka Practices @ Uber - Seattle Apache Kafka meetup

Mingmin Chen

Grafana 7.0

Juraj Hantak

Apache Flink Training Workshop @ HadoopCon2016 - #4 Advanced Stream Processing

Apache Flink Taiwan User Group

Booking.com uses Sereal in many applications. One of the biggest use case though is the events pipeline. It was built to delivery messages (events) from generation point to various processors in near real-time fashion. These days it servers billions of messages per day. One of our processors recently faced scalability issues due to growth of the volume of delivered events. In this talk I would like to share what problem we had, how we addressed it and which new features of Sereal helped us.

Solving some of the scalability problems at booking.com

Ivan Kruglov

The Dark Side Of Go -- Go runtime related problems in TiDB in production

PingCAP

Yipit - AWS Start-Up Customer

Amazon Web Services

Order from chaos: automating monitoring configuration

Sensu Inc.

Serverless for the Cloud Native Era with Fission

NATS

Cassandra in Docker at Yelp: Opportunities and Challenges

Raghavendra Prabhu

Cloud Firestore – From JSON Deserialization to Object Document Mapping (ODM)

Minh Dao

La actualidad más candente (20)

Pywps

What's new with serverless on google cloud

InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...

Infrastructure as Code with Terraform: Koombea TechTalks

Breaking the RpiDocker challenge

Jumbo the Hadoop cluster bootstrapper

Clojure News Feed Performance Testing

Stream Processing Live Traffic Data with Kafka Streams

Kafka y python

Kafka practical experience

Kafka Practices @ Uber - Seattle Apache Kafka meetup

Grafana 7.0

Apache Flink Training Workshop @ HadoopCon2016 - #4 Advanced Stream Processing

Solving some of the scalability problems at booking.com

The Dark Side Of Go -- Go runtime related problems in TiDB in production

Yipit - AWS Start-Up Customer

Order from chaos: automating monitoring configuration

Serverless for the Cloud Native Era with Fission

Cassandra in Docker at Yelp: Opportunities and Challenges

Cloud Firestore – From JSON Deserialization to Object Document Mapping (ODM)

Similar a Nyc storm meetup_robdoherty

A Tour of Apache Kafka

confluent

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Monal Daxini

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Peter Bakas

Netflix Keystone—Cloud scale event processing pipeline

Monal Daxini

When it comes to cross-microservice communication, we must build an API, and the most significant criterion is that they all agree on the same type of exchange data, data format, and error format. We may now use REST/SOAP/Binary format for this, but we must also consider the endpoints, which appears to be a little tricky. Your life will be much easier now that we can quickly connect services and share data via gRPC very easily. So, in this session, we will learn about gRPC and then I will demonstrate Akka gRPC and its features live.

Akka gRPC quick-guide

Knoldus Inc.

Akka gRPC quick-guide

Knoldus Inc.

This talk was presented at SRE NYC Meetup on August 16, 2017 at Squarespace HQ. https://www.youtube.com/watch?v=UJ1QAKprVr4 As the engineering teams at Squarespace grow, we have been building more and more microservices. However, this has added operational strain as we try to shoehorn a growing, complex dynamic environment into our static data center infrastructure. We needed to rethink how we handle deployments, dependency management, resource allocation, monitoring, and alerting. Docker containerization and Kubernetes orchestration helps us tackle many of these problems, but the journey has been challenging. In this talk, we’ll discuss the challenges of running Kubernetes in a datacenter and how we switched to a more SLA-focused alert structure than per instance health with Prometheus and AlertManager.

Kubernetes @ Squarespace: Kubernetes in the Datacenter

Kevin Lynch

Netty training

Jackson dos Santos Olveira

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

How Uber scaled its Real Time Infrastructure to Trillion events per day

DataWorks Summit

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Uber Real Time Data Analytics

Ankur Bansal

Netty training

Marcelo Serpa

umeng analytical arch

Yan Zhang

Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016

Monal Daxini

Netflix Data Pipeline With Kafka

Steven Wu

Netflix Data Pipeline With Kafka

Allen (Xiaozhong) Wang

TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month

Nicolas Brousse

QCON 2015: Gearpump, Realtime Streaming on Akka

Sean Zhong

Eugene Bova "Dapr (Distributed Application Runtime) in a Microservices Archit...

LogeekNightUkraine

This talk was presented at the Hotstar Scale Meetup in Bangalore by Jayesh Sidhwani In this talk, the presenter introduces Apache Kafka and the Apache Kafka Streams library. Starting from the need for building streaming applications to thinking the use-cases as a streaming job - this talk covers all the technicalities. It ends with a short description of how Kafka is deployed and used at Hotstar

Build real time stream processing applications using Apache Kafka

Hotstar

From Three Nines to Five Nines - A Kafka Journey

Allen (Xiaozhong) Wang

Similar a Nyc storm meetup_robdoherty (20)

A Tour of Apache Kafka

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone—Cloud scale event processing pipeline

Akka gRPC quick-guide

Kubernetes @ Squarespace: Kubernetes in the Datacenter

Netty training

How Uber scaled its Real Time Infrastructure to Trillion events per day

Uber Real Time Data Analytics

Netty training

umeng analytical arch

Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016

Netflix Data Pipeline With Kafka

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month

QCON 2015: Gearpump, Realtime Streaming on Akka

Eugene Bova "Dapr (Distributed Application Runtime) in a Microservices Archit...

Build real time stream processing applications using Apache Kafka

From Three Nines to Five Nines - A Kafka Journey

Último

Scalable LLM APIs for AI and Generative AI Application Development Ettikan Karuppiah, Director/Technologist - NVIDIA Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

apidays

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

A Beginners Guide to Building a RAG App Using Open Source Milvus

Zilliz

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

AXA XL - Insurer Innovation Award Americas 2024

The Digital Insurer

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

DBX First Quarter 2024 Investor Presentation

Dropbox

Nyc storm meetup_robdoherty

1. Storm at Rob Doherty Senior Backend Engineer rdoherty@outbrain.com @robdoherty2

2. What is Outbrain?

3. Before Storm ● Custom distributed processing system ● Python and ZMQ ● Advantages: ○ Simple components ○ Well-understood ● Disadvantages: ○ Did not scale ○ Batch-processing

4. Kafka + Storm ● Kafka: high-throughput distributed messaging ● Storm: distributed, real-time computation

5. Why Kafka? ● Need method to buffer clicks into “stream” ● Kafka + Storm common pattern for click tracking

6. Why Storm? ● “Real time” (15s latency requirements) ● Fault tolerance ● Easy to manage parallelism ● Stream grouping ● Active community ● Open-source project

7. Nginx Servers Kafka Cluster Storm Topology Elastic Load Balancer Customer Traffic AWS MongoDB Redis Algo API Architecture

8. Kafka Cluster ● 40 Producers (8 m1.large instances) ○ Python brod ● 4 Brokers (4 m1.large instances) 10k Clicks per second (peak) 14B Clicks per month Kafka v0.7.2

9. Storm Topology ● 40 Supervisors (c1.xlarge instances) ● 35 Bolts, 1 Kafka spout ● 250+ Executors (worker threads) 160k+ tuples executed per second Storm v0.82 Leiningen v1.7

10. Customer Traffic Kafka Spout Aggregate 15s Aggregate 5m Position Customer Social Arrangement Front Page @Handle Storm Topology

11. Challenges ● Shell Bolts ● Anchor Bolts/Replaying Stream ● Acking Tuples ● Monitoring

12. Monitoring ● Scribe Logging ● Munin + Nagios ● JMX-JMXTrans + Ganglia ● Storm UI ● Thrift interface into Nimbus + D3

13.

14.

15. Monitoring ● Scribe Logging ● Munin + Nagios ● JMX-JMXTrans + Ganglia ● Storm UI ● Thrift interface into Nimbus + D3

16.

17.

18. Future Plans ● Load testing ● Break topology into smaller pieces ● Move from AWS to private data center

19. Thank you Rob Doherty Senior Backend Engineer rdoherty@outbrain.com @robdoherty2

Nyc storm meetup_robdoherty

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Nyc storm meetup_robdoherty

Similar a Nyc storm meetup_robdoherty (20)

Último

Último (20)

Nyc storm meetup_robdoherty