Rootconf

•

1 recomendación•567 vistas

A

How LinkedIn is leveraging kafka for log aggregation and monitoring

How LinkedIn is leveraging kafka for log aggregation and
monitoring
Akbar KM, Muzammil Rehman
SRE Linkedin

Agenda
● What is Pub Sub?
● Kafka basics
● Kafka for monitoring
● inGraph’s
● ELK at Linkedin

What is pub sub ?
Producer Consumer
Producer
Consumer
Topic
1
Topic
2
Topic
3
subscribe
publish(topic, msg)
Publish subscribe
system
msg
msg

Kafka

Consumer Groups

inGraph
● inGraphs is a visualization frontend for a linkedin’s metrics
● Agent is a library used by all java application
● Data is transported over kafka
● Collector consume and write to rrd
● Features
○ Rest based
○ Self service
○ Little over 500,000 metrics collected per minute
○ Each RRD is roughly 815k. Each RRD is written to two collectors
to maintain data integrity
○ We currently have 1.4TB of SSD in production colo

ELK at linkedin
● ELK (Elasticsearch, Logstash,
Kibana)
○ E -> Near Real-Time Search &
Analysis
○ L -> Plumbing and Glue for all
your data
○ K -> Line graphs, pie charts,
dashboard
● Logstash kafka plugin consumes
logs and end up written into
elasticsearch

Why Kafka
1. Near real time delivery
2. Supports multiple consumers groups aka queuing
3. No overhead on clients that comes up logstash agent
4. Kafka scales horizontally
5. Supports REST out of the box
6. In house support

Reference
http://engineering.linkedin.com/52/autometrics-
self-service-metrics-collection
ELK http://www.elastic.co/guide/

Más contenido relacionado

La actualidad más candente

(Preston Thompson, Braze) Kafka Summit SF 2018 If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy. We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors. In this talk, I will discuss: -How we started our journey in designing this large-scale streaming architecture -Why streaming technologies were necessary to solve our technology and business issues -The lessons we learned along the way that can help you with your Kafka-based architecture

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Putting the Spark into Functional Fashion Tech Analystics

Putting the Spark into Functional Fashion Tech Analystics

Putting the Spark into Functional Fashion Tech Analystics

Apache Flink

Elastic Stack Basic - All The Capabilities in 6.3!

Elastic Stack Basic - All The Capabilities in 6.3!

Elastic Stack Basic - All The Capabilities in 6.3!

Session 03 data_migration_at_scale_by_sameer

Session 03 data_migration_at_scale_by_sameer

Session 03 data_migration_at_scale_by_sameer

At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

HostedbyConfluent

Building scalable data with kafka and spark

Building scalable data with kafka and spark

Building scalable data with kafka and spark

babatunde ekemode

Kafka Connectors are used extensively in data migration solutions, serving as a middle tier when migrating data across databases. In addition, microservice architectures also use Kafka Connectors heavily when communicating with one another while still operating independently on their own data stores. In this talk, we cover these use cases in more detail along with a deep dive into the architecture of the source and sink Kafka Connectors for Cosmos DB.

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

HostedbyConfluent

PayPal currently processes tens of billions of signals per day from different sources in batch and streaming mode. The data processing platform is the one powering these different analytical needs and use cases, not just at PayPal but our adjacencies like Venmo, Hyperwallet and iZettle. End users of this platform demand access to data insights with as much flexibility as possible to explore it with low processing latency. One such use case is where our Switchboard(data de-multiplexer) platform where we process approximately 20 billion events daily and provide data to different teams and platforms with-in PayPal and also to platform outside PayPal for more insights. When we started building this platform Kafka was just another asynchronous message processing platform for us but we have seen it evolving to a place where its adds value not just in terms of event processing but also for platform resiliency and scalability. Takeaway for the audience: Most people work with and have knowledge about data. With this talk I want to present information which is relevant and meaningful to the audience. Information and examples which will make it easier for attendees to understand our complex system and hopefully have some practical takeaways to use Kafka for similar problems on their hand.

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Improving Mobile Payments With Real time Spark

Improving Mobile Payments With Real time Spark

Improving Mobile Payments With Real time Spark

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Running R on AWS Lambda by Ana-Maria Niculescu

Running R on AWS Lambda by Ana-Maria Niculescu

Running R on AWS Lambda by Ana-Maria Niculescu

Paris Women in Machine Learning and Data Science

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

HostedbyConfluent

Master thesis

Fabio Arcidiacono

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Librecon 2016 bilbao: kappa architecture IoT of the cars

Librecon 2016 bilbao: kappa architecture IoT of the cars

Librecon 2016 bilbao: kappa architecture IoT of the cars

Juantomás García Molina

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

DataWorks Summit

A Tool For Big Data Analysis using Apache Spark

A Tool For Big Data Analysis using Apache Spark

A Tool For Big Data Analysis using Apache Spark

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

La actualidad más candente (20)

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Putting the Spark into Functional Fashion Tech Analystics

Putting the Spark into Functional Fashion Tech Analystics

Putting the Spark into Functional Fashion Tech Analystics

Apache Flink

Elastic Stack Basic - All The Capabilities in 6.3!

Elastic Stack Basic - All The Capabilities in 6.3!

Elastic Stack Basic - All The Capabilities in 6.3!

Session 03 data_migration_at_scale_by_sameer

Session 03 data_migration_at_scale_by_sameer

Session 03 data_migration_at_scale_by_sameer

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...

Building scalable data with kafka and spark

Building scalable data with kafka and spark

Building scalable data with kafka and spark

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Improving Mobile Payments With Real time Spark

Improving Mobile Payments With Real time Spark

Improving Mobile Payments With Real time Spark

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Running R on AWS Lambda by Ana-Maria Niculescu

Running R on AWS Lambda by Ana-Maria Niculescu

Running R on AWS Lambda by Ana-Maria Niculescu

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Master thesis

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Librecon 2016 bilbao: kappa architecture IoT of the cars

Librecon 2016 bilbao: kappa architecture IoT of the cars

Librecon 2016 bilbao: kappa architecture IoT of the cars

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

Building a Self-Service Hadoop Platform at Linkedin with Azkaban

A Tool For Big Data Analysis using Apache Spark

A Tool For Big Data Analysis using Apache Spark

A Tool For Big Data Analysis using Apache Spark

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Apache Flink @ Alibaba - Seattle Apache Flink Meetup

Destacado

From Config Management Sucks to #cfgmgmtlove

From Config Management Sucks to #cfgmgmtlove

From Config Management Sucks to #cfgmgmtlove

Mesoscon 2015

This is a presentation given by Amit Goel, Founder & CEO, Patterbuzz at Open Source India 2014 conference in Bangalore. . He spoke about the story of Patterbuzz and its journey of 2 years since its beginning to explain the product development with the pace of a start up. It covered all the necessary evils and building the technology form scratch using open source softwares and available technologies and how to leverage its advantage. Patterbuzz builds on the core philosophy of unbundling content from the publishing industry and making it available to consumers in a device reading friendly format with pay as you go model. For example: users can buy single articles if they like rather than purchasing the complete magazine. Currently, Patterbuzz enables magazines from India / Indian publishing industry. Most popular magazines available are Popular Science, Mother & Baby, Electronics for you, open source for you, computer active, Gfiles, Power Politics, Hard News, Car India, Bike India etc...

Building Product from ground up using Open Source Technologies

Building Product from ground up using Open Source Technologies

Building Product from ground up using Open Source Technologies

Data science team, a practice to setup

Data science team, a practice to setup

Data science team, a practice to setup

Send that (damn) elevator down !

Send that (damn) elevator down !

Send that (damn) elevator down !

Introducing ELK

Introducing ELK

Introducing ELK

AllBits BVBA (freelancer)

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Surasak Sanguanpong

Monitoring with ElasticSearch

Monitoring with ElasticSearch

Monitoring with ElasticSearch

Elastic Stackにハマった話

Elastic Stackにハマった話

Elastic Stackにハマった話

Kazuhiro Kosaka

Monitoring using Open source technologies

Monitoring using Open source technologies

Monitoring using Open source technologies

UTKARSH BHATNAGAR

The Rise of Real Time

The Rise of Real Time

The Rise of Real Time

Keystone - Leverage Big Data 2016

Keystone - Leverage Big Data 2016

Keystone - Leverage Big Data 2016

How Did BuzzFeed Harvest One Million Email Subscribers?

How Did BuzzFeed Harvest One Million Email Subscribers?

How Did BuzzFeed Harvest One Million Email Subscribers?

Wildcard Digital Inc

Slides for our solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

A presentation about the deployment of an ELK stack at bol.com At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure. The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling. These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.

Scaling an ELK stack at bol.com

Scaling an ELK stack at bol.com

Scaling an ELK stack at bol.com

Building an IoT Kafka Pipeline in Under 5 Minutes

Building an IoT Kafka Pipeline in Under 5 Minutes

Building an IoT Kafka Pipeline in Under 5 Minutes

Amazon Web Services gives you fast access to flexible and low cost IT resources, so you can rapidly scale and build virtually any big data application including data warehousing, clickstream analytics, fraud detection, recommendation engines, event-driven ETL, serverless computing, and internet-of-things processing regardless of volume, velocity, and variety of data. https://aws.amazon.com/webinars/anz-webinar-series/

Building a Modern Data Architecture on AWS - Webinar

Building a Modern Data Architecture on AWS - Webinar

Building a Modern Data Architecture on AWS - Webinar

Amazon Web Services

ELK at LinkedIn - Kafka, scaling, lessons learned

ELK at LinkedIn - Kafka, scaling, lessons learned

ELK at LinkedIn - Kafka, scaling, lessons learned

BDX 2016- Monal daxini @ Netflix

BDX 2016- Monal daxini @ Netflix

BDX 2016- Monal daxini @ Netflix

Destacado (20)

From Config Management Sucks to #cfgmgmtlove

From Config Management Sucks to #cfgmgmtlove

From Config Management Sucks to #cfgmgmtlove

Mesoscon 2015

Building Product from ground up using Open Source Technologies

Building Product from ground up using Open Source Technologies

Building Product from ground up using Open Source Technologies

Data science team, a practice to setup

Data science team, a practice to setup

Data science team, a practice to setup

Send that (damn) elevator down !

Send that (damn) elevator down !

Send that (damn) elevator down !

Introducing ELK

Introducing ELK

Introducing ELK

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Experiences in ELK with D3.js for Large Log Analysis and Visualization

Monitoring with ElasticSearch

Monitoring with ElasticSearch

Monitoring with ElasticSearch

Elastic Stackにハマった話

Elastic Stackにハマった話

Elastic Stackにハマった話

Monitoring using Open source technologies

Monitoring using Open source technologies

Monitoring using Open source technologies

The Rise of Real Time

The Rise of Real Time

The Rise of Real Time

Keystone - Leverage Big Data 2016

Keystone - Leverage Big Data 2016

Keystone - Leverage Big Data 2016

How Did BuzzFeed Harvest One Million Email Subscribers?

How Did BuzzFeed Harvest One Million Email Subscribers?

How Did BuzzFeed Harvest One Million Email Subscribers?

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra

Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Scaling an ELK stack at bol.com

Scaling an ELK stack at bol.com

Scaling an ELK stack at bol.com

Building an IoT Kafka Pipeline in Under 5 Minutes

Building an IoT Kafka Pipeline in Under 5 Minutes

Building an IoT Kafka Pipeline in Under 5 Minutes

Building a Modern Data Architecture on AWS - Webinar

Building a Modern Data Architecture on AWS - Webinar

Building a Modern Data Architecture on AWS - Webinar

ELK at LinkedIn - Kafka, scaling, lessons learned

ELK at LinkedIn - Kafka, scaling, lessons learned

ELK at LinkedIn - Kafka, scaling, lessons learned

BDX 2016- Monal daxini @ Netflix

BDX 2016- Monal daxini @ Netflix

BDX 2016- Monal daxini @ Netflix

Similar a Rootconf

At NMC (Nielsen Marketing Cloud) we provide our customers (marketers and publishers) real-time analytics tools to profile their target audiences. To achieve that, we need to ingest billions of events per day into our big data stores, and we need to do it in a scalable yet cost-efficient manner. In this session, we will discuss how we continuously transform our data infrastructure to support these goals. Specifically, we will review how we went from CSV files and standalone Java applications all the way to multiple Kafka and Spark clusters, performing a mixture of Streaming and Batch ETLs, and supporting 10x data growth We will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty). We will present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services’ costs. Topics include: Kafka and Spark Streaming for stateless and stateful use-cases Spark Structured Streaming as a possible alternative Combining Spark Streaming with batch ETLs ”Streaming” over Data Lake using Kafka

Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Introduction to apache kafka

Introduction to apache kafka

Introduction to apache kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Allen (Xiaozhong) Wang

theScore is one of the leading sports news and media platforms in North America. With the recent growth of sports betting, theScore is diving in head-first. Province and state regulations often enforce strict data-locality requirements often prohibiting the use of centralized hosted data storage solutions. In this talk, we'll learn how theScore built Datadex, a geographically distributed system for low-latency queries and real-time updates. We'll touch on the architecture of the system (Aggregator Leaf Tailer), the underlying technologies (RocksDB, Kafka, and the JVM), as well as some nitty-gritty tips for optimizing the JVM and RocksDB for high throughput.

Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency

Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency

Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency

Activity feeds (and more) at mate1

Activity feeds (and more) at mate1

Activity feeds (and more) at mate1

Hisham Mardam-Bey

At NMC (Nielsen Marketing Cloud) we provide our customers (marketers and publishers) real-time analytics tools to profile their target audiences. To achieve that, we need to ingest billions of events per day into our big data stores, and we need to do it in a scalable yet cost-efficient manner. In this session, we will discuss how we continuously transform our data infrastructure to support these goals. Specifically, we will review how we went from CSV files and standalone Java applications all the way to multiple Kafka and Spark clusters, performing a mixture of Streaming and Batch ETLs, and supporting 10x data growth. We will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty...). We will present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services' costs. Topics include : * Kafka and Spark Streaming for stateless and stateful use-cases * Spark Structured Streaming as a possible alternative * Combining Spark Streaming with batch ETLs * "Streaming" over Data Lake using Kafka

Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

DataWorks Summit

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Garindra Prahandono

Apache Spark vs Apache Flink

Apache Spark vs Apache Flink

Apache Spark vs Apache Flink

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

How Uber scaled its Real Time Infrastructure to Trillion events per day

How Uber scaled its Real Time Infrastructure to Trillion events per day

How Uber scaled its Real Time Infrastructure to Trillion events per day

DataWorks Summit

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 200M messages per second. In this talk, we will share the learning and challenges in our journey moving to Apache Kafka.

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

HostedbyConfluent

Timothy Spann: Apache Pulsar for ML

Timothy Spann: Apache Pulsar for ML

Timothy Spann: Apache Pulsar for ML

Event driven architectures with Kinesis

Event driven architectures with Kinesis

Event driven architectures with Kinesis

The Fn Project: A Quick Introduction (December 2017)

The Fn Project: A Quick Introduction (December 2017)

The Fn Project: A Quick Introduction (December 2017)

Oracle Developers

Centralized Logging System Using ELK Stack

Centralized Logging System Using ELK Stack

Centralized Logging System Using ELK Stack

2015 03-16-elk at-bsides

2015 03-16-elk at-bsides

2015 03-16-elk at-bsides

Similar a Rootconf (20)

Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Introduction to apache kafka

Introduction to apache kafka

Introduction to apache kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Netflix Data Pipeline With Kafka

Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency

Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency

Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency

Activity feeds (and more) at mate1

Activity feeds (and more) at mate1

Activity feeds (and more) at mate1

Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Laskar: High-Velocity GraphQL & Lambda-based Software Development Model

Apache Spark vs Apache Flink

Apache Spark vs Apache Flink

Apache Spark vs Apache Flink

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

How Uber scaled its Real Time Infrastructure to Trillion events per day

How Uber scaled its Real Time Infrastructure to Trillion events per day

How Uber scaled its Real Time Infrastructure to Trillion events per day

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

Timothy Spann: Apache Pulsar for ML

Timothy Spann: Apache Pulsar for ML

Timothy Spann: Apache Pulsar for ML

Event driven architectures with Kinesis

Event driven architectures with Kinesis

Event driven architectures with Kinesis

The Fn Project: A Quick Introduction (December 2017)

The Fn Project: A Quick Introduction (December 2017)

The Fn Project: A Quick Introduction (December 2017)

Centralized Logging System Using ELK Stack

Centralized Logging System Using ELK Stack

Centralized Logging System Using ELK Stack

2015 03-16-elk at-bsides

2015 03-16-elk at-bsides

2015 03-16-elk at-bsides

Último

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Último (20)

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

🐬 The future of MySQL is Postgres 🐘

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Rootconf

1. How LinkedIn is leveraging kafka for log aggregation and monitoring Akbar KM, Muzammil Rehman SRE Linkedin

2. Agenda ● What is Pub Sub? ● Kafka basics ● Kafka for monitoring ● inGraph’s ● ELK at Linkedin

3. What is pub sub ? Producer Consumer Producer Consumer Topic 1 Topic 2 Topic 3 subscribe publish(topic, msg) Publish subscribe system msg msg

5. Consumer Groups

6. inGraph ● inGraphs is a visualization frontend for a linkedin’s metrics ● Agent is a library used by all java application ● Data is transported over kafka ● Collector consume and write to rrd ● Features ○ Rest based ○ Self service ○ Little over 500,000 metrics collected per minute ○ Each RRD is roughly 815k. Each RRD is written to two collectors to maintain data integrity ○ We currently have 1.4TB of SSD in production colo

7. ELK at linkedin ● ELK (Elasticsearch, Logstash, Kibana) ○ E -> Near Real-Time Search & Analysis ○ L -> Plumbing and Glue for all your data ○ K -> Line graphs, pie charts, dashboard ● Logstash kafka plugin consumes logs and end up written into elasticsearch

8. Why Kafka 1. Near real time delivery 2. Supports multiple consumers groups aka queuing 3. No overhead on clients that comes up logstash agent 4. Kafka scales horizontally 5. Supports REST out of the box 6. In house support

9. Reference http://engineering.linkedin.com/52/autometrics- self-service-metrics-collection ELK http://www.elastic.co/guide/