Robust Stream Processing with Apache Flink

•

5 recomendaciones•2,320 vistas

The document discusses Apache Flink, an open source platform for distributed stream and batch data processing. It describes how Flink allows for stateful stream processing, windowed computations over streams, and robust handling of time, failures, planned downtimes, and reprocessing. The presentation provides an overview of these concepts and includes a code example and details on Flink's distributed and parallel deployment architecture.

Datos y análisis

Robust
Stream Processing
with
Apache Flink
Jamie Grier
@jamiegrier
jamie@data-artisans.com

Who am I?
• Director of Applications Engineering at data
Artisans
• Previously working on streaming computation at
Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for
about a decade
• High-speed video, social media streaming, general
frameworks for stream processing

Overview
• What is Apache Flink?
• What is Stateful Stream Processing?
• Windowed computation over streams
• Robust Time Handling (Event Time vs Processing Time)
• Robust Failure Handling
• Robust Planned Downtime Handling
• Robust Reprocessing

What is
Apache Flink?
Apache Flink is an open source platform for distributed
stream and batch data processing.

Stream Processing
Your
Code
Data Stream Data Stream

Stateful
Stream Processing
Your
Code
Data Stream Data Stream
State

More Complex
Example
Kafka
Files
Rabb
itMQ
Filter
Map
Join /
Sum
Influx
DB
C*

Distributed and Parallel
Deployment
Kafka
Files
Rabb
it
MQ
Filter
Parse
Join /
Sum
Influx
DB
C*

Robust Stream Processing
with Apache Flink

Windowing in Processing
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Processing Time
Event Time

Windowing in Event
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Event Time

We’re Hiring!
http://data-artisans.com/careers

Flink Forward 2016, Berlin
Submission deadline: June 30, 2016
Early bird deadline: July 15, 2016
www.flink-forward.org

Más contenido relacionado

La actualidad más candente

Big Data WarsawMaximilian Michels

Streaming in the Wild with Apache FlinkKostas Tzoumas

Ted Dunning-Faster and Furiouser- Flink DriftFlink Forward

Flink 1.0-slidesJamie Grier

data Artisans Product AnnouncementFlink Forward

Robust Stream Processing With Apache FlinkJamie Grier

Stream Processing with Apache FlinkC4Media

Flink. Pure StreamingIndizen Technologies

Extending the Yahoo Streaming Benchmark + MapR BenchmarksJamie Grier

The Evolution of (Open Source) Data ProcessingAljoscha Krettek

Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache ZeppelinFlink Forward

Introduction to Stream Processing with Apache Flink (2019-11-02 Bengaluru Mee...Timo Walther

Apache Flink 101 - the rise of stream processing and beyondBowen Li

A look at Flink 1.2Stefan Richter

Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...Ververica

Abstractions for managed stream processing platform (Arya Ketan - Flipkart)KafkaZone

Bay Area Apache Flink Meetup Community Update August 2015Henry Saputra

The Past, Present, and Future of Apache Flink®Aljoscha Krettek

Building a real time Tweet map with Flink in six weeksMatthias Kricke

Apache Flink and what it is used forAljoscha Krettek

La actualidad más candente (20)

Big Data Warsaw

Streaming in the Wild with Apache Flink

Ted Dunning-Faster and Furiouser- Flink Drift

Flink 1.0-slides

data Artisans Product Announcement

Robust Stream Processing With Apache Flink

Stream Processing with Apache Flink

Flink. Pure Streaming

Extending the Yahoo Streaming Benchmark + MapR Benchmarks

The Evolution of (Open Source) Data Processing

Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin

Introduction to Stream Processing with Apache Flink (2019-11-02 Bengaluru Mee...

Apache Flink 101 - the rise of stream processing and beyond

A look at Flink 1.2

Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...

Abstractions for managed stream processing platform (Arya Ketan - Flipkart)

Bay Area Apache Flink Meetup Community Update August 2015

The Past, Present, and Future of Apache Flink®

Building a real time Tweet map with Flink in six weeks

Apache Flink and what it is used for

Similar a Robust Stream Processing with Apache Flink

Robust Stream Processing with Apache FlinkJamie Grier

Robust stream processing with Apache FlinkAljoscha Krettek

Trend Micro Big Data Platform and Apache BigtopEvans Ye

Big Data Analytics Platforms by KTH and RISE SICSBig Data Value Association

Flink in actionArtem Semenenko

Near real-time anomaly detection at Lyftmarkgrover

A Practical Guide to Selecting a Stream Processing Technology confluent

Cloud lunch and learn real-time streaming in azureTimothy Spann

Data Stream Processing with Apache FlinkFabian Hueske

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...ScyllaDB

Streaming in the Wild with Apache FlinkDataWorks Summit/Hadoop Summit

Stinger.Next by Alan Gates of HortonworksData Con LA

Hail hydrate! from stream to lake using open sourceTimothy Spann

Berlin Apache Flink Meetup May 2015, Community UpdateRobert Metzger

The Purpose of Cloud-technology for Video ProductionMaarten Verwaest

Flink forward-2017-netflix keystones-paasMonal Daxini

Reintroducing the Stream Processor: A universal tool for continuous data anal...Paris Carbone

Getting started with apache flink streaming apiPreetdeep Kumar

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.Data Con LA

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)Apache Flink Taiwan User Group

Similar a Robust Stream Processing with Apache Flink (20)

Robust Stream Processing with Apache Flink

Robust stream processing with Apache Flink

Trend Micro Big Data Platform and Apache Bigtop

Big Data Analytics Platforms by KTH and RISE SICS

Flink in action

Near real-time anomaly detection at Lyft

A Practical Guide to Selecting a Stream Processing Technology

Cloud lunch and learn real-time streaming in azure

Data Stream Processing with Apache Flink

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...

Streaming in the Wild with Apache Flink

Stinger.Next by Alan Gates of Hortonworks

Hail hydrate! from stream to lake using open source

Berlin Apache Flink Meetup May 2015, Community Update

The Purpose of Cloud-technology for Video Production

Flink forward-2017-netflix keystones-paas

Reintroducing the Stream Processor: A universal tool for continuous data anal...

Getting started with apache flink streaming api

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)

Más de Flink Forward

Building a fully managed stream processing platform on Flink at scale for Lin...Flink Forward

Evening out the uneven: dealing with skew in FlinkFlink Forward

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...Flink Forward

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Flink Forward

Introducing the Apache Flink Kubernetes OperatorFlink Forward

Autoscaling Flink with Reactive ModeFlink Forward

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

One sink to rule them all: Introducing the new Async SinkFlink Forward

Tuning Apache Kafka Connectors for Flink.pptxFlink Forward

Flink powered stream processing platform at PinterestFlink Forward

Apache Flink in the Cloud-Native EraFlink Forward

Where is my bottleneck? Performance troubleshooting in FlinkFlink Forward

Using the New Apache Flink Kubernetes Operator in a Production DeploymentFlink Forward

The Current State of Table API in 2022Flink Forward

Flink SQL on Pulsar made easyFlink Forward

Dynamic Rule-based Real-time Market Data AlertsFlink Forward

Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward

Processing Semantically-Ordered Streams in Financial ServicesFlink Forward

Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward

Batch Processing at Scale with Flink & IcebergFlink Forward

Más de Flink Forward (20)

Building a fully managed stream processing platform on Flink at scale for Lin...

Evening out the uneven: dealing with skew in Flink

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing the Apache Flink Kubernetes Operator

Autoscaling Flink with Reactive Mode

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

One sink to rule them all: Introducing the new Async Sink

Tuning Apache Kafka Connectors for Flink.pptx

Flink powered stream processing platform at Pinterest

Apache Flink in the Cloud-Native Era

Where is my bottleneck? Performance troubleshooting in Flink

Using the New Apache Flink Kubernetes Operator in a Production Deployment

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Dynamic Rule-based Real-time Market Data Alerts

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Processing Semantically-Ordered Streams in Financial Services

Tame the small files problem and optimize data layout for streaming ingestion...

Batch Processing at Scale with Flink & Iceberg

Último

BPAC WITH UFSBI GENERAL PRESENTATION 18_05_2017-1.pptxMohammedJunaid861692

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAroojKhan71

꧁❤ Greater Noida Call Girls Delhi ❤꧂ 9711199171 ☎️ Hard And Sexy Vip Callshivangimorya083

Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh9953056974 Low Rate Call Girls In Saket, Delhi NCR

Midocean dropshipping via API with DroFxolyaivanovalion

Invezz.com - Grow your wealth with trading signalsInvezz1

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 nightDelhi Call girls

Schema on read is obsolete. Welcome metaprogramming..pdfLars Albertsson

Log Analysis using OSSEC sasoasasasas.pptxJohnnyPlasten

CALL ON ➥8923113531 🔝Call Girls Chinhat Lucknow best sexual service Onlineanilsa9823

Introduction-to-Machine-Learning (1).pptxfirstjob4

{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...Pooja Nehwal

100-Concepts-of-AI by Anupama Kate .pptxAnupama Kate

Carero dropshipping via API with DroFx.pptxolyaivanovalion

CebaBaby dropshipping via API with DroFX.pptxolyaivanovalion

FESE Capital Markets Fact Sheet 2024 Q1.pdfMarinCaroMartnezBerg

(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Serviceranjana rawat

Halmar dropshipping via API with DroFxolyaivanovalion

Edukaciniai dropshipping via API with DroFxolyaivanovalion

Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfRachmat Ramadhan H

Robust Stream Processing with Apache Flink

1. Robust Stream Processing with Apache Flink Jamie Grier @jamiegrier jamie@data-artisans.com

2. Who am I? • Director of Applications Engineering at data Artisans • Previously working on streaming computation at Twitter, Gnip and Boulder Imaging • Involved in various kinds of stream processing for about a decade • High-speed video, social media streaming, general frameworks for stream processing

3. Overview • What is Apache Flink? • What is Stateful Stream Processing? • Windowed computation over streams • Robust Time Handling (Event Time vs Processing Time) • Robust Failure Handling • Robust Planned Downtime Handling • Robust Reprocessing

4. What is Apache Flink? Apache Flink is an open source platform for distributed stream and batch data processing.

5. What is Apache Flink?

6. Stream Processing Your Code Data Stream Data Stream

7. Stateful Stream Processing Your Code Data Stream Data Stream State

8. More Complex Example Kafka Files Rabb itMQ Filter Map Join / Sum Influx DB C*

9. Distributed and Parallel Deployment Kafka Files Rabb it MQ Filter Parse Join / Sum Influx DB C*

10. Robust Stream Processing with Apache Flink

11. Code Example!

12. Windowing

13. Processing Time vs Event Time

14. Windowing in Processing Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Processing Time Event Time

15. Windowing in Event Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Event Time