Stream Processing: Choosing the Right Tool for the Job

•

3 recomendaciones•1,545 vistas

Due to the increasing interest in real-time processing, many stream processing frameworks were developed. However, no clear guidelines have been established for choosing a framework for a specific use case. In this talk, two different scenarios are taken and the audience is guided through the thought process and questions that one should ask oneself when choosing the right tool. The stream processing frameworks that will be discussed are Spark Streaming, Structured Streaming, Flink and Kafka Streams. The main questions are: How much data does it need to process? (throughput) Does it need to be fast? (latency) Who will build it? (supported languages, level of API, SQL capabilities, built-in windowing and joining functionalities, etc) Is accurate ordering important? (event time vs. processing time) Is there a batch component? (integration of batch API) How do we want it to run? (deployment options: standalone, YARN, mesos, …) How much state do we have? (state store options) – What if a message gets lost? (message delivery guarantees, checkpointing). For each of these questions, we look at how each framework tackles this and what the main differences are. The content is based on the PhD research of Giselle van Dongen in benchmarking stream processing frameworks in several scenarios using latency, throughput and resource utilization.

Datos y análisis

WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics

Giselle van Dongen,
Stream processing:
choosing the right tool for
the job
#UnifiedDataAnalytics #SparkAISummit

3#UnifiedDataAnalytics #SparkAISummit
●
●
○
○

Context
4#UnifiedDataAnalytics #SparkAISummit
…

Context
5#UnifiedDataAnalytics #SparkAISummit

Context
6#UnifiedDataAnalytics #SparkAISummit

Disclaimer
7#UnifiedDataAnalytics #SparkAISummit
a.k.a I will not pick a stream processing framework for you

Commonalities
8#UnifiedDataAnalytics #SparkAISummit
●
●
●
●

Imagine...
9#UnifiedDataAnalytics #SparkAISummit

10#UnifiedDataAnalytics #SparkAISummit

11#UnifiedDataAnalytics #SparkAISummit

Do we need stream processing?
12#UnifiedDataAnalytics #SparkAISummit
…

Do we need stream processing?
13#UnifiedDataAnalytics #SparkAISummit
…

14#UnifiedDataAnalytics #SparkAISummit

15#UnifiedDataAnalytics #SparkAISummit

16#UnifiedDataAnalytics #SparkAISummit

How much data?
➔
➔
17#UnifiedDataAnalytics #SparkAISummit

How much data?
18#UnifiedDataAnalytics #SparkAISummit

Spark Flink Kafka Spark Struct Flink KafkaStruct

20#UnifiedDataAnalytics #SparkAISummit

Does it need to be fast?
21#UnifiedDataAnalytics #SparkAISummit
➔ ➔

Does it need to be fast?
22#UnifiedDataAnalytics #SparkAISummit

Does it need to be fast?
●
…
●
●
23#UnifiedDataAnalytics #SparkAISummit

Event-driven
24#UnifiedDataAnalytics #SparkAISummit
https://www.cakesolutions.net/teamblogs/comparison-of-apache-stream-processing-frameworks-part-1

Micro-batching
25#UnifiedDataAnalytics #SparkAISummit
https://www.cakesolutions.net/teamblogs/comparison-of-apache-stream-processing-frameworks-part-1

Does it need to be fast?
●
26#UnifiedDataAnalytics #SparkAISummit
➔
➔

Spark Flink Kafka Spark Struct Flink KafkaStruct

28#UnifiedDataAnalytics #SparkAISummit

29#UnifiedDataAnalytics #SparkAISummit
Performance
Advanced features
Deployment & Internals

Who will build it?
30#UnifiedDataAnalytics #SparkAISummit

Who will build it?
31#UnifiedDataAnalytics #SparkAISummit

Spark Flink Kafka Spark Struct Flink KafkaStruct

33#UnifiedDataAnalytics #SparkAISummit
Performance
Advanced features
Deployment & Internals

Is accurate ordering important?
34#UnifiedDataAnalytics #SparkAISummit
4 7 10
window 1 window 2 window 3
8

Is accurate ordering important?
35#UnifiedDataAnalytics #SparkAISummit
4 7 10
window 1 window 2 window 3
88

36#UnifiedDataAnalytics #SparkAISummit
4 7 10
window 1 window 2 window 3
88
window 1
window 2
window 3
Is accurate ordering important?

Is accurate ordering important?
37#UnifiedDataAnalytics #SparkAISummit

Is accurate ordering important?
38#UnifiedDataAnalytics #SparkAISummit
➔ ➔

Spark Flink Kafka Spark Struct Flink KafkaStruct

40#UnifiedDataAnalytics #SparkAISummit
Performance
Advanced features
Deployment & Internals

What is the ecosystem like?
•
•
•
41#UnifiedDataAnalytics #SparkAISummit

What is the ecosystem like?
42#UnifiedDataAnalytics #SparkAISummit

Spark Flink Kafka Spark Struct Flink KafkaStruct

44#UnifiedDataAnalytics #SparkAISummit

45#UnifiedDataAnalytics #SparkAISummit

How do we want it to run?
46#UnifiedDataAnalytics #SparkAISummit
W W W W W
M
T T T T T

Spark Flink Kafka Spark Struct Flink KafkaStruct

48#UnifiedDataAnalytics #SparkAISummit

What if a message gets lost?
49#UnifiedDataAnalytics #SparkAISummit

What if a message gets lost?
•
•
•
50#UnifiedDataAnalytics #SparkAISummit

What if a message gets lost?
•
•

W1 W3 W4
W2 W4
What if a message gets lost?
52#UnifiedDataAnalytics #SparkAISummit
State
State
State
State
State
HDFS
ref. Flink Forward 2018 Best practices for state and time, Tzu-Li Tai

W1 W3 W4
W2 W4
What if a message gets lost?
53#UnifiedDataAnalytics #SparkAISummit
State
State
State
State
State
HDFS
ref. Flink Forward 2018 Best practices for state and time, Tzu-Li Tai

W1 W4
W2 W4
W3
What if a message gets lost?
54#UnifiedDataAnalytics #SparkAISummit
State
State
State
State
State
HDFS
ref. Flink Forward 2018 Best practices for state and time, Tzu-Li Tai

What if a message gets lost?
●
○
○
○
●
○
○
○
○

What if a message gets lost?
56#UnifiedDataAnalytics #SparkAISummit

What if a message gets lost?
57#UnifiedDataAnalytics #SparkAISummit

Spark Flink Kafka Spark Struct Flink KafkaStruct

Spark Flink Kafka SparkStruct Struct Flink Kafka

60#UnifiedDataAnalytics #SparkAISummit

61#UnifiedDataAnalytics #SparkAISummit

Want to know more?
62#UnifiedDataAnalytics #SparkAISummit

63#UnifiedDataAnalytics #SparkAISummit
THANK YOU!
Do you want to work with these tools?
We are hiring!

DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT

Más contenido relacionado

La actualidad más candente

Netflix Studio spent 8 Billion dollars on content in 2018. When the stakes are so high, it is paramount to track changes to the core studio metadata, spend on our content, forecasting and more to enable the business to make efficient and effective decisions. Embracing a Kappa architecture with Kafka enables us to build an enterprise grade message bus. By having event processing be the de-facto paved path for syncing core entities, it provides traceability and data quality verification as first class citizens for every change published.This talk will also get into the nuts and bolts of the eventing and stream processing paradigm and why it is the best fit for our use case, versus alternative architectures with similar benefits We will do a deep dive into the fascinating world of Netflix Studios and how eventing and stream processing are revolutionizing the world of movie productions and the production finance infrastructure.

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Ryan Blue explains how Netflix is building on Parquet to enhance its 40+ petabyte warehouse, combining Parquet’s features with Presto and Spark to boost ETL and interactive queries. Information about tuning Parquet is hard to find. Ryan shares what he’s learned, creating the missing guide you need. Topics include: * The tools and techniques Netflix uses to analyze Parquet tables * How to spot common problems * Recommendations for Parquet configuration settings to get the best performance out of your processing platform * The impact of this work in speeding up applications like Netflix’s telemetry service and A/B testing platform

Parquet performance tuning: the missing guide

Parquet performance tuning: the missing guide

Parquet performance tuning: the missing guide

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...

InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...

InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...

“In Spark 2.0, we have extended DataFrames and Datasets to handle real time streaming data. This not only provides a single programming abstraction for batch and streaming data, it also brings support for event-time based processing, out-or-order/delayed data, sessionization and tight integration with non-streaming data sources and sinks. In this talk, I will take a deep dive into the concepts and the API and show how this simplifies building complex “Continuous Applications”.” - T.D. Databricks Blog: "Structured Streaming In Apache Spark 2.0: A new high-level API for streaming" https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html // About the Presenter // Tathagata Das is an Apache Spark Committer and a member of the PMC. He’s the lead developer behind Spark Streaming, and is currently employed at Databricks. Before Databricks, you could find him at the AMPLab of UC Berkeley, researching datacenter frameworks and networks with professors Scott Shenker and Ion Stoica. Follow T.D. on - Twitter: https://twitter.com/tathadas LinkedIn: https://www.linkedin.com/in/tathadas

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data. This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink. Speaker Gregory Fee, Principal Engineer, Lyft

Bootstrapping state in Apache Flink

Bootstrapping state in Apache Flink

Bootstrapping state in Apache Flink

DataWorks Summit

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Igor Anishchenko Odessa Java TechTalks Lohika - May, 2012 Let's take a step back and compare data serialization formats, of which there are plenty. What are the key differences between Apache Thrift, Google Protocol Buffers and Apache Avro. Which is "The Best"? Truth of the matter is, they are all very good and each has its own strong points. Hence, the answer is as much of a personal choice, as well as understanding of the historical context for each, and correctly identifying your own, individual requirements.

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Igor Anishchenko

Airflow at lyft

Airflow at lyft

Airflow at lyft

On Improving Broadcast Joins in Apache Spark SQL

On Improving Broadcast Joins in Apache Spark SQL

On Improving Broadcast Joins in Apache Spark SQL

Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. With a data warehouse at this scale, it is a constant challenge to keep improving performance. This talk will focus on Iceberg, a new table metadata format that is designed for managing huge tables backed by S3 storage. Iceberg decreases job planning time from minutes to under a second, while also isolating reads from writes to guarantee jobs always use consistent table snapshots. In this session, you'll learn: • Some background about big data at Netflix • Why Iceberg is needed and the drawbacks of the current tables used by Spark and Hive • How Iceberg maintains table metadata to make queries fast and reliable • The benefits of Iceberg's design and how it is changing the way Netflix manages its data warehouse • How you can get started using Iceberg Speaker Ryan Blue, Software Engineer, Netflix

Iceberg: a fast table format for S3

Iceberg: a fast table format for S3

Iceberg: a fast table format for S3

DataWorks Summit

Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.

Deep Dive: Memory Management in Apache Spark

Deep Dive: Memory Management in Apache Spark

Deep Dive: Memory Management in Apache Spark

kafka

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Noritaka Sekiyama

Introduction to Kafka Cruise Control

Introduction to Kafka Cruise Control

Introduction to Kafka Cruise Control

Flink Forward San Francisco 2022. Apache Flink and Delta Lake together allow you to build the foundation for your data lakehouses by ensuring the reliability of your concurrent streams from processing to the underlying cloud object-store. Together, the Flink/Delta Connector enables you to store data in Delta tables such that you harness Delta’s reliability by providing ACID transactions and scalability while maintaining Flink’s end-to-end exactly-once processing. This ensures that the data from Flink is written to Delta Tables in an idempotent manner such that even if the Flink pipeline is restarted from its checkpoint information, the pipeline will guarantee no data is lost or duplicated thus preserving the exactly-once semantics of Flink. by Scott Sandre & Denny Lee

Building Reliable Lakehouses with Apache Flink and Delta Lake

Building Reliable Lakehouses with Apache Flink and Delta Lake

Building Reliable Lakehouses with Apache Flink and Delta Lake

Apache Flink is the foundation for Decodable's real-time SaaS data platform. Flink runs critical data processing jobs with strong security requirements. In addition, Decodable has to scale to thousands of tenants, power various use cases, provide an intuitive user experience and maintain cost-efficiency. We've learned a lot of lessons while building and maintaining the platform. In this talk, I'll share the top 3 toughest challenges building and operating this platform with Flink, and how we solved them.

The top 3 challenges running multi-tenant Flink at scale

The top 3 challenges running multi-tenant Flink at scale

The top 3 challenges running multi-tenant Flink at scale

Introduction to Apache Kafka

Introduction to Apache Kafka

Introduction to Apache Kafka

La actualidad más candente (20)

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Eventing Things - A Netflix Original! (Nitin Sharma, Netflix) Kafka Summit SF...

Parquet performance tuning: the missing guide

Parquet performance tuning: the missing guide

Parquet performance tuning: the missing guide

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

Apache Kafka - Martin Podval

InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...

InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...

InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxD...

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Bootstrapping state in Apache Flink

Bootstrapping state in Apache Flink

Bootstrapping state in Apache Flink

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Airflow at lyft

Airflow at lyft

Airflow at lyft

On Improving Broadcast Joins in Apache Spark SQL

On Improving Broadcast Joins in Apache Spark SQL

On Improving Broadcast Joins in Apache Spark SQL

Iceberg: a fast table format for S3

Iceberg: a fast table format for S3

Iceberg: a fast table format for S3

Deep Dive: Memory Management in Apache Spark

Deep Dive: Memory Management in Apache Spark

Deep Dive: Memory Management in Apache Spark

kafka

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud

Introduction to Kafka Cruise Control

Introduction to Kafka Cruise Control

Introduction to Kafka Cruise Control

Building Reliable Lakehouses with Apache Flink and Delta Lake

Building Reliable Lakehouses with Apache Flink and Delta Lake

Building Reliable Lakehouses with Apache Flink and Delta Lake

The top 3 challenges running multi-tenant Flink at scale

The top 3 challenges running multi-tenant Flink at scale

The top 3 challenges running multi-tenant Flink at scale

Introduction to Apache Kafka

Introduction to Apache Kafka

Introduction to Apache Kafka

Similar a Stream Processing: Choosing the Right Tool for the Job

"Databricks Unified Analytics Platform (UAP) is a cloud-based service for running all analytics in one place - from highly reliable and performant data pipelines to state-of-the-art Machine Learning. From the original creators of Apache Spark and MLflow, it provides data science and engineering teams ready to use pre-packaged clusters with optimized Apache Spark and various ML frameworks coupled with powerful collaboration capabilities to improve productivity across the ML lifecycle. Yada yada yada... But in addition to being a vendor Databricks is also a user of UAP. So, what have we learned by eating our own dogfood? Attend a “from the trenches report” from Suraj Acharya, Director Engineering responsible for Databricks’ in-house data engineering team how his team put Databricks technology to use, the lessons they have learned along the way and best practices for using Databricks for data engineering. "

Databricks: What We Have Learned by Eating Our Dog Food

Databricks: What We Have Learned by Eating Our Dog Food

Databricks: What We Have Learned by Eating Our Dog Food

In our experience, many problems with production workflows can be traced back to unexpected values in the input data. In a complex pipeline, it can be difficult and costly to trace the root cause of errors. Here we outline our work developing an open source data validation framework built on Apache Spark. Our goal is a tool that easily integrates into existing workflows to automatically make data validation a vital initial step of every production workflow. Our tool is aimed at data scientists and data engineers, who are not necessarily Scala/Python programmers. Our users specify a configuration file that details the data validation checks to be completed. This configuration file is parsed into appropriate queries that are executed with Apache Spark. A status report is logged, which is used to notify developers/maintainers and to establish a historical record of validator checks. This work was inspired by the many great ideas behind Google's TensorFlow Extended (TFX) platform, in particular TensorFlow Data Validation (TFDV). As such we provide optional functionality for our users to visualize their data using Facets Overview and Facets Dive.

Apache Spark Data Validation

Apache Spark Data Validation

Apache Spark Data Validation

Hybrid Transactional/Analytics Processing with Spark and IMDGs

Hybrid Transactional/Analytics Processing with Spark and IMDGs

Hybrid Transactional/Analytics Processing with Spark and IMDGs

How do we get better than good enough? Leveraging NLP techniques, we can determine the general sentiment of a sentence, phrase, or a paragraph of text. We can mine the world of social data to get a sense of what is being said. But, how do you get control of the factors that create happiness? How do you become proactive in making end-users happy? Chatbots, human chats, and conversations are the means we are using to express our ideas to each other. NLP is great for helping us process and understand this data but can fall short. In our session, we will explore how to expand NLP/sentiment analysis to investigate the intense interactions that can occur between humans and humans or humans and robots. We will show how to pinpoint the things that work to improve quality and how to use those data points to measure the effectiveness of chatbots. Learn how we have applied popular NLP frameworks such as NLTK, Stanford CoreNLP and John Snow Labs NLP to financial customer service data. Explore techniques to analyze conversations for actionable insights. Leave with an understanding of how to influence your customers' happiness.

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and...

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and...

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and...

Timely data in a data warehouse is a challenge many of us face, often with there being no straightforward solution. Using a combination of batch and streaming data pipelines you can leverage the Delta Lake format to provide an enterprise data warehouse at a near real-time frequency. Delta Lake eases the ETL workload by enabling ACID transactions in a warehousing environment. Coupling this with structured streaming, you can achieve a low latency data warehouse. In this talk, we’ll talk about how to use Delta Lake to improve the latency of ingestion and storage of your data warehouse tables. We’ll also talk about how you can use spark streaming to build the aggregations and tables that drive your data warehouse.

Near Real-Time Data Warehousing with Apache Spark and Delta Lake

Near Real-Time Data Warehousing with Apache Spark and Delta Lake

Near Real-Time Data Warehousing with Apache Spark and Delta Lake

Spark's graph capabilities are great at enabling analysis of networks for use-cases such as fraud-detection, illicit network detection, and supply chain risk analysis. However, in order for a data scientist to perform analytics on a network (e.g., Page Rank, community detection, etc.), they end up spending all their time fighting a mountain of data integration challenges. A specific challenge this talk will focus on is connecting entities in a network within and across data domains. We will explore how you can leverage the Spark ecosystem's graph capabilities to perform massive-scale entity resolution (ER). As a result, your data scientists will be able to more quickly and effectively perform graph analytics that drive business and mission value. Key takeaways: 1) The Spark ecosystem enables you to quickly get started with graph analytics use-cases at scale 2) Complementing traditional ER techniques with the context of graph relationships allows you to connect entities that you could not easily connect before

Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph

Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph

Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph

>Sarah: My Spark SQL query failed. How can I fix it? >Jeeves: Your Spark query driver went out of memory. >Jeeves: You can set spark.driver.memory to 2.2GB and rerun the query to complete it successfully. Who is Jeeves? An experienced Spark developer? A seasoned administrator? No, Jeeves is a chatbot created to simplify data operations management for enterprise Spark clusters. This chatbot is powered by advanced AI algorithms and an intuitive conversational interface that together provide answers to get users in and out of performance problems quickly. Instead of just being stuck to screens displaying performance logs and metrics, users can now have more refreshing experience; and consume performance insights via a two-way conversation with their own personal Spark expert. This talk will give an overview of the chatbot, its architecture, and how it fits in a complex Spark environment. The chatbot connects to a large number of sources to get the data to power its AI algorithms. It can detect anomalies in performance and push key insights via alerts to users when they need them the most. The chatbot can also be told to take actions like creating tickets and making configuration changes. You will learn how to build chatbots that tackle your complex data operations challenges with AI algorithms and automation, keeping a cool head at all times.

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

ML.NET is an open source, machine learning framework built in .NET and runs on Windows, Linux and macOS. It allows developers to integrate custom machine learning into their applications without any prior expertise in developing or tuning machine learning models. Enhance your .NET apps with sentiment analysis, price prediction, fraud detection and more using custom models built with ML.NET In this Session, Andy will show not only the core of ML.NET but best practices around Azure Data Lake and data in general when using .NET

ITCamp 2019 - Andy Cross - Machine Learning with ML.NET and Azure Data Lake

ITCamp 2019 - Andy Cross - Machine Learning with ML.NET and Azure Data Lake

ITCamp 2019 - Andy Cross - Machine Learning with ML.NET and Azure Data Lake

Back in the old days of Apache Spark, using Python with Spark was an exercise in patience. Data was moving up and down from Python to Scala, being serialised constantly. Leveraging SparkSQL and avoiding UDFs made things better, likewise did the constant improvement of the optimisers (Catalyst and Tungsten). But, after Spark 2.3, PySpark has sped up tremendously thanks to the addition of the Arrow serialisers. In this talk you will learn how the Spark Scala core communicates with the Python processes, how data is exchanged across both sub-systems and the development efforts present and underway to make it as fast as possible.

Internals of Speeding up PySpark with Arrow

Internals of Speeding up PySpark with Arrow

Internals of Speeding up PySpark with Arrow

Scaling Face Recognition with Big Data

Scaling Face Recognition with Big Data

Scaling Face Recognition with Big Data

Deploying machine learning models seems like it should be a relatively easy task. Take your model and pass it some features in production. The reality is that the code written during the prototyping phase of model development doesn’t always work when applied at scale or on “real” data. This talk will explore 1) common problems at the intersection of data science and data engineering 2) how you can structure your code so there is minimal friction between prototyping and production, and 3) how you can use Apache Spark to run predictions on your models in batch or streaming contexts. You will take away how to address some of productionizing issues that data scientists and data engineers face while deploying machine learning models at scale and a better understanding of how to work collaboratively to minimize disparity between prototyping and productizing.

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...

Designing a streaming application which has to process data from 1 or 2 streams is easy. Any streaming framework which provides scalability, high-throughput, and fault-tolerance would work. But when the number of streams start growing in order 100s or 1000s, managing them can be daunting. How would you share resources among 1000s of streams with all of them running 24×7? Manage their state, Apply advanced streaming operations, Add/Delete streams without restarting? This talk explains common scenarios & shows techniques that can handle thousands of streams using Spark Structured Streaming.

Blue Pill/Red Pill: The Matrix of Thousands of Data Streams

Blue Pill/Red Pill: The Matrix of Thousands of Data Streams

Blue Pill/Red Pill: The Matrix of Thousands of Data Streams

At Avast, we believe everyone has the right to be safe. We are dedicated to creating a world that provides safety and privacy for all, not matter where you are, who you are, or how you connect. With over 1.5 billion attacks stopped and 30 million new executable files monthly, big data pipelines are crucial for the security of our customers. At Avast we are leveraging Apache Spark machine learning libraries and TensorflowOnSpark for a variety of tasks ranging from marketing and advertisement, through network security to malware detection. This talk will cover our main cybersecurity usecases of Spark. After describing our cluster environment we will first demonstrate anomaly detection on time series of threats. Having thousands of types of attacks and malware, AI helps human analysts select and focus on most urgent or dire threats. We will walk through our setup for distributed training of deep neural networks with Tensorflow to deploying and monitoring of a streaming anomaly detection application with trained model. Next we will show how we use Spark for analysis and clustering of malicious files and large scale experimentation to automatically process and handle changes in malware. In the end, we will give comparison to other tools we used for solving those problems.

AI on Spark for Malware Analysis and Anomalous Threat Detection

AI on Spark for Malware Analysis and Anomalous Threat Detection

AI on Spark for Malware Analysis and Anomalous Threat Detection

Scaling face recognition with big data - Bogdan Bocse

Scaling face recognition with big data - Bogdan Bocse

Scaling face recognition with big data - Bogdan Bocse

Industrialiser spark

Industrialiser spark

Industrialiser spark

Vulnerabilities such as Spectre and Meltdown continue to plague many production servers, based on Intel CPUs. Our solution involves software-based monitoring of hardware counters and sending that data to Apache Spark clusters for threat detection. We leverage Spark's support for support vector machine (SVM) inference. Our machine learning models are trained off-line by a data scientist within a Jupyter notebook environment. As new models are validated, they can be easily deployed to the Spark cluster from the notebook. We have standardized model export and import using the ONNX machine learning open file format. In our presentation, we will demo the full pipeline, from model training to deployment. We will discuss the various challenges when deploying ML-based cyber-threat detection at scale using Apache Spark. For example, we found that gaps in detection can occur when Spark models are updated. We will describe a novel data ingestion architecture, based on Apache Kafka, that we developed to deal with this issue.

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

The traditional approach to insurance pricing involves fitting a generalized linear model (GLM) to data collected on historical claims payments and premiums received. The explosive growth in data availability and increasing competitiveness in the marketplace are challenging actuaries to find new insights in their data and make predictions with more granularity, improved speed and efficiency, and with tighter integration among business units to support strategic decisions. In this session we will share our experience implementing deep hierarchical neural networks using TensorFlow and PySpark on Databricks. We will discuss the benefits of the ML Runtime, our experience using the goofys mount, our process for hyperparameter tuning, specific considerations for the large dataset size and extreme volatility present in insurance data, among other topics. Authors: Bryn Clark, Krish Rajaram

Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide

Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide

Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide

Vulnerabilities such as Spectre and Meltdown continue to plague many production servers, based on Intel CPUs. Our solution involves software-based monitoring of hardware counters and sending that data to Apache Spark clusters for threat detection. We leverage Spark's support for support vector machine (SVM) inference. Our machine learning models are trained off-line by a data scientist within a Jupyter notebook environment. As new models are validated, they can be easily deployed to the Spark cluster from the notebook. We have standardized model export and import using the ONNX machine learning open file format. In our presentation, we will demo the full pipeline, from model training to deployment. We will discuss the various challenges when deploying ML-based cyber-threat detection at scale using Apache Spark. For example, we found that gaps in detection can occur when Spark models are updated. We will describe a novel data ingestion architecture, based on Apache Kafka, that we developed to deal with this issue.

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Have you ever struggled to smoothly integrate Spark into a production workflow? Spark is an excellent tool for processing data on the terabyte scale, but building a system to move from raw data through featurization, modeling, and prediction serving involves interacting with numerous other components. Over the past year and half my team at ShopRunner has built a production Spark workflow for data science from scratch. In this talk you'll learn about the tools we use, the challenges we encountered, an open-source library we wrote to work through them, and how you can avoid the detours we took along the way. Data science work often begins in an interactive notebook environment, exploring data and testing out different modeling approaches. However, moving towards a production environment means building reproducible workflows, packaging libraries, setting up scheduling and monitoring of jobs, and figuring out ways to serve results to clients in real time. After testing out a variety of tools, we at ShopRunner have settled on a stack including Databricks, Snowflake, Datadog, Jenkins, and S3, ECS, and RDS from the suite of AWS services. These tools each offer unique benefits for their area of focus, but crafting a cohesive pipeline from this range of tools presented a challenge. Come learn how to integrate a Spark workflow into a pipeline that analyzes many terabytes of data, builds machine learning models at scale, and serves predictions to a variety of customer-facing tools. Whether you're just getting started using Spark in productions systems or you already have Spark running in production and want to smooth the process, this talk will leave you better equipped to find and connect the tools that suit your needs.

Connecting the Dots: Integrating Apache Spark into Production Pipelines

Connecting the Dots: Integrating Apache Spark into Production Pipelines

Connecting the Dots: Integrating Apache Spark into Production Pipelines

Relationships are one of the most predictive indicators of behavior and preferences. Communities detection based on relationships is a powerful tool for inferring similar preferences in peer groups, anticipating future behavior, estimating group resiliency, finding hierarchies, and preparing data for other analysis. Centrality measures based on relationships identify the most important items in a network and help us understand group dynamics such as influence, accessibility, the speed at which things spread, and bridges between groups. Data scientists use graph algorithms to identify groups and estimate important entities based on their interactions. In this session, we'll cover the common uses of community detection and centrality measures and how some of the iconic graph algorithms compute values. We'll show examples of how to run community detection and centrality algorithms in Apache Spark including using the AggregateMessages function to add your own algorithms. You'll learn best practices and tips for tricky situations. For those that want to run graph algorithms in a graph platform, we'll also illustrate a few examples in Neo4j. Some of the Community Detection Algorithms included: * Triangle Count and Clustering Coefficient to estimate network cohesiveness * Strongly Connected Components and Connected Components to find clusters * Label Propagation to quickly infer groups and data cleans with semi-supervised learning * Louvain Modularity to uncover at group hierarchies Balanced Triad to identify unstable groups * PageRank to reveal influencers * Betweenness Centrality to predict bottlenecks and bridges. Authors: Amy Hodler, Sören Reichardt

Predicting Influence and Communities Using Graph Algorithms

Predicting Influence and Communities Using Graph Algorithms

Predicting Influence and Communities Using Graph Algorithms

Similar a Stream Processing: Choosing the Right Tool for the Job (20)

Databricks: What We Have Learned by Eating Our Dog Food

Databricks: What We Have Learned by Eating Our Dog Food

Databricks: What We Have Learned by Eating Our Dog Food

Apache Spark Data Validation

Apache Spark Data Validation

Apache Spark Data Validation

Hybrid Transactional/Analytics Processing with Spark and IMDGs

Hybrid Transactional/Analytics Processing with Spark and IMDGs

Hybrid Transactional/Analytics Processing with Spark and IMDGs

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and...

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and...

The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and...

Near Real-Time Data Warehousing with Apache Spark and Delta Lake

Near Real-Time Data Warehousing with Apache Spark and Delta Lake

Near Real-Time Data Warehousing with Apache Spark and Delta Lake

Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph

Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph

Massive-Scale Entity Resolution Using the Power of Apache Spark and Graph

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

ITCamp 2019 - Andy Cross - Machine Learning with ML.NET and Azure Data Lake

ITCamp 2019 - Andy Cross - Machine Learning with ML.NET and Azure Data Lake

ITCamp 2019 - Andy Cross - Machine Learning with ML.NET and Azure Data Lake

Internals of Speeding up PySpark with Arrow

Internals of Speeding up PySpark with Arrow

Internals of Speeding up PySpark with Arrow

Scaling Face Recognition with Big Data

Scaling Face Recognition with Big Data

Scaling Face Recognition with Big Data

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...

Blue Pill/Red Pill: The Matrix of Thousands of Data Streams

Blue Pill/Red Pill: The Matrix of Thousands of Data Streams

Blue Pill/Red Pill: The Matrix of Thousands of Data Streams

AI on Spark for Malware Analysis and Anomalous Threat Detection

AI on Spark for Malware Analysis and Anomalous Threat Detection

AI on Spark for Malware Analysis and Anomalous Threat Detection

Scaling face recognition with big data - Bogdan Bocse

Scaling face recognition with big data - Bogdan Bocse

Scaling face recognition with big data - Bogdan Bocse

Industrialiser spark

Industrialiser spark

Industrialiser spark

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide

Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide

Deploying Enterprise Scale Deep Learning in Actuarial Modeling at Nationwide

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Scaling ML-Based Threat Detection For Production Cyber Attacks

Connecting the Dots: Integrating Apache Spark into Production Pipelines

Connecting the Dots: Integrating Apache Spark into Production Pipelines

Connecting the Dots: Integrating Apache Spark into Production Pipelines

Predicting Influence and Communities Using Graph Algorithms

Predicting Influence and Communities Using Graph Algorithms

Predicting Influence and Communities Using Graph Algorithms

Más de Databricks

DW Migration Webinar-March 2022.pptx

DW Migration Webinar-March 2022.pptx

DW Migration Webinar-March 2022.pptx

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 1 | Part 2

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 2

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 4

Data Lakehouse Symposium | Day 4

Data Lakehouse Symposium | Day 4

In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Learn to Use Databricks for Data Science

Learn to Use Databricks for Data Science

Learn to Use Databricks for Data Science

Application performance monitoring (APM) has become the cornerstone of software engineering allowing engineering teams to quickly identify and remedy production issues. However, as the world moves to intelligent software applications that are built using machine learning, traditional APM quickly becomes insufficient to identify and remedy production issues encountered in these modern software applications. As a lead software engineer at NewRelic, my team built high-performance monitoring systems including Insights, Mobile, and SixthSense. As I transitioned to building ML Monitoring software, I found the architectural principles and design choices underlying APM to not be a good fit for this brand new world. In fact, blindly following APM designs led us down paths that would have been better left unexplored. In this talk, I draw upon my (and my team’s) experience building an ML Monitoring system from the ground up and deploying it on customer workloads running large-scale ML training with Spark as well as real-time inference systems. I will highlight how the key principles and architectural choices of APM don’t apply to ML monitoring. You’ll learn why, understand what ML Monitoring can successfully borrow from APM, and hear what is required to build a scalable, robust ML Monitoring architecture.

Why APM Is Not the Same As ML Monitoring

Why APM Is Not the Same As ML Monitoring

Why APM Is Not the Same As ML Monitoring

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs. There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs. The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.

Stage Level Scheduling Improving Big Data and AI Integration

Stage Level Scheduling Improving Big Data and AI Integration

Stage Level Scheduling Improving Big Data and AI Integration

In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks. Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model? The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity. The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters. In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Simplify Data Conversion from Spark to TensorFlow and PyTorch

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling your Data Pipelines with Apache Spark on Kubernetes

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

In this talk about zipline, we will introduce a new type of windowing construct called a sawtooth window. We will describe various properties about sawtooth windows that we utilize to achieve online-offline consistency, while still maintaining high-throughput, low-read latency and tunable write latency for serving machine learning features.We will also talk about a simple deployment strategy for correcting feature drift – due operations that are not “abelian groups”, that operate over change data.

Sawtooth Windows for Feature Aggregations

Sawtooth Windows for Feature Aggregations

Sawtooth Windows for Feature Aggregations

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. To address these challenges, we built whylogs, an open source standard for data logging. It is a lightweight data profiling library that enables end-to-end data profiling across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It can work with different modes of data operations, including streaming, batch and IoT data. In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and various integrations. We will demonstrate how the whylogs integration with Apache Spark achieves large scale data profiling, and we will show how users can apply this integration into existing data and ML pipelines.

Re-imagine Data Monitoring with whylogs and Spark

Re-imagine Data Monitoring with whylogs and Spark

Re-imagine Data Monitoring with whylogs and Spark

Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components. We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure. This allows us to introduce optimization rules that (i) reduce unnecessary computations by passing information between the data processing and ML operators (ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and (iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator. We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.

Raven: End-to-end Optimization of ML Prediction Queries

Raven: End-to-end Optimization of ML Prediction Queries

Raven: End-to-end Optimization of ML Prediction Queries

Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.

Processing Large Datasets for ADAS Applications using Apache Spark

Processing Large Datasets for ADAS Applications using Apache Spark

Processing Large Datasets for ADAS Applications using Apache Spark

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Massive Data Processing in Adobe Using Delta Lake

Massive Data Processing in Adobe Using Delta Lake

Massive Data Processing in Adobe Using Delta Lake

Más de Databricks (20)

DW Migration Webinar-March 2022.pptx

DW Migration Webinar-March 2022.pptx

DW Migration Webinar-March 2022.pptx

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 4

Data Lakehouse Symposium | Day 4

Data Lakehouse Symposium | Day 4

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Learn to Use Databricks for Data Science

Learn to Use Databricks for Data Science

Learn to Use Databricks for Data Science

Why APM Is Not the Same As ML Monitoring

Why APM Is Not the Same As ML Monitoring

Why APM Is Not the Same As ML Monitoring

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Stage Level Scheduling Improving Big Data and AI Integration

Stage Level Scheduling Improving Big Data and AI Integration

Stage Level Scheduling Improving Big Data and AI Integration

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Sawtooth Windows for Feature Aggregations

Sawtooth Windows for Feature Aggregations

Sawtooth Windows for Feature Aggregations

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Re-imagine Data Monitoring with whylogs and Spark

Re-imagine Data Monitoring with whylogs and Spark

Re-imagine Data Monitoring with whylogs and Spark

Raven: End-to-end Optimization of ML Prediction Queries

Raven: End-to-end Optimization of ML Prediction Queries

Raven: End-to-end Optimization of ML Prediction Queries

Processing Large Datasets for ADAS Applications using Apache Spark

Processing Large Datasets for ADAS Applications using Apache Spark

Processing Large Datasets for ADAS Applications using Apache Spark

Massive Data Processing in Adobe Using Delta Lake

Massive Data Processing in Adobe Using Delta Lake

Massive Data Processing in Adobe Using Delta Lake

Último

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand Booking Contact Details :- WhatsApp Chat :- +91-7737669865 Call Girls In Model Towh +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At #K09 Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand

Saudi Arabia [ Abortion pills) Jeddah/riaydh/dammam/+966572737505☎️] cytotec tablets uses abortion pills 💊💊 How effective is the abortion pill? 💊💊 +966572737505) "Abortion pills in Jeddah" how to get cytotec tablets in Riyadh " Abortion pills in dammam*💊💊 The abortion pill is very effective. If you’re taking mifepristone and misoprostol, it depends on how far along the pregnancy is, and how many doses of medicine you take:💊💊 +966572737505) how to buy cytotec pills At 8 weeks pregnant or less, it works about 94-98% of the time. +966572737505[ 💊💊💊 At 8-9 weeks pregnant, it works about 94-96% of the time. +966572737505) At 9-10 weeks pregnant, it works about 91-93% of the time. +966572737505)💊💊 If you take an extra dose of misoprostol, it works about 99% of the time. At 10-11 weeks pregnant, it works about 87% of the time. +966572737505) If you take an extra dose of misoprostol, it works about 98% of the time. In general, taking both mifepristone and+966572737505 misoprostol works a bit better than taking misoprostol only. +966572737505 Taking misoprostol alone works to end the+966572737505 pregnancy about 85-95% of the time — depending on how far along the+966572737505 pregnancy is and how you take the medicine. +966572737505 The abortion pill usually works, but if it doesn’t, you can take more medicine or have an in-clinic abortion. +966572737505 When can I take the abortion pill?+966572737505 In general, you can have a medication abortion up to 77 days (11 weeks)+966572737505 after the first day of your last period. If it’s been 78 days or more since the first day of your last+966572737505 period, you can have an in-clinic abortion to end your pregnancy.+966572737505 Why do people choose the abortion pill? Which kind of abortion you choose all depends on your personal+966572737505 preference and situation. With+966572737505 medication+966572737505 abortion, some people like that you don’t need to have a procedure in a doctor’s office. You can have your medication abortion on your own+966572737505 schedule, at home or in another comfortable place that you choose.+966572737505 You get to decide who you want to be with during your abortion, or you can go it alone. Because+966572737505 medication abortion is similar to a miscarriage, many people feel like it’s more “natural” and less invasive. And some+966572737505 people may not have an in-clinic abortion provider close by, so abortion pills are more available to+966572737505 them. +966572737505 Your doctor, nurse, or health center staff can help you decide which kind of abortion is best for you. +966572737505 More questions from patients: Saudi Arabia+966572737505 CYTOTEC Misoprostol Tablets. Misoprostol is a medication that can prevent stomach ulcers if you also take NSAID medications. It reduces the amount of acid in your stomach, which protects your stomach lining. The brand name of this medication is Cytotec®.+966573737505) Unwanted Kit is a combination of two medici

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Riyadh +966572737505 get cytotec

Midocean dropshipping via API with DroFx

Midocean dropshipping via API with DroFx

Midocean dropshipping via API with DroFx

olyaivanovalion

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

MoniSankarHazra

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-9155563397 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-9155563397 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-9155563397 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —9155563397 We are available 24*7 all days of the year.

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...

only4webmaster01

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K to 25K High Profile Escorts In Pune Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 (V030524]N) 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 Call Girls In Model Towh +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At #K09 Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...

(Vivek)Call Us, 8448380779,Call girls in Delhi NCr – We Offer best in class call girls. escort Service At Affordable Price At low Rate with Space Night 8000 We Are One Of The Oldest Escort and Call girls Agencies in Delhi. You Will Find That Our Female Escorts Are Full Of Fun, Sexy And They Would Love Enjoy Your Company. We Have A Fantastic Selection Of Escort Ladies Available For In-Calls As Well As Out-Calls. Our Escorts Are Not Only Beautiful But All Have Great Personalities Making Them The Perfect Companion For Any Occasion. In-Call:- You Can Come At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call:- You have To Come Pick The Girl From My Place We Are Also Provide Door Step Services (Delhi Ncr, Noida, Gurgaon, Faridabad, Ghaziabad Note:- Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic:- Full Ac room And Clean Rooms Available In Hotel 24 * 7 Hourly In Delhi NCR More Details, With WhatsApp Number, +91-8448380779

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service

Delhi Call girls

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage with sex Booking Contact Details :- WhatsApp Chat :- +91-9920725232 4-May-2024(SMW) Call Girls In Model Towh +91-9920725232 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-9920725232 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —9920725232 We are available 24*7 all days of the year. Call us — 9920725232 Thank you for Visiting.

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 4-May-2024(SMW) Call Girls In Model Towh +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (Bangalore) Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

April 2024 - Crypto Market Report's Analysis

April 2024 - Crypto Market Report's Analysis

April 2024 - Crypto Market Report's Analysis

hybrid Seed Production In Chilli & Capsicum.pptx

hybrid Seed Production In Chilli & Capsicum.pptx

hybrid Seed Production In Chilli & Capsicum.pptx

Call Girl In Dwarka ☎92055#41914 ¶¶ Indian,Russian Best Quality full Educated And Full Cooperative Independent Call Girls Escort Services In New Delhi- I Have Extremely Beautiful Broad Minded Cute Sexy & Hot Call Girls and Escorts, We Are Located in 3* 4* 5* Hotels in Delhi. Safe & Secure High Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. Any Time for Model/Teens Escort in Delhi High class luxury and premium escorts agency Indian Russian Call Girls In Delhi Booking Good High Profile Escorts (Call Girls) In Delhi 5 Star Hotel ,Incall Service,OutCall Service, We provide services by Call Girls,College Girls,Modals Get High Profile queens,Well Educated,Good Looking,Full Cooperative Model, Russian Models,Punjabi Girls Kashmeri Girls Services etc… We Provide Hottest Female With Safe And Consensual With Most Limits Respected Complete Satisfaction Guaranteed…Service. Call Me Spacial For Including Incall//outcall Service In New Delhi Indian Russian Escorts Service

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Delhi Call girls

Klinik_ Apotek Onlin 085657271886 Solusi Menggugurkan Masalah Kehamilan Anda Jual Obat Aborsi Asli KLINIK ABORSI TERPEECAYA _ Jual Obat Aborsi Cytotec Misoprostol Asli 100% Ampuh Hanya 3 Jam Langsung Gugur || OBAT PENGGUGUR KANDUNGAN AMPUH MANJUR OBAT ABORSI OLINE" APOTIK Jual Obat Cytotec, Gastrul, Gynecoside Asli Ampuh. JUAL ” Obat Aborsi Tuntas | Obat Aborsi Manjur | Obat Aborsi Ampuh | Obat Penggugur Janin | Obat Pencegah Kehamilan | Obat Pelancar Haid | Obat terlambat Bulan | Ciri Obat Aborsi Asli | Obat Telat Bulan | Pil Aborsi Asli | Cara Menggugurkan Konten | Cara Aborsi Tuntas | Harga Obat Aborsi Asli | Pil Aborsi | Jual Obat Aborsi Cytotec | Cara Aborsi Sendiri | Cara Aborsi Usia 1 Bulan | Cara Aborsi Usia 2 Tahun | Cara Aborsi Usia 3 Bulan | Obat Aborsi Usia 4 Bulan | Cara Abrasi Usia 5 Bulan | Cara Menggugurkan Konten | Kandungan Obat Penggugur | Cara Menghitung Usia Konten | Cara Mengatasi Terlambat Bulan | Penjual Obat Aborsi Asli | Obat Aborsi Garansi | Kandungan Obat Peluntur | Obat Telat Datang Bulan | Obat Telat Haid | Obat Aborsi Paling Murah | Klinik Jual Obat Aborsi | Jual Pil Cytotec | Apotik Jual Obat Aborsi | Kandungan Dokter Abrasi | Cara Aborsi Cepat | Jual Obat Aborsi Bergaransi | Jual Obat Cytotec Asli | Obat Aborsi Aman Manjur | Obat Misoprostol Cytotec Asli. "APA ITU ABORSI" “Aborsi Adalah dengan membendung hormon yang di perlukan untuk mempertahankan kehamilan yaitu hormon progesteron, karena hormon ini dibendung, maka jalur kehamilan mulai membuka dan leher rahim menjadi melunak,sehingga mengeluarkan darah yang merupakan tanda bahwa obat telah bekerja || maksimal 1 jam obat diminum || PENJELASAN OBAT ABORSI USIA 1 _7 BULAN Pada usia kandungan ini, pasien akan merasakan sakit yang sedikit tidak berlebihan || sekitar 1 jam ||. namun hanya akan terjadi pada saatdarah keluar merupakan pertanda menstruasi. Hal ini dikarenakan pada usiakandungan 3 bulan,janin sudah terbentuk sebesar kepalan tangan orang dewasa. Cara kerja obat aborsi : JUAL OBAT ABORSI AMPUH dosis 3 bulan secara umum sama dengan cara kerja || DOSIS OBAT ABORSI 2 bulan”, hanya berbedanya selain mengisolasijanin juga menghancurkan janin dengan formula methotrexate dikandungdidalamnya. Formula methotrexate ini sangat ampuh untuk menghancurkan janinmenjadi serpihan-serpihan kecil akan sangat berguna pada saat dikeluarkan nanti. APA ALASAN WANITA MELAKUKAN ABORSI? Aborsi di lakukan wanita hamil baik yang sudah menikah maupun belum menikah dengan berbagai alasan , akan tetapi alasan yang utama adalah alasan-alasan non medis (termasuk aborsi sendiri / di sengaja/ buatan] MELAYANI PEMESANAN OBAT ABORSI SETIAP HARI, SIAP KIRIM KESELURUH KOTA BESAR DI INDONESIA DAN LUAR NEGERI. HUBUNGI PEMESANAN LEBIH NYAMAN VIA WA/: 085657271886

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 Call Girls In Model Towh +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At #K09 Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Último (20)

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand

Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Midocean dropshipping via API with DroFx

Midocean dropshipping via API with DroFx

Midocean dropshipping via API with DroFx

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...

Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...

➥🔝 7737669865 🔝▻ Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service

BDSM⚡Call Girls in Mandawali Delhi >༒8448380779 Escort Service

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

April 2024 - Crypto Market Report's Analysis

April 2024 - Crypto Market Report's Analysis

April 2024 - Crypto Market Report's Analysis

hybrid Seed Production In Chilli & Capsicum.pptx

hybrid Seed Production In Chilli & Capsicum.pptx

hybrid Seed Production In Chilli & Capsicum.pptx

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...

➥🔝 7737669865 🔝▻ Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Stream Processing: Choosing the Right Tool for the Job

1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics

2. Giselle van Dongen, Stream processing: choosing the right tool for the job #UnifiedDataAnalytics #SparkAISummit

3. 3#UnifiedDataAnalytics #SparkAISummit ● ● ○ ○

4. Context 4#UnifiedDataAnalytics #SparkAISummit …

5. Context 5#UnifiedDataAnalytics #SparkAISummit

6. Context 6#UnifiedDataAnalytics #SparkAISummit

7. Disclaimer 7#UnifiedDataAnalytics #SparkAISummit a.k.a I will not pick a stream processing framework for you

8. Commonalities 8#UnifiedDataAnalytics #SparkAISummit ● ● ● ●

9. Imagine... 9#UnifiedDataAnalytics #SparkAISummit

10. 10#UnifiedDataAnalytics #SparkAISummit

11. 11#UnifiedDataAnalytics #SparkAISummit

12. Do we need stream processing? 12#UnifiedDataAnalytics #SparkAISummit …

13. Do we need stream processing? 13#UnifiedDataAnalytics #SparkAISummit …

14. 14#UnifiedDataAnalytics #SparkAISummit

15. 15#UnifiedDataAnalytics #SparkAISummit

16. 16#UnifiedDataAnalytics #SparkAISummit

17. How much data? ➔ ➔ 17#UnifiedDataAnalytics #SparkAISummit

18. How much data? 18#UnifiedDataAnalytics #SparkAISummit

19. Spark Flink Kafka Spark Struct Flink KafkaStruct

20. 20#UnifiedDataAnalytics #SparkAISummit

21. Does it need to be fast? 21#UnifiedDataAnalytics #SparkAISummit ➔ ➔

22. Does it need to be fast? 22#UnifiedDataAnalytics #SparkAISummit

23. Does it need to be fast? ● … ● ● 23#UnifiedDataAnalytics #SparkAISummit

24. Event-driven 24#UnifiedDataAnalytics #SparkAISummit https://www.cakesolutions.net/teamblogs/comparison-of-apache-stream-processing-frameworks-part-1

25. Micro-batching 25#UnifiedDataAnalytics #SparkAISummit https://www.cakesolutions.net/teamblogs/comparison-of-apache-stream-processing-frameworks-part-1

26. Does it need to be fast? ● 26#UnifiedDataAnalytics #SparkAISummit ➔ ➔

27. Spark Flink Kafka Spark Struct Flink KafkaStruct

28. 28#UnifiedDataAnalytics #SparkAISummit

29. 29#UnifiedDataAnalytics #SparkAISummit Performance Advanced features Deployment & Internals

30. Who will build it? 30#UnifiedDataAnalytics #SparkAISummit

31. Who will build it? 31#UnifiedDataAnalytics #SparkAISummit

32. Spark Flink Kafka Spark Struct Flink KafkaStruct

33. 33#UnifiedDataAnalytics #SparkAISummit Performance Advanced features Deployment & Internals

34. Is accurate ordering important? 34#UnifiedDataAnalytics #SparkAISummit 4 7 10 window 1 window 2 window 3 8

35. Is accurate ordering important? 35#UnifiedDataAnalytics #SparkAISummit 4 7 10 window 1 window 2 window 3 88

36. 36#UnifiedDataAnalytics #SparkAISummit 4 7 10 window 1 window 2 window 3 88 window 1 window 2 window 3 Is accurate ordering important?

37. Is accurate ordering important? 37#UnifiedDataAnalytics #SparkAISummit

38. Is accurate ordering important? 38#UnifiedDataAnalytics #SparkAISummit ➔ ➔

39. Spark Flink Kafka Spark Struct Flink KafkaStruct

40. 40#UnifiedDataAnalytics #SparkAISummit Performance Advanced features Deployment & Internals

41. What is the ecosystem like? • • • 41#UnifiedDataAnalytics #SparkAISummit

42. What is the ecosystem like? 42#UnifiedDataAnalytics #SparkAISummit

43. Spark Flink Kafka Spark Struct Flink KafkaStruct

44. 44#UnifiedDataAnalytics #SparkAISummit

45. 45#UnifiedDataAnalytics #SparkAISummit

46. How do we want it to run? 46#UnifiedDataAnalytics #SparkAISummit W W W W W M T T T T T

47. Spark Flink Kafka Spark Struct Flink KafkaStruct

48. 48#UnifiedDataAnalytics #SparkAISummit

49. What if a message gets lost? 49#UnifiedDataAnalytics #SparkAISummit

50. What if a message gets lost? • • • 50#UnifiedDataAnalytics #SparkAISummit

51. What if a message gets lost? • •

52. W1 W3 W4 W2 W4 What if a message gets lost? 52#UnifiedDataAnalytics #SparkAISummit State State State State State HDFS ref. Flink Forward 2018 Best practices for state and time, Tzu-Li Tai

53. W1 W3 W4 W2 W4 What if a message gets lost? 53#UnifiedDataAnalytics #SparkAISummit State State State State State HDFS ref. Flink Forward 2018 Best practices for state and time, Tzu-Li Tai

54. W1 W4 W2 W4 W3 What if a message gets lost? 54#UnifiedDataAnalytics #SparkAISummit State State State State State HDFS ref. Flink Forward 2018 Best practices for state and time, Tzu-Li Tai

55. What if a message gets lost? ● ○ ○ ○ ● ○ ○ ○ ○

56. What if a message gets lost? 56#UnifiedDataAnalytics #SparkAISummit

57. What if a message gets lost? 57#UnifiedDataAnalytics #SparkAISummit

58. Spark Flink Kafka Spark Struct Flink KafkaStruct

59. Spark Flink Kafka SparkStruct Struct Flink Kafka

60. 60#UnifiedDataAnalytics #SparkAISummit

61. 61#UnifiedDataAnalytics #SparkAISummit

62. Want to know more? 62#UnifiedDataAnalytics #SparkAISummit

63. 63#UnifiedDataAnalytics #SparkAISummit THANK YOU! Do you want to work with these tools? We are hiring!

64. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT