Challenges in building a Data Pipeline

•Descargar como PPTX, PDF•

0 recomendaciones•241 vistas

The document discusses challenges in building a data pipeline including making it highly scalable, available with low latency and zero data loss while supporting multiple data sources. It covers expectations for real-time vs batch processing and explores stream and batch architectures using tools like Apache Storm, Spark and Kafka. Challenges of data replication, schema detection and transformations with NoSQL are also examined. Effective implementations should include monitoring, security and replay mechanisms. Finally, lambda and kappa architectures for combining stream and batch processing are presented.

Software

Manish Singh
Engineer at Hevo
https://linkedin.com/in/manishsingh123/
Challenges in Building a
Data Pipeline

● Data Pipeline
● Possible Implementations
● Challenges
● Data Processing Architectures
Agenda

● Highly scalable
● Highly available
● Low latency
● Zero data loss
● Support for multiple data sources (e.g. MySQL, NoSQL,
Mixpanel, Analytics)
● Instrumentation, monitoring, and alerting
● Real-time vs Batch
Expectations

Stream
● Usages: Live dashboards
(count, average), rate
limiting, triggers
● Processing: Apache Storm,
Apache Spark, Apache
Samza
● Store: Elastic Search, Druid,
Spark SQL, Kafka SQL
Stream vs Batch
Batch
● Batch Processing
and
pre-computation
● Immutable Store: HDFS,
Cassandra, Event Stream to
S3
● Data Warehouse: HBase,
Hive, Redshift, Postgres

● ETL (Extract -> Transform -> Load)
● ELT (Extract -> Load -> Transform)
ETL vs ELT

● Complexity of transformation logic compromises latency
● Hardware systems today are better equipped
● Efficient, reduces load time
● Cost effective in the cloud, less components required
Moving from traditional ETL
to ELT

● Query Source DB and keep offset (ID, Updated timestamp)
● Database change logs (e.g. Mysql Binlogs, MongoDB Oplogs)
Replication Modes

● New fields can be added to a source at any point in time
● Character lengths of String columns in source can increase
● Data Type incompatibility between Source and Destination
● Varying type casting
● Data loss during loads - Power failure, Server failure, Code
bugs, etc
Challenges

● Schema detection cannot be done upfront
● Different documents in a single collection can have a different
set of fields
● Different documents in a single collection can have
incompatible field data types
● Nested objects and arrays with a dynamic structure
Additional Challenges with
NoSQL

● Transformations
● Security (Filter, Hashing)
● Replay Mechanism
● Integrity and Anomaly Detection
● Monitoring and Alerts for failures
● Activity Log
Effective Implementations

● How to beat the CAP theorem by Nathan Marz
● Different layers for stream and batch processing
● Need to manage two different layers of the system
Lambda Architecture

● Questioning the Lambda Architecture by Jay Kreps
● Only stream processing with parallelism
● Set Kafka retention policy
● Reprocess into separate table
● Switch table when done and delete the old one
Kappa Architecture

Thank You
Manish Singh, Hevo
https://linkedin.com/in/manishsingh123/

Más contenido relacionado

La actualidad más candente

Challenges in Building a Data Pipeline

Manish Kumar

A tenet of the Agile Manifesto is ‘Working software over comprehensive documentation’, and many have interpreted that to mean that data models are not necessary in the agile development environment. Others have seen the value of data models for achieving the other core tenets of ‘Customer Collaboration’ and ‘Responding to Change’. This webinar will discuss how data models are being effectively used in today’s Agile development environment and the benefits that are being achieved from this approach.

Agile & Data Modeling – How Can They Work Together?

DATAVERSITY

Active Governance Across the Delta Lake with Alation

Databricks

Reference matter data management: Two categories of structured data : Master data: is data associated with core business entities such as customer, product, asset, etc. Transaction data: is the recording of business transactions such as orders in manufacturing, loan and credit card payments in banking, and product sales in retail. Reference data: is any kind of data that is used solely to categorize other data found in a database, or solely for relating data in a database to information beyond the boundaries of the enterprise .

Reference master data management

Dr. Hamdan Al-Sabri

MDM Strategy & Roadmap

victorlbrown

A data warehouse is a database designed for query and analysis rather than for transaction processing. An appropriate design leads to scalable, balanced and flexible architecture that is capable to meet both present and long-term future needs. This session covers a comparison of the main data warehouse architectures together with best practices for the logical and physical design that support staging, load and querying.

Data Warehouse Design and Best Practices

Ivo Andreev

Data Platform Architecture Principles and Evaluation Criteria

ScyllaDB

In this talk we will present how Databricks has enabled the author to achieve more with data, enabling one person to build a coherent data project with data engineering, analysis and science components, with better collaboration, better productionalization methods, with larger datasets and faster. The talk will include a demo that will illustrate how the multiple functionalities of Databricks help to build a coherent data project with Databricks jobs, Delta Lake and auto-loader for data engineering, SQL Analytics for Data Analysis, Spark ML and MLFlow for data science, and Projects for collaboration.

Databricks: A Tool That Empowers You To Do More With Data

Databricks

Summary introduction to data engineering

Novita Sari

Data modeling continues to be a tried-and-true method of managing critical data aspects from both the business and technical perspective. Like any tool or methodology, there is a “right tool for the right job”, and specific model types exist for both business and technical users across operational, reporting, analytic, and other use cases. This webinar will provide an overview of the various data modeling techniques available, and how to use each for maximum value to the organization.

Data Modeling Techniques

DATAVERSITY

Data mesh was among the most discussed and controversial enterprise data management topics of 2021. One of the reasons people struggle with data mesh concepts is we still have a lot of open questions that we are not thinking about: Are you thinking beyond analytics? Are you thinking about all possible stakeholders? Are you thinking about how to be agile? Are you thinking about standardization and policies? Are you thinking about organizational structures and roles? Join data.world VP of Product Tim Gasper and Principal Scientist Juan Sequeda for an honest, no-bs discussion about data mesh and its role in data governance.

Five Things to Consider About Data Mesh and Data Governance

DATAVERSITY

Data Architecture Brief Overview

Hal Kalechofsky

Data Quality Integration (ETL) Open Source

Stratebi

Watch full webinar here: https://bit.ly/3zaPGLO Our approach for data virtualization advisory takes the following 3 dimensions/areas into consideration: - Technology / Architecture - Business User Groups (your clients) - IT Organization To deliver quick results, Q-PERIOR uses a multitude of accelerators in predefined topics within these three dimensions. In our presentation we will elaborate on client examples why such an exercise makes sense before rolling out Denodo and what kind of risks you can avoid doing so.

Key Considerations While Rolling Out Denodo Platform

Denodo

Graph databases

Vinoth Kannan

Traditional data warehouse ETL has become too slow, too complicated, and too expensive to address the torrent of new data sources and new analytic approaches needed for decision making. The new ETL environment is already looking drastically different. In this webinar, Ralph Kimball, founder of the Kimball Group, and Manish Vipani, Vice President and Chief Architect of Enterprise Architecture at Kaiser Permanente will describe how this new ETL environment is actually implemented at Kaiser Permanente. They will describe the successes, the unsolved challenges, and their visions of the future for data warehouse ETL.

The Future of Data Warehousing: ETL Will Never be the Same

Cloudera, Inc.

For business users, always using AI is about easy access to the tools without writing any code. This session is not about learning how to do AI but how to make AI usable and add value. AI powered visuals such as Key Influencer in Power BI desktop to analyse the data without deep knoledge of the machine learning concepts. Machine Learning is approaching a peak of inflated expectations, although we see AI daily and in all contexts. Media pressure is high, governments are overly optimistic, plenty of ventures are putting money in unviable ideas or some brilliant engineers fail to reach business users. But Microsoft bring all of this under the same roof and unleash the power of AI by integrating Power BI ecosystem with Azure ML and Cognitive services. The result is as simple and effective as great technology at end-user's hand.

ML with Power BI for Business and Pros

Ivo Andreev

Data engineering design patterns

Valdas Maksimavičius

Correctly Architecting your Solutions for Analytical & Operational Uses reviews the two main types of use cases that can be solved with the Denodo Platform. Both high concurrency scenarios and big reporting use cases are discussed in this presentation in a comparative way, explaining the different approaches that you must take to be successful in any situation. This presentation is part of the Fast Data Strategy Conference, and you can watch the video here goo.gl/wdZgpo.

Data Virtualization Reference Architectures: Correctly Architecting your Solu...

Denodo

Snowflake Overview

Snowflake Computing

La actualidad más candente (20)

Challenges in Building a Data Pipeline

Agile & Data Modeling – How Can They Work Together?

Active Governance Across the Delta Lake with Alation

Reference master data management

MDM Strategy & Roadmap

Data Warehouse Design and Best Practices

Data Platform Architecture Principles and Evaluation Criteria

Databricks: A Tool That Empowers You To Do More With Data

Summary introduction to data engineering

Data Modeling Techniques

Five Things to Consider About Data Mesh and Data Governance

Data Architecture Brief Overview

Data Quality Integration (ETL) Open Source

Key Considerations While Rolling Out Denodo Platform

Graph databases

The Future of Data Warehousing: ETL Will Never be the Same

ML with Power BI for Business and Pros

Data engineering design patterns

Data Virtualization Reference Architectures: Correctly Architecting your Solu...

Snowflake Overview

Similar a Challenges in building a Data Pipeline

Cloud Lambda Architecture Patterns

Asis Mohanty

JPoint'15 Mom, I so wish Hibernate for my NoSQL database...

Alexey Zinoviev

We will start from understanding how Real-Time Analytics can be implemented on Enterprise Level Infrastructure and will go to details and discover how different cases of business intelligence be used in real-time on streaming data. We will cover different Stream Data Processing Architectures and discus their benefits and disadvantages. I'll show with live demos how to build Fast Data Platform in Azure Cloud using open source projects: Apache Kafka, Apache Cassandra, Mesos. Also I'll show examples and code from real projects.

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...

Fwdays

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson

Spark Summit

Spark Concepts - Spark SQL, Graphx, Streaming

Petr Zapletal

Les mégadonnées représentent un vrai enjeu à la fois technique, business et de société : l'exploitation des données massives ouvre des possibilités de transformation radicales au niveau des entreprises et des usages. Tout du moins : à condition que l'on en soit techniquement capable... Car l'acquisition, le stockage et l'exploitation de quantités massives de données représentent des vrais défis techniques. Une architecture big data permet la création et de l'administration de tous les systèmes techniques qui vont permettre la bonne exploitation des données. Il existe énormément d'outils différents pour manipuler des quantités massives de données : pour le stockage, l'analyse ou la diffusion, par exemple. Mais comment assembler ces différents outils pour réaliser une architecture capable de passer à l'échelle, d'être tolérante aux pannes et aisément extensible, tout cela sans exploser les coûts ? Le succès du fonctionnement de la Big data dépend de son architecture, son infrastructure correcte et de son l’utilité que l’on fait ‘’ Data into Information into Value ‘’. L’architecture de la Big data est composé de 4 grandes parties : Intégration, Data Processing & Stockage, Sécurité et Opération.

Big Data_Architecture.pptx

betalab

Introduction to Apache Apex

Apache Apex

Researching an alternative to the MS SQL database - first of all in order to gain additional technological benefits, secondly moving towards an open source way of development. The idea behind this presentation was to introduce PostgreSQL (ver. 9.4+) in a different manner than a conventional "Pros Vs. Cons" style, it is more likely to be a "Buzz Word" thesaurus (of course based on a deep research). P.S. Since it's a presentation, there was no intention going over and covering all of the PostgreSQL features - most of the interesting parts.

PostgreSQL as an Alternative to MSSQL

Alexei Krasner

Kylin and Druid Presentation

argonauts007

Big Data Streaming processing using Apache Storm - FOSSCOMM 2016

Adrianos Dadis

Data streaming fundamentals

Mohammed Fazuluddin

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Luan Moreno Medeiros Maciel

Drill architecture 20120913

jasonfrantz

Cassandra training

András Fehér

This talk will address how a new architecture is emerging for analytics, based on Spark, Mesos, Akka, Cassandra and Kafka (SMACK). Popular architecture like Lambda separate layers of computation and delivery and require many technologies which have overlapping functionality. Some of this results in duplicated code, untyped processes, or high operational overhead, let alone the cost (i.e. ETL). I will discuss the problem domain and what is needed in terms of strategies, architecture and application design and code to begin leveraging simpler data flows. We will cover how the particular set of technologies addresses common requirements and how collaboratively they work together to enrich and reinforce each other.

Streaming Analytics with Spark, Kafka, Cassandra and Akka

Helena Edelson

Azure DocumentDB Overview

Andrew Liu

Glint with Apache Spark

Venkata Naga Ravi

Introduction to Apache NiFi dws19 DWS - DC 2019

Timothy Spann

NoSQL.pptx

RithikRaj25

This is the introductory presentation on HBase given by Hayden Marchant in the monthly Amobee Tech Talk. In this session, we'll learn about HBase, a NoSQL database that provides real-time, random read and write access to tables meant to store billions of rows and millions of columns. HBase is an open-source, non-relational distributed column-oriented database, is linearly scalable, and is designed to run on commodity hardware. HBase clusters can be in the hundreds and thousands of nodes, serving extraordinary amounts of information. Tight integration with Hadoop gives way to allows powerful analytical processing on data residing in HBase.

HBase introduction talk

Hayden Marchant

Similar a Challenges in building a Data Pipeline (20)

Cloud Lambda Architecture Patterns

JPoint'15 Mom, I so wish Hibernate for my NoSQL database...

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson

Spark Concepts - Spark SQL, Graphx, Streaming

Big Data_Architecture.pptx

Introduction to Apache Apex

PostgreSQL as an Alternative to MSSQL

Kylin and Druid Presentation

Big Data Streaming processing using Apache Storm - FOSSCOMM 2016

Data streaming fundamentals

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Drill architecture 20120913

Cassandra training

Streaming Analytics with Spark, Kafka, Cassandra and Akka

Azure DocumentDB Overview

Glint with Apache Spark

Introduction to Apache NiFi dws19 DWS - DC 2019

NoSQL.pptx

HBase introduction talk

Último

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal

%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...

masabamasaba

Craft an AI & Machine Learning Pitch with our Editable Professional PowerPoint Template. Ignite your AI & Machine Learning pitch with our cutting-edge PowerPoint template tailored for the industry. Perfect for AI conferences, investor presentations, sales pitches to tech-focused companies, training sessions, and educational programs. - 20+ editable slides: Get a variety of options to choose from for your presentation. - Time-saving solution: Download, replace text/images with a few clicks. - User-friendly customization: Easy to use and personalize. - Modern and attractive design: Captivating visuals, sleek layout. - Tailored to your requirements: Fully alterable for customization. - Well-organized slides: Complete control over content. - Thematic specificity: Reflects healthcare industry with relevant graphics. - Showcase your business idea: Communicate value proposition effectively.

AI & Machine Learning Presentation Template

Presentation.STUDIO

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

WSO2

%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein

masabamasaba

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

ADR, or Architecture Decision Record, is a valuable tool in software development for several reasons. It provides a centralized location for documenting and tracking architectural decisions, aiding both current and future team members. ADRs enhance communication among team members by documenting the rationale behind architectural decisions, especially beneficial during onboarding of new team members or when revisiting decisions. They serve as a knowledge base, enabling teams to learn from past decisions and refine their decision-making process. Additionally, ADRs contribute to transparency by helping stakeholders understand the reasons behind specific architectural choices. As with any other tool or process, introducing them into an organization can face several obstacles, and overcoming these challenges is crucial for successful implementation. In this talk I go through some common problems and our way of solving them.

Architecture decision records - How not to get lost in the past

Papp Krisztián

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

WSO2

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

This presentation covers the following topics: What is logging? The purpose of logging: Debugging The purpose of logging: Security The purpose of logging: Stats & analytics Traditional logging Traditional logging: Advantages Traditional logging: Disadvantages The solution: Large-scale logging Large-scale logging: Core principles Large-scale logging: Solution types Large-scale logging: Cloud vs on-prem Large-scale logging: Operational complexity Large-scale logging: Security Large-scale logging: Costs Large-scale logging: On-prem comparison - Elasticsearch - Grafana Loki - VictoriaLogs On-prem comparison: Setup and operation On-prem comparison: Costs On-prem comparison: Full-text search support On-prem comparison: How to efficiently query 100TB of logs? On-prem comparison: Integration with CLI tools VictoriaLogs for large-scale logging VictoriaLogs demo instance - Ingestion rate: 3600 messages / minute - The number of log messages: 1.1 billion - Uncompressed log messages’ size: 1.5TB - Compressed log messages’ size: 23GB - Compression ratio: 47x - Memory usage: 150MB VictoriaLogs CLI integration demo - Which errors have occurred in all the apps during the last hour? - How many errors have occurred during the last hour? - Which apps generated the most of errors during the last hour? - The number of per-minute errors for the last 10 minutes - Status codes for the last hour - Non-200 status codes for the last week - Top client IPs for the last 4 weeks with 404 and 500 response status codes - Per-month stats for the given IP across all the logs Large-scale logging solution MUST provide excellent CLI integration VictoriaLogs: (temporary) drawbacks VictoriaLogs: Recap - Easy to setup and operate - The lowest RAM usage and disk space usage (up to 30x less than Elasticsearch and Grafana Loki) - Fast full-text search - Excellent integration with traditional command-line tools for log analysis - Accepts logs from popular log shippers (Filebeat, Fluentbit, Logstash, Vector, Promtail, Grafana Agent) - Open source and free to use!

Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024

VictoriaMetrics

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of Winnipeg back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal. Attraction Spells

%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...

masabamasaba

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pretoria ● Abortion Pills For Sale In Pretoria ● Pretoria 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

%in Midrand+277-882-255-28 abortion pills for sale in midrand

masabamasaba

%in Soweto+277-882-255-28 abortion pills for sale in soweto

masabamasaba

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

SelfMade bd

Artyushina_Guest lecture_YorkU CS May 2024.pptx

AnnaArtyushina1

WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source

WSO2

Challenges in building a Data Pipeline

1. Manish Singh Engineer at Hevo https://linkedin.com/in/manishsingh123/ Challenges in Building a Data Pipeline

2. ● Data Pipeline ● Possible Implementations ● Challenges ● Data Processing Architectures Agenda

3. ● Highly scalable ● Highly available ● Low latency ● Zero data loss ● Support for multiple data sources (e.g. MySQL, NoSQL, Mixpanel, Analytics) ● Instrumentation, monitoring, and alerting ● Real-time vs Batch Expectations

4. Stream ● Usages: Live dashboards (count, average), rate limiting, triggers ● Processing: Apache Storm, Apache Spark, Apache Samza ● Store: Elastic Search, Druid, Spark SQL, Kafka SQL Stream vs Batch Batch ● Batch Processing and pre-computation ● Immutable Store: HDFS, Cassandra, Event Stream to S3 ● Data Warehouse: HBase, Hive, Redshift, Postgres

5. ● ETL (Extract -> Transform -> Load) ● ELT (Extract -> Load -> Transform) ETL vs ELT

7. ● Complexity of transformation logic compromises latency ● Hardware systems today are better equipped ● Efficient, reduces load time ● Cost effective in the cloud, less components required Moving from traditional ETL to ELT

8. ● Query Source DB and keep offset (ID, Updated timestamp) ● Database change logs (e.g. Mysql Binlogs, MongoDB Oplogs) Replication Modes

9. ● New fields can be added to a source at any point in time ● Character lengths of String columns in source can increase ● Data Type incompatibility between Source and Destination ● Varying type casting ● Data loss during loads - Power failure, Server failure, Code bugs, etc Challenges

10. ● Schema detection cannot be done upfront ● Different documents in a single collection can have a different set of fields ● Different documents in a single collection can have incompatible field data types ● Nested objects and arrays with a dynamic structure Additional Challenges with NoSQL

11. ● Transformations ● Security (Filter, Hashing) ● Replay Mechanism ● Integrity and Anomaly Detection ● Monitoring and Alerts for failures ● Activity Log Effective Implementations

12.

13.

14. ● How to beat the CAP theorem by Nathan Marz ● Different layers for stream and batch processing ● Need to manage two different layers of the system Lambda Architecture

15. Lambda Architecture

16. ● Questioning the Lambda Architecture by Jay Kreps ● Only stream processing with parallelism ● Set Kafka retention policy ● Reprocess into separate table ● Switch table when done and delete the old one Kappa Architecture

17. Kappa Architecture

18. Questions?

19. Thank You Manish Singh, Hevo https://linkedin.com/in/manishsingh123/

Notas del editor

https://youtu.be/YzAIjEQ75_c?t=6892 Explain Kafka SQL
Yahoo’s Hadoop clusters sorted 1 TB of data in 209 seconds Petabyte sort using Spark in 4 hours
Petabyte sort using Spark in 4 hours
Petabyte sort using Spark in 4 hours
Petabyte sort using Spark in 4 hours
Petabyte sort using Spark in 4 hours
Lambda - 11th Greek letter
Kappa - 10th Greek letter

Challenges in building a Data Pipeline

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Challenges in building a Data Pipeline

Similar a Challenges in building a Data Pipeline (20)

Último

Último (20)

Challenges in building a Data Pipeline

Notas del editor