Enterprise integration is more challenging than ever before. The IT evolution requires the integration of more and more technologies. Applications are deployed across the edge, hybrid, and multi-cloud architectures. Traditional middleware such as MQ, ETL, ESB does not scale well enough or only processes data in batch instead of real-time.
This presentation explores why Apache Kafka is the new black for integration projects, how Kafka fits into the discussion around cloud-native iPaaS (Integration Platform as a Service) solutions, and why event streaming is a new software category.
A concrete real-world example shows the difference between event streaming and traditional integration platforms respectively cloud-native iPaaS.
Video Recording of this presentation:
https://www.youtube.com/watch?v=I8yZwKg_IJc&t=2842s
Blog post about this topic:
https://www.kai-waehner.de/blog/2021/11/03/apache-kafka-cloud-native-ipaas-versus-mq-etl-esb-middleware/
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
1. Apache Kafka vs. Traditional Middleware
(MQ, ETL, ESB, iPaaS)
Friends, Enemies, or Frenemies?
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
@KaiWaehner
confluent.io
kai-waehner.de
2. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agreement:
Kafka is the de facto standard for …
• messaging at scale!
• decoupling of microservices!
• reliable, lightweight stream processing!
• real-time backbone for data mesh
Controversial discussion:
Apache Kafka vs. Traditional Middleware
Use Apache Kafka
as middleware!
3. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agenda
1. Traditional Middleware
2. Data Streaming Platform
3. Enemies
4. Friends
5. Frenemies
4. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agenda
1. Traditional Middleware
2. Data Streaming Platform
3. Enemies
4. Friends
5. Frenemies
5. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Source A
Source B
Sink X
Sink Y
Connector A
REST / SOAP
Connector X
JMS
ETL / ESB
Product
XYZ
Dream
6. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Passive Server
Source A
Source B
Sink X
Sink Y
Connector A
REST / SOAP
Connector X
JMS
ETL / ESB
Active Server
Messaging System
A
(Real Time)
Messaging System
B
(Big Data)
Database
(Important Data)
In-Memory Data
Grid
(Cache)
API Gateway
Stream Processing
Engine
Another
integration tool
(because one tool cannot
integrate everything, buy
more…)
Reality
7. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Gartner Magic Quadrant for iPaaS
2021
(Enterprise Integration Platform as
a Service)
iPaaS as Game Changer?
8. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Business Digitalization Trends are Driving the Need to Process Events at a
whole new Scale, Speed and Efficiency
The World has Changed
Mobile Cloud Microservices Internet of Things Machine Learning
9. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
#NoMQ #NoESB #NoETL #NoiPaaS can handle this well!
• Tight coupling
• Limited scale
• Zoo of components
?
10. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agenda
1. Traditional Middleware
2. Data Streaming Platform
3. Enemies
4. Friends
5. Frenemies
11. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Source A
Source B
Sink X
Sink Y
Connector A
Java
Connector X
REST
Data
Streaming
Platform
Hmm…
Yet another one of
those magic boxes
in the middle, huh?
12. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Events
What is an event?
14. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Events
A Sale An Invoice A Trade A Customer
Experience
15. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Haven’t we seen all
this before?
16. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?
17. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
26
18. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Only Data Streaming Has All 4 Requirements
Messaging
Databases
Data Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
27
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration
19. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
A Data Streaming Platform
is the Underpinning of an Event-driven Architecture
Microservices
DBs
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Streaming Apps
Streaming Apps Streaming Apps
20. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Data in Motion across the Public and Private Sector
Real-time data
without context!
Possible with
simple messaging
solutions.
Real-time data
with context!
What does the
ride cost?
How long does the
driver need?
Data is correlated
in real time.
https://www.confluent.io/events/kafka-summit-europe-2021/development-of-dynamic-pricing-for-tours-using-real-time-data-feeds/
21. kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Customer timetable
Operational
timetable
Assignments
Railway station
knowledge
Dispositions
Train positions
Matching
Aggregation
Consolidation
Apache
Kafka
Analysis
Railway station
Trains
Mobile Apps
Employees
Deutsche Bahn AG | Reisendeninformation
RI-Plattform
22. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
RI-Platform @ Deutsche Bahn
Traveler Information with Single Point of Truth
3rd Party Integration with Google Maps
Real-time train schedule updates
Reach new people and expand business
https://railway-news.com/db-partners-with-google-maps-on-journey-planning/
23. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C C
C
Data Streaming
24. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
TRADITIONAL
DATABASE
EVENT STREAM
PROCESSING
SELECT * FROM
DB_TABLE
CREATE TABLE T
AS SELECT * FROM
EVENT_STREAM
Active Query: Passive Data:
DB Table
Active Data: Passive Query:
Event Stream
25. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Producing to Kafka
P
Time
26. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Consuming from Kafka
P
Time
C2
Near
Real Time
C3
Real Time
C1
Batch
27. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
28. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
Bookings
Team
Fraud
Team
…
Mobile
Team
…
29. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Apache Kafka (kafka.apache.org) includes
Kafka Connect and Kafka Streams
Kafka Streams
Your app
Sink
Source
Kafka
Connect
Kafka
Connect
30. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
ksqlDB
Continuous queries
on infinite streams
31. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agenda
• Traditional Middleware
• Data Streaming Platform
• Enemies
• Friends
• Frenemies
32. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Source A
Source B
Sink X
Sink Y
Connector A
REST
Connector X
MQ
Integration
Layer
Middleware: Message Queue… ETL… ESB… iPaaS
Data Streaming Platform: Apache Kafka. (and some niche players)
Spoilt for Choice!
33. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Business Digitalization Trends are Driving the Need to Process Events at a
whole new Scale, Speed and Efficiency
Why do you need an Data Streaming Platform?
Remember:
Mobile Cloud Microservices Internet of Things Machine Learning
The World has Changed!
34. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
== de facto standard API for OBJECT STORAGE
The market has already decided for Data Streaming…
== de facto standard API for Data Streaming
35. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
36. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Data Streaming Platform - Value per Use Case
Improve Customer
Experience (CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-car
experience: Audi
Customer 360 Simplifying Omni-channel Retail at Scale: Target
Faster transactional processing /
analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation, Splunk
replacement)
Middleware replacement
Regulatory
Digital Transformation
Application Modernization: Multiple Examples
Website / Core Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives; LinkedIn,
Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated environment
(e.g. Electronic Medical Records): Celmatix
Real-time app updates
Real Time Streaming Platform for Communications and
Beyond: Capital One
Developer Velocity - Building Stateful Financial
Applications with Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and Multi-
Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
37. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Data Streaming in the Finance and Insurance Industries
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/
38. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
STREAM
PROCESSING
CONNECTORS
Example Architecture for Data in Motion
ksqlDB
KStreams
Real-time decision making for claim processing and fraud detection
Dashboard
Oracle
DB
Oracle
CDC
CONNECTOR
Salesforce CDC
CONNECTOR
Salesforce
Source / Sink
CONNECTOR
Fraud Detection App
39. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Global Data Streaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking
40. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Why Apache Kafka instead of Traditional Middleware?
Data Streaming Platform
• But also the core is event-based
• Supports real time stream processing
• Fire-and-Forget, Publish / Subscribe, Request-Response / RPC, Batch, …
Single infrastructure
• Messaging, storage, processing
• Scalability and throughput can be increased dynamically
Reliability and zero downtime
• High availability
• Exactly-once semantics
• Rolling upgrades and dynamic configuration changes
• Backwards compatibility
Decoupling of clients
• Agile microservices
• Dumb pipes, smart endpoints
• Handling backpressure
No vendor lock-in
41. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Why Apache Kafka instead of traditional middleware?
”Eat your own dog food!”
42. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Enemies!
ESB MQ
Storage
Streaming
Engine
Messaging: Kafka Core
Storage: Kafka Core
Caching: Kafka Core
Real-Time, Batch: Kafka Clients
Integration: Kafka Connect
Stream Processing: Kafka Streams / KSQL
Request-Response: REST Proxy
”Eat your own dog food”
vs.
Enemy 1
Enemy 2
Enemy 3
Enemy 4
…. More components, clusters, technologies means more conflicts, incompatibility, operations burden!
Enemy 5
43. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
ETL / ESB
Kafka Broker X
Source A
Source B
Sink X
Sink Y
Kafka Connect
Python
Kafka Connect
Java
Kafka
Broker 1
Schema Registry
REST Proxy
Kafka Streams / KSQL
Monitoring Tool “Yet another Kafka
addon”
44. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Don’t use your “ESB knowledge” with Kafka!
https://www.thoughtworks.com/radar/techniques/recreating-
esb-antipatterns-with-kafka
!
45. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design for your Integration Layer
Legacy Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema Registry
Data Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
Data Governance
Security
47. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Cloud-native Kafka
for Business Continuity and Elastic Scale
63
Confluent Cloud
A fully managed, cloud-native service for Apache Kafka
Confluent Platform
A complete, enterprise-grade distribution of Apache Kafka
Cloud-native
Confluent for
Kubernetes
Ansible
Playbooks
Packages:
Docker, RPMs,
Tarball
Public Cloud Workloads On-Premise and Edge Workloads
On Kubernetes On VMs / Bare Metal
48. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
You
Manage
Provider
Managed
Self-managed IaaS Hosted Cloud Service Fully Managed SaaS
Scaling Scaling Scaling
Load balancing Load balancing Load balancing
Partition placement Partition placement Partition placement
Logical Storage Logical Storage Logical Storage
Broker settings Broker settings Broker settings
Zookeeper Zookeeper Zookeeper
Kafka patching Kafka patching Kafka patching
JVM JVM JVM
O/S O/S O/S
VMs VMs VMs
Servers Servers Servers
Provider
managed
features
Product
ease of use
Confluent Cloud
Other Kafka
Services
Apache Kafka
What is a fully-managed SaaS?
49. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Case Study: Advance Auto Parts
50. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Legacy infrastructure and architecture did not meet real-
time performance requirements [Before]
CRM
WMS
MDM
HRM Data Lake Finance
Legacy ITSM
DC MOBILE/WEB
STORE
(PoS)
DC
STORE
(PoS)
Branch (PoS)
ECOMM Catalog
OMS
Inventory
ON PREMISES
Merchandising
(Legacy)
CCDB
CTDB
HJ1
HJ2
JDA
?
51. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Future-proof data architecture for real-time invoicing and
dynamic pricing [After]
CRM
WMS
MDM
HRM Finance
Legacy ITSM
DC MOBILE/WEB
STORE (PoS) DC
STORE (PoS) BRANCH (PoS)
?
Revenue (5)
Location (2)
Pricing (1)
Catalog
Customer (6)
Inventory (4)
Employee (7) Orders (8)
Data Lake
ECOMM Catalog
OMS
Inventory
ON PREMISES
Merchandising
(Legacy)
Products (3)
52. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agenda
• Traditional Middleware
• Data Streaming Platform
• Enemies
• Friends
• Frenemies
53. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Partnerships, not Competitors…
54. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Friends!
• Kafka Connect source / sink connectors
(JMS, IBM MQ, RabbitMQ, etc.)
• JMS Client
(Kafka-native JMS Implementation)
• ESB or ETL tools
with their own Kafka source / sink connectors
• Kafka’s Client APIs
(like Java, .NET, Go, Python, JavaScript)
• REST Proxy
• Etc.
Plenty of integration options between Kafka and traditional middleware
Traditional
Middleware
Apache
Kafka
55. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Why friends?
Kafka is NOT THE ALLROUNDER for every single problem!
Maybe you don’t need a scalable, reliable, distributed system, but “just”:
• Synchronous Web Service (SOAP / WS* or REST) integration between
two endpoints
• Integration with legacy components (Cobol, Edifact, …)
• Point-to-point messaging with active / passive HA for (“a few”) messages
or for XA transactions
• Very specific messaging solution for “less than millisecond performance”
• Managed file transfer
• API Management
• “Real” BATCH Processing (Hadoop, Flink, Informatica, …)
X
56. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Legacy integration and visual coding
à This is where the ESB / ETL tool shines!
• Visual coding for complex
graphical mappings
• Integration components for
complex legacy standard
software and protocols
• SAP BAPI / iDoc, Cobol, Edifact,
SOAP / WS*, etc.
57. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Confluent Stream Designer for Kafka Data Streams
58. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Agenda
• Traditional Middleware
• Data Streaming Platform
• Enemies
• Friends
• Frenemies
59. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Big Bang fails for critical 24/7 deployments
No!
60. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Some technologies never die… Cough…
Can we ever get away from all
legacy applications?
Probably not!
Cobol
61. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Kafka and API Management
in the Enterprise Architecture
Orders Customers
Payments
Stock
REST
JMS
ESB
REST
CRM
Mainframe
SOAP
…
Kafka
Kafka
Kafka
Kafka
SOAP
API Management
HTTP
62. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Frenemies?
How to integrate
the old and new world?
63. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important reason
to consider a strangler fig
application over a cut-over
rewrite is reduced risk.”
Martin Fowler
64. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
65. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDC)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool
66. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New Projects and Applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDC)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool
67. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
70. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
97
71. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Data Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
98
72. Apache Kafka vs. Traditional Middleware (MQ / ETL / ESB / iPaaS) - @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.