Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

1. Mainframe Integration, Offloading and Replacement
with Apache Kafka
Stand up to the Dinosaur!
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
confluent.io
kai-waehner.de

2. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Data in Mo:on
3. Apache Ka@a as Cloud-na:ve and Hybrid Infrastructure
4. Mainframe Integra:on, Offloading and Replacement
5. Case Study
Agenda

3. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Data in Motion
3. Apache Kafka as Cloud-native and Hybrid Infrastructure
4. Mainframe Integration, Offloading and Replacement
5. Case Study
Agenda

4. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
The Mainframe is here to stay!
“Mainframes are still hard at work,
processing over 70 percent of the world’s
most important computing transactions
every day. Organizations like banks, credit
card companies, airlines, medical facilities,
insurance companies, and others that can
absolutely not afford downtime and errors
depend on the mainframe to get the job
done. Nearly three-quarters of all Fortune
500 companies still turn to the mainframe to
get the critical processing work completed”
https://www.bmc.com/blogs/mainframe-mips-an-introduction/

5. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
What is a Mainframe?
Modern mainframe design is characterized less by raw
computa8onal speed and more by:
• High reliability and security
• Extensive input-output ("I/O") facili8es with the ability to
offload to separate engines
• Strict backward compa8bility with older soGware
• High hardware and computa8onal u8liza8on rates through
virtualiza8on to support massive throughput
• Hot-swapping of hardware, such as processors and memory
Vendors: “IBM and the Seven Dwarfs”
The IBM z15, announced in 2019,
with up to 40TB RAM and 190 Cores,
typically costs millions $$$
(variable software costs not included)

6. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Neobanks and FinTechs Hunting the Traditional Banks
Monolithic
Proprietary
Complex
Inflexible

7. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Not just the FinTechs Modernize their Architecture!
9
https://www.confluent.io/kafka-summit-london18/distributing-computing-key-player-in-corebanking-platforms/
Sberbank re-implemented their Core Banking Pla6orm
around Ka9a and Event Streaming to be ready for the future!

8. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
MIPS (million instructions per second)
to normalize CPU usage across CPU types and models or hardware configs
MSU (million service units)
hardware and software metrics calculated directly by the operating system
12

9. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
No Cobol developers out there!
13
https://edition.cnn.com/2020/04/08/business/coronavirus-cobol-programmers-new-jersey-trnd/index.html

10. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Huge demand to build an open, flexible, scalable platform
• Real-&me
• Scalability
• High availability
• Decoupling
• Cost-efficient
• Flexibility
• Elas&city
• Standards-based
• Extensibility
• Security
• Infrastructure-independent
• Mul&-region / global

11. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Data in Motion
3. Apache Kafka as Cloud-native and Hybrid Infrastructure
4. Mainframe Integration, Offloading and Replacement
5. Case Study
Agenda

12. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Data in Motion in Financial Services
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/

13. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Data in Motion for
Traditional and New Innovative Use Cases
in Financial Services
Real Time Processing Digital Transforma3on Strategic Goals
Short-Sale Risk Calculation
/ Trade Approval
Mainframe Offloading and
Replacement
Credit Card
Fraud Detection
Next-Best
Offer
Robotic Process Automation
(e.g. Know Your Customer, KYC)
Customer Service
(e.g. Chat Bots)
IT ModernizaNon
Regulatory
Reporting
Account Login
Fraud Detection
Anomaly Detection
across Assets and Locations
Derivatives
Pricing Compliance
Trading
Post-Processing
Strategic Planning
and Simulations

14. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
https://www.confluent.io/customers/rbc/
“… rescue data off of the mainframe, in a cloud native, microservice-
based fashion … [to] … significantly reduce the reads on the mainframe,
saving RBC fixed infrastructure costs (OPEX). RBC stayed compliant with
bank regulations and business logic, and is now able to create new
applications using the same event-based architecture.”

15. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Data in Motion
3. Apache Kafka as Cloud-native and Hybrid Infrastructure
4. Mainframe Integration, Offloading and Replacement
5. Case Study
Agenda

16. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C C
C
Data in Motion

17. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
23
An Event Streaming Platform
is the Underpinning of an Event-driven Architecture

18. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Ka#a
C++ Client
Kafka Cluster
Monolith
Kafka Decouples Storage and Compute
Cloud DWH
Snowflake Connector
Schema Registry
Storage
Compute Compute
(+ non-Kafka Storage)
KSQL
App
KSQL
App
KSQL
App
Compute
Compute Compute

19. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Disaster Recovery – RPO and RTO
RPO = Recovery Point Objective
RTO = Recovery Time Objective

20. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
IBM GDPS
● Synchronous and asynchronous mirroring
● Different RTO / RPO setups
● Built for resiliency and disaster recovery
● Supports multiple sites
● Uses concepts like XCF (Cross-system
Coupling Facility), Parallel Sysplex, Disk
Mirroring, HyperSwap, etc.
● Independent of transaction manager (e.g.
CICS, IMS) or database manager (e.g. DB2 ,
IMS, VSAM)
https://www.ibm.com/it-infrastructure/z/technologies/gdps
https://ibmsystemsmag.com/IBM-Z/07/2019/resiliency-gdps-solutions

21. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Global Event Streaming with Kafka
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking

22. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Multi-Region Cluster
(Only available in Confluent Platform)
Kafka Clusters
can Stretch over Regions
Zero Downtime + Zero Data loss
(RPO = 0 and RTO ~ 0)
e.g. Stretched over US East + Mid + West
Automate Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without Custom Code

23. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Example of a Multi-Region Cluster in a Bank
Large FinServ Customer
Payment
Log
Payment
Log
Location Loca-on
synchronous
asynchronous
● ‘Payment’ transactions enter from us-
east and us-west with fully
synchronous replication
● ‘Log’ and ‘Location’ information in the
same cluster use async - optimized for
latency
● Automated disaster recovery (zero
downtime, zero data loss)
Result: Clearing time from ‘deposit’ to
‘available’ goes from 5 days to 5 seconds
(including security checks)

24. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Data in Motion
3. Apache Kafka as Cloud-native and Hybrid Infrastructure
4. Mainframe Integration, Offloading and Replacement
5. Case Study
Agenda

25. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe Offloading
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU

26. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe Replacement
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU

27. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Mainframe Replacement (without Writing New Apps)
https://learnworthy.net/could-java-be-the-next-cobol/
https://medium.com/@FranzRoses/the-enterprise-journey-to-decompose-the-cobol-banking-core-into-java-the-developer-perspective-2e8a53bb528e
https://bs2manuals.ts.fujitsu.com/psBEANCONNECTV65en/beanconnect-user-guide-user-guide-13821/cobol2java-bc-ug-499/mapping-cobol-data-types-to-java-classes-bc-ug-500
What about
the 5% of
(complex) code
that cannot
be migrated
automatically?
Showstopper?

28. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Orders Customers
Payments
Stock
WebSockets / SSE
JMS
ESB
REST
Java
Connect
RPC
MQ
Integration between Kafka and Mainframes
MQ
C++ File
???

29. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Integration Options between Kafka and Mainframe
• IBM InfoSphere Data Replication (IIDR) Change Data Capture (CDC) solution for Mainframe
• Open-source CDC solution (e.g. Debezium) – but you still need an IIDR license, this is the same challenge as with Oracle
and GoldenGate CDC)
• 3rd Party middleware solution (e.g. DBS-H, B.O.S. Software, …) – some provide much more than just CDC for mainframe
integration
• DB2 SQL Integration: Create interface tables + Kafka Connect + JDBC connector
• IBM MQ interface + Kafka Connect’s IBM MQ connector
• VSAM File Integration / CDC (e.g. B.O.S, Luminex, Syncsort/Precisely, Qlik Replicate, IBM InfoSphere CDC, …)
• Confluent REST Proxy, HTTP Sink connector and HTTP(S) communication with the mainframe
• Kafka Client APIs on the mainframe
Evaluate the performance / scalability / feature set / cost
of the tools and the footprint on the mainframe!
Don’t underestimate politics…

30. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Change Data Capture
Transaction log-based CDC pushes data changes (insert, update, delete) from the
mainframe database to Kafka in real time.
+ Real time push updates to Kafka
+ Eliminate disruptive full loads, i.e. minimize production impact
+ Reduce MIPS consumption
+ Full support
+ IIDR or 3rd Party tools
- Typically very high licensing costs
- Referential integrity between related tables

31. Mainframe Integra?on, Offloading and Replacement with Apache KaAa – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Integration of Kafka and Mainframe Applications via IBM DB2 or IBM MQ
Mainframe
42
CICS
Batch I/O
Requests
CICS
Program
Failure
Flow
CICS
Program
Cloud
On-Prem
JDBC
Connector
z/OS
MQ Connect Worker
running on zIIP

32. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
44
HTTP(S)
Mainframe
Cobol or other ApplicaNon Code
Confluent Cluster
REST API
Client App with
HTTP Kafka Producer /
HTTP Kafka Consumer
HTTP(S)
REST API for Kafka
Workaround for the Integration of Kafka and Mainframe Applications

33. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
1. Mainframe Status Quo and Challenges
2. Use Cases for Data in Motion
3. Apache Kafka as Cloud-native and Hybrid Infrastructure
4. Mainframe Integration, Offloading and Replacement
5. Case Study
Agenda

34. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Cloud
Adop4on
Journey from Mainframe
to Hybrid and Cloud
PHASE 3
Hybrid
Cloud
Cloud-First
Development
PHASE 2
PHASE 1
https://www.accenture.com/_acnmedia/pdf-70/accenture-moving-to-the-cloud-strategy-for-banks-in-north-america.pdf
Case Study - Bank CEO
“This is the last 5-year $20M IBM contract.
Get rid off the mainframe!”

35. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important
reason to consider a
strangler fig application
over a cut-over rewrite
is reduced risk.”
Martin Fowler

36. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)

37. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

38. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New Projects and Applications
Applica3on
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

39. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14

40. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
What about Transactions?
53

41. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
IBM Mainframe Database and Transaction Managers
54
IMS
• Hierarchical Database
• Transaction Manager
• Supports Cobol, Assembler, PL/1, Java
• IMS Connect for Integration with WebSphere MQ, SOAP, …
DB2
• Relational Database
CICS
• Transaction Manager
• Database “Lite” (VSAM Datasets)
• Integration and Application Programming Capabilities similar to IMS,
but much easier to use
• Advanced Features like Transaction Prioritization
The Heart
of your
Business
App

42. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
“Transactions” in Apache Kafka
55
Exactly-Once Semantics (EOS)
available since Kafka 0.11:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka/

43. Why Confluent

44. I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
60

45. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
61

46. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

47. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
IBM + Confluent for Mainframe Integration
64

48. Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!