The document provides an overview of leveraging mainframe data for modern analytics using Attunity Replicate and Confluent streaming platform powered by Apache Kafka. It discusses the history of mainframes and data migration, how Attunity enables real-time data migration from mainframes, the Confluent streaming platform for building applications using data streams, and how Attunity and Confluent can be combined to modernize analytics using mainframe data streams. Use cases discussed include query offloading and cross-system customer data integration.
2. 2Confidential
Today’s Speakers
Jordan Martz, Director of Technology
Solutions, Attunity
David Tucker, Director of Partner
Engineering, Confluent
Keith Reid, Principal, Insights and Data:
Client Engagement and Practice Leader,
Capgemini
9. 9Confidential
This all changes with streaming / big data platforms (eg Kafka and Hadoop)
Source CDC
Source History
History
In-Memory Analytics
(latest view and events)
Point in Time End of Day
Data Lake
Streaming Platform CEP
10. 10Confidential
So why does CDC work in a Big Data world?
Big Data likes volume and likes history
• Storage isn't an issue
• History helps machine learning
Re-creating any point in time is simple
• 8 lines of Scala code simple
Easiest way to get data without large system performance impacts
• Reduces concerns on data integration
Enables very rapid response to transactional events
• Fraud detection and even consumer response becomes much simpler
18. 18Confidential
Confluent: Open source enterprise streaming built on Apache Kafka
Open Source ExternalCommercial
Confluent Platform
Monitoring
Analytics
Custom Apps
Transformations
Real-time
Applications
…
CRM
Data Warehouse
Database
Hadoop
Data
Integration
Mainframe
Control Center
Auto-data
Balancing
Multi-Data
Center Replication
24/7 Support
Supported
Connectors
Clients
Schema
Registry
REST
Proxy
Apache Kafka
Kafka
Connect
Kafka
Streams
Kafka
Core
Database Changes Log Events loT Data Web Events …
19. 19Confidential
Stream Data is
The Faster the Better
Stream Data can be
Big or Fast (Lambda)
Stream Data will be
Big AND Fast (Kappa)
From Big Data to Stream Data
Apache Kafka is the Enabling Technology of this Transition
Big Data was
The More the Better
ValueofData
Volume of Data
ValueofData
Age of Data
Job 1 Job 2
Streams
Table 1 Table 2
DB
Speed Table Batch Table
DB
Streams Hadoop
21. 21Confidential
Apache KafkaTM Connect – Streaming Data Capture
JDBC
Mongo
MySQL
Elastic
Cassandra
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources Sinks
Fault tolerant
Manage hundreds of
data sources and sinks
Preserves data schema
Part of Apache Kafka
project
Integrated within
Confluent Platform’s
Control Center
22. 22Confidential
Kafka Connect Library of Connectors
* Denotes Connectors developed at Confluent and distributed with the Confluent Platform. Extensive validation and testing has been performed.
Databases
*
Datastore/File Store
*
Analytics
*
Applications / Other
24. 24Confidential
Architecture of Kafka Streams, a Part of Apache Kafka
Kafka
Streams
Producer
Kafka Cluster
Topic TopicTopic
Consumer Consumer
Key benefits
• No additional cluster
• Easy to run as a service
• Supports large aggregations and joins
• Security and permissions fully
integrated from Kafka
Example Use Cases
• Microservices
• Continuous queries
• Continuous transformations
• Event-triggered processes
25. 25Confidential
Kafka Streams: the Easiest Way to Process Data in Apache Kafka™
Example Use Cases
• Microservices
• Large-scale continuous queries and transformations
• Event-triggered processes
• Reactive applications
• Customer 360-degree view, fraud detection, location-
based marketing, smart electrical grids, fleet
management, …
Key Benefits of Apache Kafka’s Streams API
• Build Apps, Not Clusters: no additional cluster required
• Elastic, highly-performant, distributed, fault-tolerant,
secure
• Equally viable for small, medium, and large-scale use
cases
• “Run Everywhere”: integrates with your existing
deployment strategies such as containers, automation,
cloud
Your App
Kafka
Streams
26. 26Confidential
Architecture Example
Before: Complexity for development and operations, heavy footprint
1 2 3
Capture business
events in Kafka
Must process events with separate,
special-purpose clusters
Write results
back to Kafka
Your Processing Job
27. 27Confidential
Architecture Example
With Kafka Streams: App-centric architecture that blends well into your existing infrastructure
1 2 3a
Capture business
events in Kafka
Process events fast, reliably, securely
with standard Java applications
Write results
back to Kafka
Your App
Kafka
Streams
3b
External apps can directly
query the latest results
AppApp
29. 29Confidential
Back to the high-level platform integration …
Mainframe CDC
Source History
History
In-Memory Analytics
(latest view and events)
Point in Time End of Day
Data Lake
Streaming Platform CEP
30. 30Confidential
… made real in Attunity / Confluent Data Flow
Topic Data Flow
• Attunity publishes DB changes to Kafka
• ”Raw” connectors (eg FileSink or HDFS)
persist change records where needed
• K-Streams app reads CDC topic and
transforms (as necessary) for other data
systems.
• Sink connectors (JDBC or K-V as needed)
persist that transformed data for other uses.
Kafka
Streams
Producer
Kafka Cluster
Topic TopicTopic
Consumer Consumer
Data System
Sink
Attunity Replicate
Raw Sink
31. 31Confidential
Use Cases
Query off-load
• Mainframe system accepts
operational updates
• Attunity CDC publishes table
updates to Kafka
• Certified Confluent
Connectors replicate tables
to other data systems for
read-only queries
Business Value
Greater analytics flexibility at
lower cost, without disrupting
operational system
Enhanced security
• Mainframe audit trails
published to Kafka
• Syslog and other access
events published to other
topics
• Event correlation via
LogStash or similar tools
Business Value
Enhanced threat detection
and end-to-end work-flow
auditing
Cross-system integration
• K-Streams application
joins customer data from
mainframe customer-
specific mobile information
• External applications use
interactive queries to
leverage up-to-the-second
customer state
Business Value
Improved customer
engagement, more efficient
marketing spend