Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Kafka Vienna Meetup 020719
Similar a Kafka Vienna Meetup 020719 (20)
Último
Último (20)
Kafka Vienna Meetup 020719
- 1. 1 Apache Kafka Introduction and Use Cases Apache Kafka Meetup Vienna
- 2. 2 About Me • Patrik Kleindl • Solution Architect @ BearingPoint • patrik.kleindl@bearingpoint.com • LinkedIn: https://www.linkedin.com/in/pkleindl/
- 5. 5 Enabling event sharing to connect the world's largest professional network of more than 500 million users Providing on-demand digital content for over 130 millions subscribers accessible worldwide on any device Disrupting the transportation industry by connecting providers with consumers in real-time Pushing breaking news in real time while storing all past publishings online to provide a single source of truth From those that were born cloud-native To more traditional ones that continue to adapt The most Successful Digital Businesses are Inherently Event-driven Leveraging IoT sensors to create an intelligent swarm of connected cars with real-time traffic routing Building a microservices architecture to enable a robust ground transportation booking and management offering
- 6. 6 Mainframes Hadoop Data Warehouse ... Device Logs ... Splunk ... App App Microservice ... Data Stores Custom Apps/MicroservicesLogs 3rd Party Apps Universal Event Pipeline Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Contextual Event-Driven Apps Apache Kafka® STREAMS CONNECT CLIENTS
- 7. 7 Implementing an Event-driven Architecture Requires a Paradigm Shift SaaS appsApps Custom appsMicroservices Relational DB From data represented in static tables... ...to data represented as streams of events Relational DBs Apps Microservices SaaS apps Custom apps Data warehouse Data warehouse
- 8. 8 A Streaming Platform is the Underpinning of an Event-driven Architecture Ubiquitous connectivity Globally scalable platform for all event producers and consumers Immediate data access Data accessible to all consumers in real time Single system of record Persistent storage to enable reprocessing of past events Continuous queries Stream processing capabilities for in-line data transformation Microservice s DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Stream processing apps
- 9. 9 How do I get streams of data into and out of Kafka? Development & Connectivity
- 10. 10Producing to Kafka Time C CC
- 11. 11 Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic! Schema Registry Community Feature
- 12. 12 Apache Kafka Connect API: Import and Export Data In & Out of Kafka JDBC Mongo MySQL Elastic Cassandra HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Integrated within Confluent Control Center Apache Kafka Feature
- 13. Confluent Partner Briefing 13 Apache Kafka Kafka Connect API Reliable and scalable integration of Kafka with other systems – no coding required.
- 14. 14 REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP Simplifies administrative actions Simplifies message creation and consumption Provides a RESTful interface to a Kafka cluster REST Proxy: Talk to Non-native Kafka Apps and Outside the Firewall Community Feature
- 15. 15 How do I build real- time applications? Stream Processing
- 16. 16 Shoulders of Streaming Giants Consumer, Producer KSQL Kafka Streams powers powers Flexibility Ease of Use CREATE STREAM, CREATE TABLE, SELECT, JOIN, GROUP BY, SUM, … KStream, KTable, filter(), map(), flatMap(), join(), aggregate(), … subscribe(), poll(), send(), flush(), beginTransaction(), …
- 17. 17 1 Things Kafka Streams Does Runs everywhere Clustering done for you Exactly-once processing Event-time processing Integrated database Joins, windowing, aggregation S/M/L/XL/XXL/XXXL sizes
- 18. 1818C K O 2 0 1 8 J U L Y Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Walmart Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: ING Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple $↑ $↓ $ Example Case Studies (of many) Example Case Studies Digital Transformation 10 business use case Strategic Driver 20 business use case
- 19. 19 Mainframe in Private DC Confluent-Mainframe Offload Architecture z/OS Connect REST / Web services Applications Visualization tools Access Layer Mainframe Confluent REST Proxy CDC Kafka Connect JDBC Kafka Streams & KSQL Transformations Message Queue (MQ) Search Cloud Data Services Infosphere Data Replicator Confluent Kafka on Public Cloud Confluent Schema Registry Kafka Connect Mainframe + Confluent interconnect Integration & Transformation Confluent Replicator Confluent Schema Registry Public Cloud Services Confluent Platform in the Cloud
- 20. 20 Oracle Exadata in Private DC Confluent - Oracle Exadata to BigQuery Architecture Oracle Golden Gate Connector Applications Visualization tools Access Layer Confluent Replicator Kafka ConnectJDBC Kafka Streams & KSQL Transformations Search Cloud Data Services Confluent Kafka on Public Cloud Confluent Schema Registry Kafka Connect Oracle Exadata + Confluent interconnect Integration & Transformation Transaction log Public Cloud Services Confluent Platform in the Cloud
- 21. 21 Confluent Kafka REST / Web services Transport Layer Integration & Transformation Layer Destinations Data Sources Confluent REST Proxy Network Traffic Confluent Schema Registry Kafka Connect Kafka Connect JDBC/CDC Firewall Logs RDBMS Syslog Application Logs Arcsight, Splunk, etc Modern Monitoring Curated Event Stream Alien Vault ,Protectwise, etc SIEM Accumulo, Graph, etc Modeling Richer insights & archival Curated Event Stream Curated Event Stream Syslog Syslog Legacy Systems KSQL and KStreams • Aggregation • Rules engine • Reference data join • Enrichment • Filtering and Curation Confluent- SIEM Modernization/Offload Architecture Public Cloud Services
- 22. 22Event-Driven Analytics & Machine Learning Production ML App KStreams Confluent Kafka Confluent REST Proxy App Kafka Connect Confluent Schema Registry Model Building KStreams AppAppApps AppAppAppKafka Producers AppAppAppDBs AppAppAppLegacy Systems Training Data Model Params Model Params, Features Output
- 23. 23 Payment Fraud Detection System with Confluent Kafka Producer Elastic search Grafana Kafka Cluster Kafka Connect KSQL Payment App in Car Emergency Fraud System All Data Potential Fraud Apply Analytic Model Filter Predictions At the edge Otherdata Operational Reporting
- 24. 24 Kafka Brokers High retention x/mesg/sec Publish x/mesg/sec Kafka Streams / KSQL part of CEP/Machine Learning workflow Business Applications DB / noSQL / Hadoop / Elastic / Machine Learning Confluent MQTT/Rest Server / ProxyLoad Balancer Firewall Architecture Event-Driven IOT with Confluent Subscribe Kafka Connect