Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry
•
1 recomendación•896 vistas
Kafka Summit SF 2017 by Praveen Hirsave and Rene Parra
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry
Similar a Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry (20)
Más de confluent
Más de confluent (20)
Último
Último (20)
Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry
- 1. DNS for Data The need for a Stream Registry August 28, 2017 Get HomeAway from it all
- 2. Praveen HirsaveRené X. Parra
- 3. “Revolutionizing travel through the power of technology.”
- 4. History
- 5. 2015 Original Twin Drain Problem
- 9. Title Text 2015 Producer Config Topic Config Kafka Monitoring Zookeeper Monitoring
- 10. So we asked for some help 2015
- 11. Then we got acquired! 2016 Migrate to AWS!
- 12. 2016 - Microservices, Microservices, Microservices 2016
- 13. µ2016
- 14. 2016 µ
- 15. 2016 µ • Service Registration • Service Discovery • Health Checks • Circuit Breakers • Resilient - control plane “DNS for µ-services”
- 16. Essence of micro services and streams Commands Queries Events Domain2016 SYNC ASYNC
- 17. ρ2016
- 18. 2017 ρ
- 19. ρ- control plane • Stream Registration • Stream Discovery • Stream Health • Throttling • Multi-Region / Multi-Cloud / Multi-Company “DNS for streams” 2017
- 20. Topic Registration (old, manual way) • A Wiki page • Team • Owner • Consumers (known) • Topic • Schema • Partition Key • Num of Partitions
- 21. Stream Registration • Portfolio • Product • Service • Stream • Schema • Partition Key • Num of Partitions • Config Confluent Schema Registry Stream Registry Topic Manager 2017
- 22. Stream Registration: Modelling Stream Registry 2017 Code First Design First
- 23. Stream Registration: Model Library Stream Registry 2017
- 24. Stream Discovery Stream Registry 2017 Elastic Search Stream Discovery Portal
- 27. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally”
- 28. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally” Consumer {topic}.global Consumer {topic}.global Consume “globally”
- 29. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally” Consumer {topic}.global Consumer {topic}.global Consume “globally” Replicate “locally” Mirror-Maker Mirror-Maker
- 30. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally” Consumer {topic}.global Consumer {topic}.global Mirror-Maker {topic}.global Mirror-Maker Consume “globally” Replicate “globally” Replicate “locally” Mirror-Maker Mirror-Maker
- 31. Stream Registry, Actors and Streams 2017
- 32. Multi-Region Stream Registry Stream Registry Stream Registry DATA-CENTER US-WEST-1 US-EAST-1 2017
- 33. Multi-Cloud Stream Registry Stream Registry Stream Registry 2017
- 34. Multi-Company Stream Registry Stream Registry Stream Registry 2017
- 35. What we wanted Self Service Streams • Log Events • Domain Events • Message Bus • Third Party Clean Data Lake • HDFS • S3 • Google Cloud Storage Streaming Data Kafka 2017
- 36. Recap µ- control plane “DNS for µ-services” ρ- control plane “DNS for ρ-streams”
- 37. Recap µ ρ+ = $
- 38. Key Takeaway Democratizing Streams requires a Stream Registry Stream Registry