Pairing socks with ksqlDB, Kafka and Kafka Connect
•
0 recomendaciones•228 vistas
Simon Aubury, Thoughtworks, Principal Data Engineer Sorting socks with a streaming solution. An adventure in pairing my socks using ksqlDB plus transfer learning to build a custom sock image classification model. Kafka plus ksqlDB can retrieve sock classification messages from a camera and perform time-window stream transformations to find pairs of matching socks. https://www.meetup.com/KafkaMelbourne/events/270471169/
Recomendados
Recomendados
Más contenido relacionado
Más de confluent
Más de confluent (20)
Último
Último (20)
Pairing socks with ksqlDB, Kafka and Kafka Connect
- 1. Pairing socks with ksqlDB, Kafka and Kafka Connect linkedin.com/in/simonaubury @SimonAubury github.com/saubury/socksort
- 2. I am Simon Aubury Principal Data Engineer @ ThoughtWorks / Sydney I am here because I love streaming Hello! 2
- 3. @SimonAubury Event Driven sock sorting Pairing socks with Kafka Can I arrange my sock drawer with ksqlDB, Kafka and Kafka Connect? 3 github.com/saubury/socksort
- 5. Events are everywhere Event-driven architecture is an architecture paradigm promoting the production, detection, consumption of and reaction to events. ◉ Object detection is a stream of events 5
- 6. @SimonAubury Events are very messy 6 What I think will happen What actually happens
- 7. @SimonAubury Let’s start at the finish ◉ Hold a sock in front of camera ◉ Stuff happens involving Kafka ◉ ksqlDB displays the location of the other sock 7
- 8. Warning! Some non-Kafka stuff ahead ... 8
- 9. @SimonAubury DeepLens & MQTT BrokerSock ksqlDB Server Stream Processing Kafka Connect Kafka Sock Match Found
- 10. @SimonAubury Deep learning video camera AWS Deeplens AWS Deeplens Hardware ◉ A “deep-learning enabled video camera”. ○ 4MP video camera, ○ Intel Atom Processor ○ 8GB RAM ○ Runs Ubuntu. ◉ Plenty of hardware to help sort my socks. 10
- 11. @SimonAubury Many photos of socks Supervised learning image classification ◉ Requires training data ◉ Prepare a set of training images ◉ I need to take a lot of photos of socks 11
- 12. @SimonAubury Model Training Image classification of socks using transfer learning mode. ◉ Use Amazon Sagemaker image classification algorithm in transfer learning ◉ Deploy a temporary classifier to test the inference function ◉ Test a few demonstration images can be correctly classified 12
- 13. @SimonAubury Inference Lambda Function ◉ Run all images captured by the camera through the classification model. ○ Review a live camera feed within as web-browser ○ OpenCV adds text overlaid on the image. ◉ Write to a MQTT topic 13
- 15. @SimonAubury What is MQTT? ◉ MQTT is lightweight TCP/IP protocol ○ Small footprint ○ Low power ◉ MQTT acts more like a key/value store ○ Whereas Kafka is a complete streaming platform. 15
- 17. @SimonAubury DeepLens & MQTT BrokerSock ksqlDB Server Stream Processing Kafka Connect Kafka Sock Match Found
- 18. @SimonAubury Kafka connect Kafka Connect is a framework for streaming data between Apache Kafka and other data systems ○ Kafka connect sources – from something into Kafka ○ Kafka connect sinks – from Kafka into something . 18 MQTT driver MQTT source
- 19. @SimonAubury MQTT to Kafka with Kafka Connect 19 ◉ Save config to file ◉ Manage the secrets ◉ Find the end point of connect server ◉ Fire up curl/postman to submit config ◉ … do more stuff to check
- 20. @SimonAubury KSQL to ksqlDB ◉ KSQL - build event streaming applications “with SQL” ◉ ksqlDB – KSQL with ○ Push and Pull queries ○ Connector management 20
- 21. @SimonAubury Kafka Connect with ksqlDB ksqlDB now has commands so you can directly setup and control Kafka Connect. 21
- 22. @SimonAubury MQTT to Kafka with Kafka Connect by using ksqlDB 22 ksqlDB command credentials.properties
- 23. @SimonAubury Do we have any socks in Kafka? We can check incoming MQTT messages are landing in Kafka by querying the Kafka topic kafka-console-consumer 23 kafka-console-consumer --bootstrap-server localhost:9092 --topic data_mqtt --from- beginning
- 24. @SimonAubury Do we have any socks in Kafka? We can check incoming MQTT messages are landing in Kafka by querying the Kafka topic with KSQL 24
- 25. @SimonAubury DeepLens & MQTT BrokerSock ksqlDB Server Stream Processing Kafka Connect Kafka Sock Match Found
- 26. @SimonAubury Create a stream with ksqlDB Goal: create a stream for our Kafka topic containing JSON 26
- 27. @SimonAubury Ghost socks Image classifier identifies 3-4 images each second. 27
- 28. @SimonAubury Stream processing with ksqlDB Goal: find similar messages within a windows of time 28
- 29. @SimonAubury Window Hopping with ksqlDB Goal: find 4 or more identical socks in a rolling 5 second window 29
- 30. @SimonAubury Staring at the wall with ksqlDB Goal: eliminate pictures of the wall 30
- 31. @SimonAubury Pairing socks with ksqlDB Goal: find pairs of identical socks 31
- 32. @SimonAubury Place your screenshot here Demo 32
- 33. What did I learn? 33
- 34. @SimonAubury What did I learn? ◉ Object detection is a stream of events ◉ Events are very messy ○ Stream processing doesn’t need to be ◉ Kafka connect is cool and easy way to integrate systems ○ Even easier to manage with ksqlDB ◉ IoT architecture with Kafka is really scalable ◉ Sock sorting does not make you interesting at parties 34
- 35. Any questions ? linkedin.com/in/simonaubury github.com/saubury/socksort Presentation template by SlidesCarnival Thanks! 35 @SimonAubury