Real-time Fraudulent Trips Detection with Xueyao Jiang

1. Real-time Fraudulent Trips Detection Xueyao Jiang

2. FREENOW FEEL FREENOW FEEL FREENOW FEEL FREENOW FEEL FREEN 2 Xueyao Jiang [pronunciation: sh-üe y-ah-oh] ● Principal Data Engineer @ FREENOW ● Leading real-time data streaming @ FREENOW ABOUT ME

3. FREENOW FEEL FREENOW FEEL FREENOW FEEL FREENOW FEEL FREEN 3 Mobility Super App 48 million users 150+ cities 9 market s Taxi PHV e-scooter car- sharing The largest vehicle choice for consumers across Europe

4. AGENDA 01 02 03 04 05 Background Architecture Overview Kafka Streams DSL vs Processor API Kafka Connect Summary 4

5. Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection Fraudulent Trips Detection TAKE RATE UNDER REPORTED VALUE REVENUE LOSS 5

6. 6 BACKGROUND Kafka Streams Use driver gps locations to aggregate distance and duration Passenger onboard Driving to destination Initiate payment Detect under report value Driver add missing value Prediction Model Predict estimated tour value base on aggregated tour duration and distance In real-time Driver inputs trip value into the App

7. 7 GPS and Routes

8. 8 GPS and Routes Haversine formula

9. 9 Why Kafka Streams Tech-wise ● Lightweight ○ Java library ○ No cluster setup required ● Scalable ● Exact once processing ● Low latency Company-wise ● Solid k8s platform ● Limited data engineers

10. 10 ARCHITECTURE OVERVIEW

11. 11 ARCHITECTURE OVERVIEW

12. 12 Kafka Streams DSL ● DSL = Domain Specific Language ● High level & encapsulates most of the stream processing complexity ● Declarative, functional programming style ○ Stateless: filter, map, flatMap etc.. ○ Stateful: reduce, aggregate, join etc..

13. 13 Trips Route DSL - Fairytale World

14. 14 Trips Route DSL - Reality ● Multiple Input Topics ○ passenger-onboard-topic ○ driver-location-topic ○ payment-topic ● Limitations on join

15. 15 Trips Route DSL - Reality

18. 18 Processor API ● Overcome any limitations encounter in DSL ● Fine-grained control ● More granular access to state store ● Advance features e.g. schedule periodic operations

19. 19 Trips Route Processor API ● Transformers over Processors ● Stateful process ● Attach a state store

20. 20 Transformer Code Example

21. 21 Topology Topology is a directed acyclic graph of stream processing operations that represents the stream processing logic

22. 22 Topology Topology is a directed acyclic graph of stream processing operations that represents the stream processing logic

23. 23 Co-partitioning

24. 24 Co-partitioning

25. 25 Co-partitioning ● Records must be keyed with same field ● Topics must contain the same number of partitions

26. 26 Schedule Periodic Cleanup ● driver-location-topic has 200m+ records daily ● Punctuator schedules every 15 mins to delete expired records in state store ● State store is backed up by a changelog topic (retention 1 day default) ● K8s pod has no volume attached

27. 27 Schedule Periodic Cleanup

28. 28 Rebalancing ● Rebalancing happens when a consumer joins or leaves the group ○ Application deployment ○ Application scale up/down ○ k8s pod restarts ● ✅ Kafka static membership + k8s StatefulSet ○ Each k8s pod in a StatefulSet gets a unique, stable hostname (e.g. “route-distance-0”, “route-distance-1”) ○ properties.put(“group.instance.id”, k8s_pod_hostname) ○ Must happen within the “session.timeout.ms” limit

29. 29 Trip-route-topic view

30. 30 Kafka Connect

31. 31 Fallback Mechanism

32. 32 ❤ Kafka Streams ● Comprehensive unit test suite ○ TestInputTopic, TestOutputTopic, TopologyTestDriver ● Extensive metrics reported through JMX ○ Easy to expose to prometheus and build grafana dashboard

33. 33 ❤ Kafka Streams ● Comprehensive unit test suite ○ TestInputTopic, TestOutputTopic, TopologyTestDriver ● Extensive metrics reported through JMX ○ Easy to expose to prometheus and build grafana dashboard

34. 34 ❤🩹 Kafka Streams ● Scala / Java Conversions ● Tons of internal topics to cleanup ● Backfill is not easy ● Learning Curve is steep ○ Strongly recommend book Mastering Kafka Streams and ksqlDB by Mitch Seymour

35. 35 ❤🩹 Kafka Streams ● Scala / Java Conversions ● Tons of internal topics to cleanup ● Backfill is not easy ● Learning Curve is steep ○ Strongly recommend book Mastering Kafka Streams and ksqlDB by Mitch Seymour

36. 36 Key Takeaways ● Low-Level Processor API vs High-Level DSL ● Importance of co-partitioning ○ Records keyed with same field ○ Topics have same number of partitions ● Schedule periodic cleanup state store ● Rebalancing -> “group.instance.id” ● Kafka Connect JDBC Sink Connector upsert

37. FREENOW FEEL FREENOW FEEL FREENOW FEEL FREENOW FEEL FREEN 37 Thank You

Real-time Fraudulent Trips Detection with Xueyao Jiang

Real-time Fraudulent Trips Detection with Xueyao Jiang

Recomendados

Más contenido relacionado

Similar a Real-time Fraudulent Trips Detection with Xueyao Jiang

Similar a Real-time Fraudulent Trips Detection with Xueyao Jiang(20)

Más de HostedbyConfluent

Más de HostedbyConfluent(20)

Último

Último(20)

Real-time Fraudulent Trips Detection with Xueyao Jiang