Worldsensing: A Real World Use Case for Flux by Albert Zaragoza, CTO & Head of Engineering, Fuad Mimoun, Software Developer and Daniel Lázaro Iglesias, Software Developer & Team Lead | Worldsensing
•
0 recomendaciones•501 vistas
Albert Zaragoza, Fuad Mimoun and Daniel Lázaro from Worldsensing will be sharing how their team built an end-to-end IoT solution for cities — from traffic flow management and smart parking to emergency & security response and critical infrastructure monitoring. They will focus this talk on how they used Flux to pull together lots of data sources into their real-time platform to provide alerts to the many constituents of the data.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Worldsensing: A Real World Use Case for Flux by Albert Zaragoza, CTO & Head of Engineering, Fuad Mimoun, Software Developer and Daniel Lázaro Iglesias, Software Developer & Team Lead | Worldsensing
Similar a Worldsensing: A Real World Use Case for Flux by Albert Zaragoza, CTO & Head of Engineering, Fuad Mimoun, Software Developer and Daniel Lázaro Iglesias, Software Developer & Team Lead | Worldsensing (20)
Más de InfluxData
Más de InfluxData (20)
Último
Último (20)
Worldsensing: A Real World Use Case for Flux by Albert Zaragoza, CTO & Head of Engineering, Fuad Mimoun, Software Developer and Daniel Lázaro Iglesias, Software Developer & Team Lead | Worldsensing
- 1. A Real World Use Case for Flux Influx Days London 2019
- 2. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 3. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 4. TRANSFORM OPERATIONS WITH REAL TIME INTELLIGENCE #BEYONDSMARTCITIES
- 5. Department IT silos Data exchange between IT silos Smart Cities Connected Operational Intelligence There is an ongoing shift from data silos to CONNECTED OPERATIONAL INTELLIGENCE
- 6. THE BEST CITIES & INFRASTRUCTURE PROJECTS ARE REINVENTING THEMSELVES
- 7. 1 DIGITIZE 2 MONITOR 4 ACT 3 PREDICT Connect your assets & infrastructures Gain actionable insights Make better, faster decisions Achieve operational efficiency 5 ENGAGE Involve citizens and workers in the process They leverage CONNECTED OPERATIONAL INTELLIGENCE to act based on real time data and to predict anomalies.
- 8. It is a new way of thinking Silos Connected Intelligencevs
- 9. HOW WORLDSENSING IS HELPING CITIES & CONSTRUCTION SITES?
- 10. We build sensors, systems & business applications. End to end CONNECTED OPERATIONAL INTELLIGENCE SYSTEMS SENSORS BITCARRIER PREDICTION ANOMALIESPREDICTIONS ANOMALIES BITCARRIER Apps & 3rd Party Services WAZE, TOMTOM, GOOGLE PUBLIC APIs OTHER DATA SOURCES INCIDENTS LOADSENSINGFASTPRK CCTV WEATHER STATIONS VEHICLE COUNTING SENSORS 3rd PARTY SENSORS WORLDSENSING BUSINESS APPLICATIONS
- 11. MONITORING MOBILITY, PARKING, SECURITY AND TRAFFIC IN 60 CITIES COLLECTING DATA FROM 10,000 SENSORS ON 200 CONSTRUCTION SITES AND 100 CRITICAL INFRASTRUCTURES SUPPORTING DECISION MAKING IN 6 MAJOR CONSTRUCTION SITES ENSURING THAT 50 MINES ARE SAFE AND MAINTAINED
- 13. In today’s fast-paced and data-driven economies, many companies are still dependent on traditional methods to gather data.
- 14. We use Industrial Internet of Things sensing technology WIRELESS LONG-RANGE LOW-POWER
- 15. THE WIRELESS MONITORING SYSTEM ▪ Leading global pioneer in the wireless sensing industry ▪ Battery-powered, long-range, low-power devices compatible with a wide range of geotechnical sensors ▪ With a web-based software and mobile app that facilitates real-time data capture, analytics and alarm configuration
- 16. How it works System Architecture DATA COLLECTION COMMUNICATION VISUALIZATION How it works System Architecture DATA COLLECTION COMMUNICATION VISUALIZATION
- 17. How it works in Mines and Dams
- 18. Ponte Vecchio, Florence, Italy
- 20. Crossrail tunnel, London, UKCrossrail tunnel, London, UK
- 22. Rail network tracks and surroundings
- 23. THE LEADING OUTDOOR PARKING MANAGEMENT SYSTEM
- 24. Smart Parking ecosystem Digitize Detection Monitor Real-time Act Instantly Predict Parking Behavior Engage Citizens Parking occupancy is detected using sensors Drivers are guided to available parking slots through Dynamic Message Signs (DMS) or a Mobile app Once parked, drivers may activate their parking timers/meters and pay also through the app. Parking enforcers are guided to areas with high fraud probability through a Guided Enforcement App. Cities may also plan out future parking requirements through information gathered on a Software Suite. Drivers may also send incident reports and access other information such as traffic conditions through the app. Smart Parking ecosystem Digitize Detection Monitor Real-time Act Instantly Predict Parking Behavior Engage Citizens Parking occupancy is detected using sensors Drivers are guided to available parking slots through Dynamic Message Signs (DMS) or a Mobile app Once parked, drivers may activate their parking timers/meters and pay also through the app. Parking enforcers are guided to areas with high fraud probability through a Guided Enforcement App. Cities may also plan out future parking requirements through information gathered on a Software Suite. Drivers may also send incident reports and access other information such as traffic conditions through the app. Smart Parking ecosystem Digitize Detection Monitor Real-time Act Instantly Predict Parking Behavior Engage Citizens Parking occupancy is detected using sensors Drivers are guided to available parking slots through Dynamic Message Signs (DMS) or a Mobile app Once parked, drivers may activate their parking timers/meters and pay also through the app. Parking enforcers are guided to areas with high fraud probability through a Guided Enforcement App. Cities may also plan out future parking requirements through information gathered on a Software Suite. Drivers may also send incident reports and access other information such as traffic conditions through the app.
- 27. Digitize Detection Monitor Real-time Act Instantly Predict Driver Behavior Engage Citizens Detect incidents from citizen reports using an app, sensors, cameras, or field agents. Monitor traffic anomalies real-time through an incident management platform Deploy emergency services and tow trucks and notify drivers through multiple channels and TMC Gather feedback and provide value added services through a mobile app Traffic Ecosystem Anticipate traffic build-up through historical data and trigger re-routes or speed limits to manage vehicle volume. Digitize Detection Monitor Real-time Act Instantly Predict Driver Behavior Engage Citizens Detect incidents from citizen reports using an app, sensors, cameras, or field agents. Monitor traffic anomalies real-time through an incident management platform Deploy emergency services and tow trucks and notify drivers through multiple channels and TMC Gather feedback and provide value added services through a mobile app Traffic Ecosystem Anticipate traffic build-up through historical data and trigger re-routes or speed limits to manage vehicle volume. Digitize Detection Monitor Real-time Act Instantly Predict Driver Behavior Engage Citizens Detect incidents from citizen reports using an app, sensors, cameras, or field agents. Monitor traffic anomalies real-time through an incident management platform Deploy emergency services and tow trucks and notify drivers through multiple channels and TMC Gather feedback and provide value added services through a mobile app Traffic Ecosystem Anticipate traffic build-up through historical data and trigger re-routes or speed limits to manage vehicle volume. Digitize Detection Monitor Real-time Act Instantly Predict Driver Behavior Engage Citizens Detect incidents from citizen reports using an app, sensors, cameras, or field agents. Monitor traffic anomalies real-time through an incident management platform Deploy emergency services and tow trucks and notify drivers through multiple channels and TMC Gather feedback and provide value added services through a mobile app Traffic Ecosystem Anticipate traffic build-up through historical data and trigger re-routes or speed limits to manage vehicle volume. Digitize Detection Monitor Real-time Act Instantly Predict Driver Behavior Engage Citizens Detect incidents from citizen reports using an app, sensors, cameras, or field agents. Monitor traffic anomalies real-time through an incident management platform Deploy emergency services and tow trucks and notify drivers through multiple channels and TMC Gather feedback and provide value added services through a mobile app Traffic Ecosystem Anticipate traffic build-up through historical data and trigger re-routes or speed limits to manage vehicle volume.
- 28. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 30. An Integrated Solution for City Mobility DepartmentsAn Integrated Solution for City Mobility Departments
- 31. OneMind Architecture CORE Connector 1 Control Panel (FE) Connector M External API 1 External API 2 Simulator or JSON Connector M-1 External Source N Custom object service Vertical Vertical Vertical OneMind Architecture CORE Connector 1 Control Panel (FE) Connector M External API 1 External API 2 Simulator or JSON Connector M-1 External Source N Custom object service Vertical Vertical Vertical
- 32. OneMind - Demo TimeOneMind - Demo Time
- 33. OneMind - Demo TimeOneMind - Demo Time
- 34. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 35. From Ops to Software development ○ TICK for monitoring ■ Google cloud deployments ■ Tens of docker containers per installation ● Basic installation with docker compose ● Moving to Kubernetes ■ Monitor machine and container resources ■ Visualization with Chronograf ○ Quick interfaces for prototypes and innovation projects ■ Insert time-series data in InfluxDB ■ Visualize with Grafana or Chronograf ■ Processing our alerts with Kapacitor TICK at Worldsensing From Ops to Software development ○ TICK for monitoring ■ Google cloud deployments ■ Tens of docker containers per installation ● Basic installation with docker compose ● Moving to Kubernetes ■ Monitor machine and container resources ■ Visualization with Chronograf ○ Quick interfaces for prototypes and innovation projects ■ Insert time-series data in InfluxDB ■ Visualize with Grafana or Chronograf ■ Processing our alerts with Kapacitor TICK at Worldsensing
- 36. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 37. Use cases: generic time-series data (I) Custom Object Service: A service that allows us to insert static and generic data in our platform. It allows us to show the geolocated data in our map and also a popup visualization with the current sensor data. Examples: ● Tiltmeter inclination values in degrees, ● Environment data ● Variable message panels ● CCTV Cameras ● ... Use cases: generic time-series data (I) Custom Object Service: A service that allows us to insert static and generic data in our platform. It allows us to show the geolocated data in our map and also a popup visualization with the current sensor data. Examples: ● Tiltmeter inclination values in degrees, ● Environment data ● Variable message panels ● CCTV Cameras ● ...
- 38. COS (1) Register Schema (2) Send static data matching the schema (3) Save Static data Use cases: generic time series data (II) Connector Subsystem or Sensor Maps data visualizations Popup data visualizations Example: Variable Message Signs COS (1) Register Schema (2) Send static data matching the schema (3) Save Static data Use cases: generic time series data (II) Connector Subsystem or Sensor Maps data visualizations Popup data visualizations Example: Variable Message Signs
- 39. Use cases: generic time series data (III) Add time series data to COS: ○New custom type - time series ○ Stored in influx ○ Values: ■ Time series fields ○ Tags: ■ object_type ■ id ■ other custom fields if needed Example: environmental data (air quality, tilt measurements, etc...) Use cases: generic time series data (III) Add time series data to COS: ○New custom type - time series ○ Stored in influx ○ Values: ■ Time series fields ○ Tags: ■ object_type ■ id ■ other custom fields if needed Example: environmental data (air quality, tilt measurements, etc...)
- 40. COS (1) Register Schema (2) Send time series data Time series data Maps data (Coordinates) Use cases: generic time series data (III) Connector COS (1) Register Schema (2) Send time series data Time series data Maps data (Coordinates) Use cases: generic time series data (III) Connector
- 41. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 42. Use cases: alerts (I) Detect anomalies and trigger alerts in a city Use cases: alerts (I) Detect anomalies and trigger alerts in a city
- 43. Use cases: alerts (II) Kapacitor as simple business rules engine Queue Consumer API Historical data in Influx: Vehicles speed (kmh), tiltmeter data (degrees), etc... Kapacitor scripts to detect alerts: Simple threshold rules Kapacitor udf: send data to queues. Use cases: alerts (II) Kapacitor as simple business rules engine Queue Consumer API Historical data in Influx: Vehicles speed (kmh), tiltmeter data (degrees), etc... Kapacitor scripts to detect alerts: Simple threshold rules Kapacitor udf: send data to queues. Use cases: alerts (II) Kapacitor as simple business rules engine Queue Consumer API Historical data in Influx: Vehicles speed (kmh), tiltmeter data (degrees), etc... Kapacitor scripts to detect alerts: Simple threshold rules Kapacitor udf: send data to queues.
- 44. Use cases: alerts (III)Use cases: alerts (III)
- 45. Use cases: alerts (IV) Possible Flux implementation: ○Define function with parameters ○Create task calling the defined function with desired parameters ○ No templates for now Use cases: alerts (IV) Possible Flux implementation: ○Define function with parameters ○Create task calling the defined function with desired parameters ○ No templates for now
- 46. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 47. Use cases: aggregations for parking data (I) Parking as core vertical: ● KPIs ○ Occupancy percentage ○ Turnover ○ Session time ● Aggregate data by ○ Area ■ Sector ■ District ■ City ○ Time ■ Hourly average ■ Daily average Use cases: aggregations for parking data (I) Parking as core vertical: ● KPIs ○ Occupancy percentage ○ Turnover ○ Session time ● Aggregate data by ○ Area ■ Sector ■ District ■ City ○ Time ■ Hourly average ■ Daily average
- 48. Use cases: aggregations for parking data (II) Implementation ● Historical data in influx - OK ● Aggregation queries in influx - OK ● Aggregations in kapacitor - not OK ○ Complex aggregations, difficult to implement ○ Hard to debug ○ Replaced with cron jobs in python and InfluxQL queries Use cases: aggregations for parking data (II) Implementation ● Historical data in influx - OK ● Aggregation queries in influx - OK ● Aggregations in kapacitor - not OK ○ Complex aggregations, difficult to implement ○ Hard to debug ○ Replaced with cron jobs in python and InfluxQL queries
- 49. Use cases: aggregations for parking data (III)Use cases: aggregations for parking data (III)
- 50. Use cases: aggregations for parking data (IV) Two Flux alternatives: Pivot Join Use cases: aggregations for parking data (IV) Two Flux alternatives: Pivot Join
- 51. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 52. Use cases: Anomaly Detection (I) Detect anomalies in e.g. traffic data ● Custom forecast algorithm based on historical data ● Compare real-time values to forecasted ● Compute “anomaly score” Anomaly detection in Kapacitor examples (HoltWinters prediction) So… let’s try it! Use cases: Anomaly Detection (I) Detect anomalies in e.g. traffic data ● Custom forecast algorithm based on historical data ● Compare real-time values to forecasted ● Compute “anomaly score” Anomaly detection in Kapacitor examples (HoltWinters prediction) So… let’s try it!
- 53. Use cases: Anomaly Detection (II) ● Implementing forecast algorithm in Kapacitor ○ Discarded because ■ It’s based on historical data, so we can predict in batch values for the next day, week… ■ Too complex to put it as a UDF (use pandas, scikit…) Forecast algorithm Predicted Speed data Predict week Real time speed data Cron job Weighted moving average Averaged speed data UDF Use cases: Anomaly Detection (II) ● Implementing forecast algorithm in Kapacitor ○ Discarded because ■ It’s based on historical data, so we can predict in batch values for the next day, week… ■ Too complex to put it as a UDF (use pandas, scikit…) Forecast algorithm Predicted Speed data Predict week Real time speed data Cron job Weighted moving average Averaged speed data UDF
- 54. Use cases: Anomaly Detection (III) ● Compare current value with predicted to compute trend deviation/anomaly score ○ Need to read two data sources (real and predicted) ○ Couldn’t get to join them in a tickscript ○ Implemented as UDF that ■ Receives a stream ■ Queries InfluxDB from python Batch query Stream Predicted Speed data Compute score Anomaly score data Kapacitor task: Join and compute score UDF Averaged speed data Use cases: Anomaly Detection (III) ● Compare current value with predicted to compute trend deviation/anomaly score ○ Need to read two data sources (real and predicted) ○ Couldn’t get to join them in a tickscript ○ Implemented as UDF that ■ Receives a stream ■ Queries InfluxDB from python Batch query Stream Predicted Speed data Compute score Anomaly score data Kapacitor task: Join and compute score UDF Averaged speed data
- 55. Use cases: Anomaly Detection (IV) ● Send computed values to a queue ○ Same UDF than for alerts ○ Can be taken by the Traffic vertical or other services in the system Compute score Anomaly score data Queue Traffic vertical Use cases: Anomaly Detection (IV) ● Send computed values to a queue ○ Same UDF than for alerts ○ Can be taken by the Traffic vertical or other services in the system Compute score Anomaly score data Queue Traffic vertical
- 56. Use cases: Anomaly Detection (V) Forecast algorithm Batch query Stream Real time speed data Predicted Speed data Speed sensor Compute score Anomaly score data Predict week Use historical speed data Kapacitor task: Join and compute score UDF Queue Traffic vertical Weighted moving average Averaged speed data UDF Use cases: Anomaly Detection (V) Forecast algorithm Batch query Stream Real time speed data Predicted Speed data Speed sensor Compute score Anomaly score data Predict week Use historical speed data Kapacitor task: Join and compute score UDF Queue Traffic vertical Weighted moving average Averaged speed data UDF
- 57. Use cases: Anomaly Detection (VI) Forecast algorithm Batch query Stream Real time speed data Predicted Speed data Speed sensor Compute score Anomaly score data Predict week Use historical speed data Kapacitor task: Join and compute score UDF Queue Traffic vertical Weighted moving average Averaged speed data Flux script Use cases: Anomaly Detection (VI) Forecast algorithm Batch query Stream Real time speed data Predicted Speed data Speed sensor Compute score Anomaly score data Predict week Use historical speed data Kapacitor task: Join and compute score UDF Queue Traffic vertical Weighted moving average Averaged speed data Flux script
- 58. Use cases: Anomaly Detection (VI) Replaced UDF with Flux script Use cases: Anomaly Detection (VI) Replaced UDF with Flux script
- 59. ○Introduction ○ Worldsensing ○ OneMind ○ TICK at Worldsensing - use cases ○Advanced use cases ○ Generic time-series data ○ Alerts ○ Aggregations (parking data) ○ Anomaly detection ○Conclusions Agenda
- 60. Conclusions We have tried InfluxDB and Kapacitor in several application-level use cases Pros: ● InfluxDB is great for time series queries ● Kapacitor is good for ○ Simple triggers (alerts, etc) ○ Simple processing of data streams ● UDFs provide a lot of flexibility Cons: ● Kapacitor is hard to test and debug, so long and complex tickscripts are not recommended ○ Using it as a scheduler to query InfluxDB and process in python isn’t worth it
- 61. Conclusions (Flux) We have tried InfluxDB 2.0 with Flux in several application-level use cases Pros: ● Replaces almost all of Kapacitor use cases ● Cleaner, more organized code (functions) ● Repeatable execution ● Testable (asserts) Cons: ● Some queries may be simpler with InfluxQL ● No support for UDFs ● No task templates ● Other missing functions from InfluxQL/Kapacitor, like “elapsed”
- 62. Thanks! Check our tech blog @ https://medium.com/worldsensing-techblog