Fluvius is the network operator for electricity and gas in Flanders, Belgium. Their goal is to modernize the way people look at energy consumption using a digital meter that captures consumption and injection data from any electrical installation in Flanders ranging from households to large companies. After full roll-out there will be roughly 7 million digital meters active in Flanders collecting up to terabytes of data per day. Combine this with regulation that Fluvius has to maintain a record of these reading for at least 3 years, we are talking petabyte scale. delaware BeLux was assigned by Fluvius to setup a modern data platform and did so on Azure using Databricks as the core component to collect, store, process and serve these volumes of data to every single consumer and beyond in Flanders. This enables the Belgian energy market to innovate and move forward. Maarten took up the role as project manager and solution architect.

1. Building the next-gen digital
meter platform for Fluvius
Maarten Herthoge
Team Lead Data Science & Engineering, delaware BeLux

2. 64% 29% 17%
got measurable results is successful reaches production

3. It’s easy to understand why so many projects fail
Deep Learning
ETL vs ELT
PaaS vs IaaS
Data Visualisation
Data Quality
Master Data
Big Data
Databricks
SMP vs MPP
Data Catalog
Spark
Storm
Data Mart
No-SQL
Cloud vs On-prem
Velocity, Variety and VolumeSemantic Layer
Predictive
Prescriptive
IoT
Streaming
AI

4. Start with the problem,
not with the technology

7. The core data platform needs to support various use cases
Portal access to 3 years of history
Interfaces to allow scalable sharing of data

9. Digital meters collect data… a lot of data
7+ million IoT devices after full roll-out7M+
LEGAL
12TB Sending up to 12 TB of data every day
GDPR and strict government regulations

10. The “golden hammer” does not exist,
use a portfolio approach

11. Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#1 Increasing
volume

12. Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#2 Need to collect and
combine any data

13. Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#3 From data consumer
to data explorer

14. Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#4 Real-time becomes
the standard

15. Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#1 Increasing
volume
#3 From data consumer
to data explorer
#2 Need to collect and
combine any data
#4 Real-time becomes
the standard

16. Prioritize software-as-a-service to create a data portfolio
Managing all data
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PROCESS SERVE
Analyzing all data
Integrating all data
From GB’s
to TB’s
Seamless
scaling
1000+ API
requests/sec
7M+ devices

17. Avoid the data lake pitfall

18. Data lakes often presented as the go-to solution in these cases
Massive scale
Optimized performance
Integration flexibility
Cost effectiveness

19. But data lake implementations fail in 60% of the cases
Massive scale
Optimized performance
Integration flexibility
Cost effectiveness
Becomes data swamp
Capabilities mismatch
Schema-on-read
Misused as “strategy”

20. Focus on collaboration

21. Azure Databricks
Aim for one unified analytics processing engine
Cloud storage
Data warehouses
Big data storage
IoT / streaming data
ML models
BI tools
Data exports
Data warehouses
Collaborative Workspace
DATA
ENGINEER
DATA
SCIENTIST
BUSINESS
ANALYST
Deploy and manage production jobs & workflows
ETL / ELT JOB SCHEDULER MONITORING DEVOPS
Single Runtime Foundation
DATA LAKE
MANAGEMENT
SERVERLESS
/ SaaS
SCALE WITHOUT
LIMITS
PAY-AND-SCALE-
AS-YOU-GO

22. Narrow the scope and decouple storage from serve
INGEST STORE PROCESS SERVE
Why is this successful?
Unified, consistent data ingest path
Cost efficient for full detail
Why is this successful?
Allows for workload-specific platforms
tailored at specific cases independent
from the raw datastore
SQL DB
Service Fabric
Data Lake Gen2
& Delta

23. Databricks acts as glue between a data lake and various
workload specific application platforms
INGEST STORE PROCESS SERVE
Databricks
Why is this successful?
Unified runtime and collaboration
One-click setup
Native integration with Azure services
Enterprise security & SLA
SQL DB
Service Fabric
Data Lake Gen2
& Delta

24. Fluvius’ data portfolio drives innovation
on the Flemish energy market
Bringing consumption insights
and awareness to millions of
households
Lower barrier for data- and
market-driven change
Portfolio projects TCO 20x lower
compared to “golden hammer”

25. Focus on collaboration
Avoid the data lake pitfall
The “golden hammer” does not exist,
use a portfolio approach
Start with the problem,
not with the technology
Interested in more?
Visit delaware.ai

26. Focus on collaboration
Avoid the data lake pitfall
The “golden hammer” does not exist,
use a portfolio approach
Start with the problem,
not with the technology
Thank you!
Maarten Herthoge
TL DS&E, delaware BeLux

27. Feedback
Your feedback is important to us.
Don’t forget to rate
and review the sessions.