1.
Building the next-gen digital
meter platform for Fluvius
Maarten Herthoge
Team Lead Data Science & Engineering, delaware BeLux

2.
64% 29% 17%
got measurable results is successful reaches production

3.
It’s easy to understand why so many projects fail
Deep Learning
ETL vs ELT
PaaS vs IaaS
Data Visualisation
Data Quality
Master Data
Big Data
Databricks
SMP vs MPP
Data Catalog
Spark
Storm
Data Mart
No-SQL
Cloud vs On-prem
Velocity, Variety and VolumeSemantic Layer
Predictive
Prescriptive
IoT
Streaming
AI

4.
Start with the problem,
not with the technology

5.
The core data platform needs to support various use cases
Portal access to 3 years of history
Interfaces to allow scalable sharing of data

6.
Digital meters collect data… a lot of data
7+ million IoT devices after full roll-out7M+
LEGAL
12TB Sending up to 12 TB of data every day
GDPR and strict government regulations

7.
The “golden hammer” does not exist,
use a portfolio approach

8.
Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#1 Increasing
volume

9.
Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#2 Need to collect and
combine any data

10.
Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#3 From data consumer
to data explorer

11.
Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#4 Real-time becomes
the standard

12.
Think about the types of use cases, processing flexibility and
enablement you require
Data consumers
LOB
CRM
ERP
#1 Increasing
volume
#3 From data consumer
to data explorer
#2 Need to collect and
combine any data
#4 Real-time becomes
the standard

13.
Prioritize software-as-a-service to create a data portfolio
Managing all data
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PROCESS SERVE
Analyzing all data
Integrating all data
From GB’s
to TB’s
Seamless
scaling
1000+ API
requests/sec
7M+ devices

14.
Avoid the data lake pitfall

15.
Data lakes often presented as the go-to solution in these cases
Massive scale
Optimized performance
Integration flexibility
Cost effectiveness

16.
But data lake implementations fail in 60% of the cases
Massive scale
Optimized performance
Integration flexibility
Cost effectiveness
Becomes data swamp
Capabilities mismatch
Schema-on-read
Misused as “strategy”

17.
Focus on collaboration

18.
Azure Databricks
Aim for one unified analytics processing engine
Cloud storage
Data warehouses
Big data storage
IoT / streaming data
ML models
BI tools
Data exports
Data warehouses
Collaborative Workspace
DATA
ENGINEER
DATA
SCIENTIST
BUSINESS
ANALYST
Deploy and manage production jobs & workflows
ETL / ELT JOB SCHEDULER MONITORING DEVOPS
Single Runtime Foundation
DATA LAKE
MANAGEMENT
SERVERLESS
/ SaaS
SCALE WITHOUT
LIMITS
PAY-AND-SCALE-
AS-YOU-GO

19.
Narrow the scope and decouple storage from serve
INGEST STORE PROCESS SERVE
Why is this successful?
Unified, consistent data ingest path
Cost efficient for full detail
Why is this successful?
Allows for workload-specific platforms
tailored at specific cases independent
from the raw datastore
SQL DB
Service Fabric
Data Lake Gen2
& Delta

20.
Databricks acts as glue between a data lake and various
workload specific application platforms
INGEST STORE PROCESS SERVE
Databricks
Why is this successful?
Unified runtime and collaboration
One-click setup
Native integration with Azure services
Enterprise security & SLA
SQL DB
Service Fabric
Data Lake Gen2
& Delta

21.
Fluvius’ data portfolio drives innovation
on the Flemish energy market
Bringing consumption insights
and awareness to millions of
households
Lower barrier for data- and
market-driven change
Portfolio projects TCO 20x lower
compared to “golden hammer”

22.
Focus on collaboration
Avoid the data lake pitfall
The “golden hammer” does not exist,
use a portfolio approach
Start with the problem,
not with the technology
Interested in more?
Visit delaware.ai

23.
Focus on collaboration
Avoid the data lake pitfall
The “golden hammer” does not exist,
use a portfolio approach
Start with the problem,
not with the technology
Thank you!
Maarten Herthoge
TL DS&E, delaware BeLux

24.
Feedback
Your feedback is important to us.
Don’t forget to rate
and review the sessions.