Building data as a product: The key to unlocking Data Mesh's potential Data as a product is an exciting concept. It brings product thinking into datasets and facilitates data driven culture by encouraging teams to share data rather than data being in a silo. Once we are convinced with the philosophy of data as a product then there is an immediate question of how to build them? In this talk we will uncover ways to design and architect data as a product that meets the needs of a business use case. We will also discuss creating a blueprint for better resiliency via contracts and service level objectives. Speakers: Harmeet Sokhi, Lead Data Consultant, Thoughtworks and Vishal Srivastava, Senior Data Engineer, Thoughtworks Harmeet has extensive experience in Cloud, data engineering and machine learning operations. She has worked on designing large enterprise-scale data applications and has also implemented mature machine learning engineering solutions for clients in several industries. She is always in the pursuit of learning and keeps herself current in the ever-changing technology landscape. She is an experienced team leader who helps address challenges, both technical and non-technical, to deliver highly credible results. Vishal is a Senior Data Engineer with DevOps skills who has worked across a range of industries. He has experience in establishing cloud infrastructure foundations, event-driven data lake, data visualisation, master data management, data quality and data governance frameworks. He is passionate about real time event driven distributed systems. Vishal has used these experiences to enable use cases which help businesses realise real value from data.

1. © 2022 Thoughtworks | Confidential
By
Thoughtworks
1

2. © 2022 Thoughtworks | Confidential
By Thoughtworks
Building data as a product: the key to
unlocking Data Mesh's potential
2

3. © 2022 Thoughtworks | Confidential
Harmeet Sokhi
3
Vishal Srivastava
● Rich experience in implementing data platforms.
● Passionate about data mesh
● Member & Contributor of Global Data Mesh guild at
Thoughtworks.
● Design and Implement data engineering platforms, data
Products and Machine Learning architectures
● Passionate about helping organizations unleash the potential
of data thru democratisation
● Co-organizer of the Data Engineering Melbourne meetup

4. © 2022 Thoughtworks | Confidential
Agenda
4
Story of Essence Financials
K’s new use case
Data product plot
Conclusion
Filling the gaps

5. © 2022 Thoughtworks | Confidential
Essence
Financials
5
5
© 2021 Thoughtworks
Vision : To keep market position as a
leading financial service provider

6. © 2022 Thoughtworks | Confidential
Significant investment in data
Reflection of past
Ambitions and Challenges
6
Ambition was to become data oriented
organisation
Data Teams were not able to keep up
with the demand.
Data Trustworthiness was questionable
Reliance on central data engineering
team.
Siloed and hyper-specialized ownership

7. © 2022 Thoughtworks | Confidential
Data Mesh to the rescue
Scale
Analytical Data
7
Socio-technical
Decentralized

8. © 2022 Thoughtworks | Confidential 8
8
Why Essence Financials moved away from “data sets”?
Data
Discoverable
Addressable
Understandable
Trustworthy
Natively Accessible
Valuable
Secure
Data Product > Dataset
Interoperable

9. © 2022 Thoughtworks | Confidential
Data Mesh to the
rescue
Essence Financials got benefited by four interconnected & non-negotiable pillars
9
Dehghani, Z. (2022). Data Mesh: Delivering Data-Driven Value at Scale (1st ed.). O’Reilly Media.
Domain-oriented
Ownership
Data as a
Product
Self-serve data
platform
Federated Computational
Governance
{G} {G} {G}

10. © 2022 Thoughtworks | Confidential
Logical architecture of “a” Data Product in Essence
Financials
Transformation
Governance
Platform
Output
Port(s)
Input
Port(s)

11. © 2022 Thoughtworks | Confidential
Data Product: Categories in Essence Financials
11
Source oriented Data
Product
Fit-for-purpose Data
Product
Customer oriented
Data Product
*
*
*
*
*
*

12. © 2022 Thoughtworks | Confidential
Data Product: Typical deployment in Essence
Financials
Data Product
Developers
Data Product Code
(Business Logic)
Data Product Model Spec (Platform
& Governance Configuration)
Data Product
Owner
Problem / use case
Utility Deploy

13. © 2022 Thoughtworks | Confidential
Process after adoption
13

14. © 2022 Thoughtworks | Confidential 14
14
Tying it back: Designing Data Product
14
© 2022 Thoughtworks
Gap analysis Build & Measure
Data
Strategy
Data Mesh is the right
architectural choice
Use Case
Define

15. © 2022 Thoughtworks | Confidential
K’s new use case
15
/Define
Gap analysis
Use Case
Build & Measure
Define

16. © 2022 Thoughtworks | Confidential
Use Case
16
Promote prime loans

17. © 2022 Thoughtworks | Confidential
Elevator Pitch
FOR Marketing Team
WHO Want to sell high value loans
THE “Promote Prime Loans”
IS A Recommendation Engine
THAT will allow them to spend more time in engaging with relevant customers
UNLIKE now when then send blanket communication to everyone
OUR PRODUCT Resulting in more conversions as well as saving Time spend in Operations
17
K’s new use case
PROMOTE
PRIME
LOANS

18. © 2022 Thoughtworks | Confidential
Agenda
18
Story of Essence Financials
K’s new use case
Data product plot
Conclusion
Filling the gaps

19. © 2022 Thoughtworks | Confidential
SLO
19
Gap analysis
Use Case
Build &
Measure
Define
Data Landscape Data Product Map CFRs
Define

20. © 2022 Thoughtworks | Confidential
Data Landscape - SLO-Data Product Map - CFRs
20
Gap analysis
Use Case
Build &
Measure
Define

21. © 2022 Thoughtworks | Confidential
21
Data Product: Data Landscape
Understand Current State,Pains & Aspirations
Customer
Information
Customer
transaction History
Acquire Productionize &
Execute
Curate Model

22. © 2022 Thoughtworks | Confidential
Data Landscape -SLO- Data Product Map - CFRs
22
Gap analysis
Use Case
Build &
Measure
Define

23. © 2022 Thoughtworks | Confidential
23
How often it
is used?
How many
people or
services use
it?
How
complete the
data need to
be?
How often is it
updated?
When does
it need to be
updated?
How
accurate
does the
data need
tobe?
How fresh
the data
need to be?
Every
second
Millions
100%
complete
Every second
24/7
Perfectly
Accurate
Realtime
Every Hour Thousands Every Hour
Within
minutes
Several
times a day
Hundreds
Mostly
complete
Several times
a day
Extended
Workday
Some
room of
error
Within
Hours
Daily Dozens Daily
Within
the day
Weekly
A couple
of teams Can
tolerate
Missing
data
Weekly
During
one part
of the day
As long as
its
representa
tive
Within
the week
Monthly +
Just a few
people
Monthly +
Within
the
month
PROMOTE
PRIME
LOANS
Records with 80%
complete data with
100% accuracy are
refreshed daily
Data Product: SLOs
Service Level Objectives assist in determining technical design

24. © 2022 Thoughtworks | Confidential
Data Landscape - SLO-Data Product Map - CFRs
24
Gap analysis
Use Case
Build &
Measure
Define

25. © 2022 Thoughtworks | Confidential
Data Product: Integrated Data Product Map
What could be different types of data products?
25
Sources
Source oriented Data
Product
Fit-for-purpose Data
Product
Customer Oriented
Data Product
Operational
Reporting/Analytics
PROMOTE
PRIME
LOANS

26. © 2022 Thoughtworks | Confidential
26
Sources Source oriented DP Fit-for-purpose Data
Product
Customer Oriented
Data Product
Operational
Reporting/Analytics
PROMOTE
PRIME
LOANS
Salesforce
Snowflake
Data Product: - Integrated Data Product Map
Start from customer oriented data product
Domain: Marketing
Subdomain: Promotion

27. © 2022 Thoughtworks | Confidential
27
Sources Source oriented DP Fit-for-purpose Data
Product
Customer Oriented
Data Product
Operational
Reporting/Analytics
PROMOTE
PRIME
LOANS
Salesforce
Snowflake
Data Product: Integrated Data Product Map
Now lets focus on source oriented data product and their attributes
Domain: Marketing
Subdomain: Promotion
Customer
Information
CUSTOMER
INFO
Customer
History Table
CUSTOMER
REPAYMENT
HISTORY
Domain: Customer
Subdomain: Payment

28. © 2022 Thoughtworks | Confidential
Data Product: Integrated Data Product Map
We need intermediate Data Product also
CUSTOMER
REPAYMENT
STATUS
Sources Source oriented DP Fit-for-purpose Data
Product
Customer Oriented
Data Product
Operational
Reporting/Analytics
PROMOTE
PRIME
LOANS
Salesforce
Snowflake
Domain: Marketing
Subdomain: Promotion
Customer
Information
CUSTOMER
INFO
Customer
History Table
CUSTOMER
REPAYMENT
HISTORY
Domain: Customer
Subdomain: Payment
Domain: Marketing
Subdomain: Promotion

29. © 2022 Thoughtworks | Confidential
SLO-Data Landscape - Data Product Map - CFRs
29
Gap analysis
Use Case
Build &
Measure
Define

30. © 2022 Thoughtworks | Confidential 30
30
Data Product: Cross
Functional
Requirements (CFRs)
30
© 2021 Thoughtworks
Data Access Policy
Data standardisation
Rules
Data Classification
Rules
Data retention policy
TFN

31. © 2022 Thoughtworks | Confidential
SLO-Data Landscape - Data Product Map - CFRs
31
Gap analysis
Use Case
Build &
Measure
Define

32. © 2022 Thoughtworks | Confidential
Agenda
32
Story of Essence Financials
M’s new use case
Data product plot
Conclusion
Filling the gaps

33. © 2022 Thoughtworks | Confidential
Architecture Gap analysis
33
Gap analysis
Use Case
Build & Measure
Define

34. © 2022 Thoughtworks | Confidential
Architecture Gap Analysis
34
Data Product
Architecture
Governance
Platform
Capabilities
Gap
analysis
Use Case
Build &
Measure
(
Define

35. © 2022 Thoughtworks | Confidential
Orchestration
Managed Airflow
35
Data Product: Gap Analysis
Data storage
Snowflake
Marketing
data mart
PROMOTE PRIME LOANS
Governance
Transformation
Data product code
Platform Capabilities
Snowflake
connector
Salesforce
connector
Snowflake
connector

36. © 2022 Thoughtworks | Confidential
Data Product- Promote Prime Loan
CUSTOMER
REPAYMENT
STATUS
PROMOTE
PRIME
LOANS
Salesforce
Snowflake
Customer
Information
CUSTOMER
INFO
Customer
transaction
history
CUSTOMER
REPAYMENT
HISTORY
Platform Capabilities
Governance
Managed Airflow Snowflake ci/cd Observability

37. © 2022 Thoughtworks | Confidential
Agenda
37
Story of Essence Financials
M’s new use case
Data product plot
Conclusion
Filling the gaps

38. © 2022 Thoughtworks | Confidential
/Build & Measure
38
Gap analysis
Use Case
Build &
Measure
Define

39. © 2022 Thoughtworks | Confidential
Build & Measure
39
Gap analysis
Use Case
Build &
Measure
(
Define
Learn
Build
Measure

40. © 2022 Thoughtworks | Confidential
Build & Measure
40
Gap analysis
Use Case
Build &
Measure
(
Define
Data Product Score
Rule based User feedback
KPIs
SLOs

41. © 2022 Thoughtworks | Confidential
Happy K
41

42. © 2022 Thoughtworks | Confidential
Recap
42
Gap analysis
Define
Data Landscape - SLO
Data Product Map - CFRs
Use Case
Build &
Measure
Data Product
Architecture
Governance
Platform
Capabilities
Learn
Build
Measure

43. © 2022 Thoughtworks | Confidential
“A Journey of Thousand Miles start with a
single step”
-Lau Tze
43

44. © 2022 Thoughtworks | Confidential
We look forward to
working with you
Harmeet Kaur Sokhi
Lead Data Consultant
harmeet.sokhi@thoughtworks.com
Vishal Srivastava
Senior Data Consultant
vishal.srivastava@thoughtworks.com
44
Please share your feedback