Snowflake Data Mesh: Distributing Data Ownership and Democratizing Access

© 2021 Snowflake Inc. All Rights Reserved
SNOWFLAKE
DATA MESH
FOR DINNER
KENT GRAZIANO , CHIEF TECHNICAL EVANGELIST
@kentgraziano

WHAT IS A DATA MESH?
“ A decentralized socio
technical approach in
managing and accessing
analytical data at scale”

Inventor
Thoughtworks:
>8000 employees
High-end Consulting
Customers such as Daimler, Delta, Otto…
DATA MESH IS ONLY MENTIONED IN
RELATION TO SNOWFLAKE AND DELTA LAKE

Data Mesh
https://martinfowler.com/articles/data-monolith-to-mesh.html
• Originated at Thoughtworks
in 2019/2020
• Driven by Zhamak Dehghani
• Data Mesh Community:
https://datameshlearning.com/
https://martinfowler.com/articles/data-mesh-principles.html

Motivations for a Data Mesh
Limitations of centralized warehouses and lakes
Diverse data sources
from many domains
Diverse set of consumers
and requirements
Monolithic DWH
or Data Lake
ETL ETL ETL
ELT ELT ELT
Data
Model
• Proliferation of complex ETL / ELT processes
• Centralized data engineering team lacks data source domain knowledge
• Limited ability to address data quality issues at the sources or react to change
• Difficult to scale to many sources & consumers, slow to integrate new sources
ETL/ELT pipelines
ETL ETL ETL
ETL ETL ELT ELT Data
Model
Data
Model
Data
Model

DATA MESH PRINCIPLES
Distribute responsibility for data pipelines and data quality to people with domain knowledge.
Serve data as-a-product using a common self-service IT infrastructure platform.
Domain-Centric
Ownership &
Architecture
Data as-a-Product
Self-Serve
Infrastructure as-a-
Platform
Federated
Governance
● Data is discoverable
● Data is easy to obtain and use
● Data is documented
● Domains responsible for the
quality of their data
● Domain-agnostic, common tool
set
● Easy to use and low
maintenance to support
● Easy to deploy repeatable
patterns for common
requirements:
cleansing, transformation,
automation, storage, security,
governance, sharing
● Global interoperability
standards across domains
● Define and use global data
governance policies
● Define and apply governance
within each domain and
propagate downstream
● Data pipelines owned by teams
with domain knowledge
● Domains own cleansing,
refinement, historization, pre-
aggregation, etc.
● Domains responsible for
governance, lineage, etc.
● Domains treat data with
consumers in mind

Data Mesh: Domain-centric Architecture
Data Domain 1
Data
sources
from
different
domains
Consumers
• Domain-centric ownership of data sources, pipelines, and data quality
• Ownership sits with domain knowledge --> better data quality for consumers
• Domain teams can react faster to source format changes or quality issues
• Overall easier to scale the number of sources & consumers
• Consumers pull from >= 1 domains
• Data assets offered as products
• “Serve & pull” instead of “push &
ingest” model
Data Domain 2
Data Domain 4
Interoperability Standards, Federated Governance, Data Catalog
Data Domain 5
ELT ELT
ETL ETL ETL
Data
Model
Data
Model
ETL ETL
ETL ETL
ETL ETL
ETL
Data Domain 3
Data Domain 6

Data Mesh changes Scope and Responsibilities
Different split of
responsibilities!
One team per data product
domain, responsible across
all or most stages of the data
life cycle.
Separate teams for different
stages of bringing data from
sources to consumers.
Data Mesh is a “socio-technical shift”. An organizational paradigm for managing, sharing, and democratizing data.

Data Domains vs Data Silos
Data Domains Data Silos
Treat data as a product with consumers in mind Yes No
Data easily discoverable Yes No
Data is documented for consumers Yes No
Downstream consumers have easy access Yes No
Global data interoperability standards Yes No
Responsible for data quality for the benefit of
downstream consumers
Yes No
Ownership of data pipelines to expose data in
forms and shapes that are useful for others
Yes No
Historical snapshots and aggregations for the
benefit of data consumers
Yes No
Common self-service infrastructure Yes No
Federated governance Yes No

DATA MESH ON
SNOWFLAKE

BENEFITS OF SNOWFLAKE FOR A DATA MESH
Snowflake extends a Data Mesh approach by enabling domains to not only share data as a product but
also processing logic as a product
Domain-Centric
Ownership &
Architecture
Data as-a-Product
Self-Serve
Infrastructure as-a-
Platform
Federated
Governance
● Enables sharing of data and
functions as products (public or
private) via secure data sharing
● Out-of-the box capabilities to
publish, discover, request, and
access data products
● Data monetization available for
new revenue streams and
easier procurement
● Fully managed for ease of use
and near-zero maintenance
● Instantly deploy or scale
compute resources
● Rich set of capabilities across
workloads
● Native governance controls
that follow the data across
accounts and clouds through
Snowgrid
● Data Marketplace offers
cataloguing for data products
● Data exchange allows private
governed sharing
● Integrated with leading
governance tools
● Monitoring and alerting on
compute
● Designed for distributed use in
a global cloud network
(Snowgrid)
● Separate compute clusters
● Data sharing within Data Cloud
● Easy to maintain cost
governance within and across
domains
● Can have as many databases
and accounts as you want

THE DATA CLOUD IS A GLOBAL DATA MESH!
A software company shares
terabytes of data with hundreds of
customers
COVID-19 data is available live on
Snowflake Data Marketplace from a
US State, and other organizations
Today’s financial data is
accessible immediately without
data pipelines
Thousands of companies share
data with suppliers, partners, or
other business units
* Visualization based on actual Data Cloud sharing activity as of Oct 28 2021
THOUSANDS OF
ORGANIZATIONS
ARE SHARING
DATA WITH
THEIR
ECOSYSTEM

Data Mesh Reference Architecture
Domain: Customer
Domain: Sales
Domain: Products
Domain: Marketing
Domain: Customer 360
Inventory of shared
data products
Snowflake
Reader Account
Snowflake Data Cloud
Consumers
Data Sources
Interoperability Standards, Federated Governance, 3rd Party Tools
Snowflake Data Sharing as the preferred interoperability standard. Data Marketplace makes data discoverable.
Data Marketplace / Catalog:
• Connects providers to consumers
• Inventory of available assets
• No central storage of shared data
• Providers retain full control over shared
assets (data, functions)
• Consumers access live provider data, no
copies or ETL required
Data domains:
• Can consume and share data or
functions
• Control access policies, data masking,
etc. for downstream consumers
• Can share external tables, i.e. provide
access to data outside of Snowflake
• Can provide reader accounts for
non-Snowflake consumers
Data Consumers:
• Register shared data for local SQL
access in their environment (no copy)
Snowflake
Data
Marketplace
or 3rd-party
catalog
3rd party
marketing
agency
Reseller
Sales
Analysts
Churn &
Retention
Business
optimization
Finance &
Controlling

Snowflake Data Marketplace
Contact &
Support
Sample SQL
Data product
descriptions,
meta data
Searchable inventory of data products
Request & access
process

Global and Multi-Cloud Data Mesh
Data Domain 1
Data Domain 2
Data Domain 3
Data Domain 5
Data Domain 4
US East
FRA
Snowflake
Reader Account
Consumers
Snowflake enables a truly global and multi-cloud data mesh across cloud platforms and regions.
• Data sources, data domains, and
consumers can sit in different regions
and different cloud platforms
• Snowflake enables a truly global and
multi-cloud data mesh
Tokyo
Zurich
Data Sources
Inventory of shared
data products
Snowflake
Data Market-
place or 3rd-
party catalog

Data Sources
Data Mesh Reference Architecture: Governance
Data Domain 1
Data Domain 2
Data Domain 3
Data Domain 5
Data Domain 4
Snowflake
Reader Account
Consumers
In the Snowflake Data Cloud the governance follows the data !
1
2
3
4
Create
Listing
Create
local link
Creat
e
local
link
5
6
Data Mesh with
Federated Governance:
• Domain 2 shares a data asset by
creating a listing in the marketplace
• Domain 2 also defines access
restrictions on that data asset, based
on consumer roles and other attributes
• Domain 5 and consumer 3 discover the
asset in the inventory and link to it in
their local environment for live SQL
access against the table in domain 2
• Domain 5 and consumer 3 are subject to
the access restrictions defined by
domain 2
• Domain 5 transforms the data, defines
additional access restrictions , and
shares a copy with consumer 6
• Consumer 6 can use the data, subject to
the propagated access restrictions
that domain 5 has added.
Inventory of shared
data as a product
assets
via Data
Marketplace or
third-party tools

Important Data Mesh Considerations
Data domains must
not be data silos
Data Mesh requires
organizational changes
Consistent use of
federated governance
How to scale the
number of data
domains?
Data-as-a-product
is a culture shift
Data Mesh changes
responsibilities and
processes
Consistent use of meta
data, lineage, and data
quality metrics
Cost control across
distributed domains?
Which form of
centralized control?
Common self-service
tool set for all domains
Consistent use of
interoperability
standards
Avoid duplication of
effort across domains

Data Mesh Summary
• Data Mesh: benefits and challenges for enterprise data management
• Snowflake offers unique benefits for data mesh implementations
• Snowflake’s distributed cross-cloud platform is a natural fit for
distributed domains
• Ease of use and near-zero maintenance: true self-service for a broad
range of data pipeline and data management capabilities
• Data Marketplace and Data Sharing: Out-of-the box capabilities to
publish, discover, request, and obtain data products
• Live access to data products across domains,
no ETL or copies required
• Data as a product as well as function as a product
• Native governance controls follow the data across domains
• Easy integration with leading 3rd party tools
Domain-centric
Ownership and
Architecture
Data as a
Product
Self-Serve
Infrastructure as
a Platform
Federated
Governance

THANK YOU

Appendix

Replication
Global
Data Marketplace
Reader
Account
Direct data sharing
Offer data or functions
Consume data or functions
Region
Replication
Consume data/functions
Repl.
Failover /
Failback
2-way
sharing
• Currently 24 cloud regions
• Integrated into a single cloud data network
• Eliminates data silos
• Enables secure sharing of live data & functions, no ETL
• Enables replication for DR and failover
• Enables global Data Mesh architectures
Private Data
Marketplace
Failover / Failback
Snowgrid: the backbone of a global data mesh
Sample set of Snowflake nodes exploiting Snowgrid capabilities

Secure Data Sharing Within and Across Regions
Provider
SHARE
Consumer A
Enriched
Table
View, Query,
Join
SHARE
Consumer B
Provider Replica
SHARE
Consumer D
Consumer C
Database Replication
Share Replication*
Data
Cleanroom
S3
Database
Replication
Share
Replication*
Consumer

SOURCES
Zhamak's initial article introducing the data mesh concept: https://martinfowler.com/articles/data-monolith-to-mesh.html
Sven Balnojan's article on how one could implement a data mesh: https://towardsdatascience.com/data-mesh-applied-
21bed87876f2
Zhamak's second article, delving deeper: https://martinfowler.com/articles/data-mesh-principles.html
Barr Moses' article on Data Mesh basics: https://towardsdatascience.com/what-is-a-data-mesh-and-how-not-to-mesh-it-up-
210710bb41e0
Zhamak Dehghani | Kafka Summit Europe 2021 Keynote: How to Build the Data Mesh Foundation
https://www.youtube.com/watch?v=QF41q10NSAs
https://datameshlearning.github.io/intro-to-data-mesh/

Snowflake Data Mesh: Distributing Data Ownership and Democratizing Access

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Snowflake Data Mesh: Distributing Data Ownership and Democratizing Access

Similar a Snowflake Data Mesh: Distributing Data Ownership and Democratizing Access (20)

Más de Kent Graziano

Más de Kent Graziano (20)

Último

Último (20)

Snowflake Data Mesh: Distributing Data Ownership and Democratizing Access