SlideShare una empresa de Scribd logo
1 de 118
Descargar para leer sin conexión
Copyright Third Nature, Inc.
Architecting a Data
Platform For
Enterprise Use
May 2018
Mark Madsen
Todd Walter
Copyright Third Nature, Inc.
The business complaints about data and analytics
You really
should
fire IT.
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
Data
deficient
Takes
too long
Costs
too much
Function
deficient
IT root
causes
IT proximate
causes
What is said
in disputes
Lack of
agility
People & vendor
cost basis
Client-server
infrastructure
Lack of
“good enough”
competency
1980s-era
methods
Inappropriate
technology
Data hygiene
fetishes
Vendor lock
1990s-era
procurement
IT skills
deficit
Dysfunctional
OLTP portfolio
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
Data
deficient
Takes
too long
Costs
too much
Function
deficient
IT root
causes
IT proximate
causes
What is said
in disputes
How business
responds
How IT
responsds
Lack of
agility
People & vendor
cost basis
Client-server
infrastructure
Lack of
“good enough”
competency
SaaS /
Cloud BI
Consultants
Self-service
BI
Shadow BI
Workarounds &
spreadmarts
Hidden costs
Loss of
control
Loss of
visibility
Loss of
knowledge
Security risks
Bad data risks
1980s-era
methods
Inappropriate
technology
Data hygiene
fetishes
Vendor lock
1990s-era
procurement
IT skills
deficit
Dysfunctional
OLTP portfolio
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
Data
deficient
Takes
too long
Costs
too much
Function
deficient
IT root
causes
IT proximate
causes
What is said
in disputes
How business
responds
How IT
responds
Lack of
agility
People & vendor
cost basis
Client-server
infrastructure
Lack of
“good enough”
competency
SaaS /
Cloud BI
Consultants
Self-service
BI
Shadow BI
Workarounds &
spreadmarts
Hidden costs
Loss of
control
Loss of
visibility
Loss of
knowledge
Security risks
Bad data risks
1980s-era
methods
Inappropriate
technology
Data hygiene
fetishes
Vendor lock
1990s-era
procurement
IT skills
deficit
Dysfunctional
OLTP portfolio
This is not a technology problem –
it is an architecture problem.
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
What do we hear in the field?
"I want to do analytics, beyond what I can do with BI tools"
"I need an analytics strategy"
"I need an analytics roadmap"
"I have a specific analytics project of type..."
“We have a data lake. What should we do with it?”
“Technology Y replaces technology X”
"I want to modernize my DW" aka "I want to speed up my DW"
“Don’t need schema, curation, ETL, governance … anymore”
Good judgement is the result of experience.
Experience is the result of bad judgement.
—Fred Brooks
Copyright Third Nature, Inc.
“Build a data lake and put all the data there first”
Copyright Third Nature, Inc.
DW: Centralize, that solves all problems!
Creates bottlenecks
Causes scale problems
Availability?
Copyright Third Nature, Inc.
The data lake solution: no central authority
wtf, it was fully
operational!
Copyright Third Nature, Inc.
The data lake solution?
There’s a problem: as
the lake is envisioned,
it is still a centralized
data architecture, but
this time there is no
single global model.
Instead it’s files and
not modeled. It can be
operational while
under construction.
It’s still a death star.
Copyright Third Nature, Inc.
Eventually we run into the same problems
Seriously, wtf?
It was agile
and operational
Rising complexity and scale break centralized models
Copyright Third Nature, Inc.
“Standardize on one tech, it’s simpler for everyone”
It’s called stack think. Pick your vendor. Cede all architecture.
Copyright Third Nature, Inc.
Agile your way into the Analytics Shantytown
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
“Infrastructure one PoC at a time”
• Use case driven
• Leverage (any) new
technology
• Re-use of the
technology stack
• Data is captive to the
application or to the
technology stack
• Developers tend to
think “function first”
• Analytics people also
think “function first”,
but generally require
integrated data
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
A key aspect of operational analytics is “end-to-end”
E2E does not allow you to take a random
walk through technologies and BYOT
because the complexity of integration leads
to a mountain of technical debt.
• Most data
applications are
organized around a
process, not a task or
function.
• Real-time analytics
systems pass their
data over the
network, but the
dependencies can
cross application
boundaries, as well
as requiring
persistence.
• Developers are
being forced to think
of data outside the
local context.
Copyright Third Nature, Inc.
“Start with the platform. The rest will follow.”
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
Big data reality: What users seeBig data promise: What you see
Copyright Third Nature, Inc.
Persisting data
is not the end
of the line.
If you stop
here you win
the battle and
lose the war
Copyright Third Nature, Inc.
We don’t have an analytics problem, just like we
didn’t have a BI problem
The origin of analytics as “business
intelligence” was stated well in 1958:
…the ability to apprehend the
interrelationships of presented facts in
such a way as to guide action towards a
desired goal. ~ H. P. Luhn
“A Business Intelligence System”, http://altaplana.com/ibmrd0204H.pdf
”
“
Our goal is analytics as a capability, not a technology
Copyright Third Nature, Inc.
Three constituencies
Stakeholder Analyst Builder
aka the recipient aka the data scientist aka the engineer
Copyright Third Nature, Inc.
Starting points for analytics strategy
Many organizations choose to start with
the analysts. Create a data science team.
Turn them loose to find a problem.
Many more start with builders: technology
solutions looking for problems, e.g. 65% of
the IT driven Hadoop and Spark projects
over the last five years.
The right place to start? Stakeholders. The
goal to achieve, the problem to solve.
Copyright Third Nature, Inc.
NATURE OF THE PROBLEM FROM
THE STAKEHOLDER’S PERSPECTIVE
Each constituency has their own set of problems to deal with
BI and Analysis: Repeatability and Discoverability
Discover
Explore
Analyze
Model
Consume
Promote
Focus on repeatability
Application cycle time
90% of users
Focus on discoverability
Analyst cycle time
9% analysts, 1% data scientists
80% of data use 80% of analysis
The myth that still drives analytics – analytic gold
All we need is a fat
pipe and pans
working in parallel…
Analytic insights that result in no action are expensive trivia.
It’s not the insight, but what you do with it, that matters
Applying analytics is not an analytics problem
Applying analytics is not in the
analyst’s control.
It’s not in the engineer’s control.
It’s in the control of the people
involved in the process.
Failures are often in execution, not
in analytics development.
For example, we saw unexpectedly
poor performance in a number of
geographies. Was it the new
analytics we tried? Was it a data
problem? No, it was a simple
compliance problem.
NATURE OF THE PROBLEM FROM
THE ANALYST’S PERSPECTIVE
The analytics process at a high level
Diagram: Kate Matsudaira
The nature of analytics problems is researching the
unknown rather than accessing the known.
Repeat for each new problem
Diagram: Kate Matsudaira
The main hurdle: just getting the data
Do you know where to find it? Because it’s
may not be in the data warehouse.
Do you have access to it?
Is access fast enough? Because DWs are for
QRD, not for moving huge piles of data. And
ERP systems and SaaS apps are right out.
These are why developers jump on “data lake”
as the solution. But that’s a small, easy part.
Do you have the right data?
Many machine learning
techniques require labeled
(known good) training data:
Supervised learning: a person
has to define the correct
output for some portion of
the data. Data is divided into
training sets used for model
building and test sets for
validating the results.
• What is spam and what isn’t?
• What does a fraudulent
transaction look like
36
Do you have enough of the right data?
ML needs a lot, you may be disappointed in your efforts
Where do analysts spend their time? mostly data work
Slide 38
Define the
business problem
Translate the
problem into an
analytic context
Select appropriate
data
Learn the data
Create a model set
Fix problems with
data
Transform data
Build models
Assess models
Deploy models
Assess results
% of time spent
70% 30%
Source: Michael Berry, Data Miners Inc.
The analyst’s workspace in BI is relatively spare
The analyst’s workspace needs to be more like a
kitchen than like BI vending machines
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
Schema control, space limitations, resource
limitations, production lockdown
Analytics work is a production workload, but…
NATURE OF THE PROBLEM FROM
THE BUILDER’S PERSPECTIVE
IT and Ops people want to know “what to build?”
Giant data platform? Self service tools?
Analytics has different processes and workloads
None of this analytics work
is the same as what IT
considered “analysis” to be,
which is usually equated
with BI or ad-hoc query.
Ad-hoc analysis =
Exploratory data analysis =
Batch analytics =
Real-time analytics
A real analytics production workflow
Hatch, CIKM ‘11 Slide 44
The world changes, do the models?
In BI you maintain ETL and
schemas, in ML you maintain
models.
“Model decay” happens as the
assumptions around which a
model is built change, e.g. spam
techniques change.
When you adjust the model you
need to know it is normal again
▪ Better save the data used to
build the model
▪ Better save the model
▪ Baseline and measurements
THREE PERSPECTIVES, ONE SOLUTION?
There are requirements from all constituents. You need to put them
together to have a complete picture of what’s needed.
The missing stakeholder
There is another stakeholder:
analytics management - the
CAO, CDO, VP of analytics, aka
“your boss” if you’re a data
scientist.
This is the perspective and
problems of the person
responsible for oversight of
the team and efforts is across
the organization and across
multiple projects
Job #1 - Repeatability
Job # 2 - Operational predictability
Job #3 - Reproducibility
You need a system of record for analytics
There is an extensive list of requirements to support
Primary requirements needed by constituents S D E
Data catalog and ability to search it for datasets X X
Self-service access to curated data X
Self-service access to uncurated (unknown, new) data X X
Temporary storage for working with data X
Data integration, cleaning, transformation, preparation tools and environment X X
Persistent storage for source data used by production models X X
Persistent storage for training, testing, production data used by models X X
Storage and management of models X X
Deployment, monitoring, decommissioning models X
Lineage, traceability of changes made for data used by models X X
Lineage, traceability for model changes X X X
Managing baseline data / metrics for comparing model performance X X X
Managing ongoing data / metrics for tracking ongoing model performance X X X
S = stakeholder, user, D = data scientist, analyst, E = engineer, developer
© Third Nature Inc.
How did we get to this point
with BI & analytics?
There’s a difference
between having no past
and actively rejecting it.
54
100
1,000
10,000
1 2 3 4 5 6 7 8 9 10
Customer Data Plateaus
Add users,
accumulate
history, add
attributes, add
dimensions
Entirely new data
set enabled by
new technology
at new price
point – value has
to exceed cost
(ROI). Earliest
adopters have
vision ahead of
ROI
55
1
10
100
1,000
10,000
100,000
1,000,000
10,000,000
1 2 3 4 5 6 7 8 9 10 11 12 13
User Adoption of New Data
Customer Data
Users - Data Set 1
Users - Data Set 2
Users of Integrated Data
User adoption of new data sets
starts over. Very small number of
experts growing to wider
audience, sophisticated users
moving to business analysts, then
business users, B2B customers and
even to consumers
Greater value is derived when
data sets are linked – see bigger
picture (eg who buys what,
when). Comes after initial
extraction of easy value from
standalone data
56
1
10
100
1,000
10,000
100,000
1,000,000
10,000,000
100,000,000
1,000,000,000
10,000,000,000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
Data Size Plateaus over Time
Lather, Rinse, Repeat.
Trend line is roughly
Moore’s Law.
Delay or skip a
generation if new data
set is two orders of
magnitude instead of
one.
57
1
10
100
1,000
10,000
100,000
1,000,000
10,000,000
100,000,000
1,000,000,000
10,000,000,000
100,000,000,000
1,000,000,000,000
10,000,000,000,000
100,000,000,000,000
1,000,000,000,000,000
10,000,000,000,000,000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43
Customer Data Size vs. 2x per 12mo
Customer Data
Moore's Law - 2x/18mo
2x/12 mo
But all studies show
demand/creation of
data increasing
significantly faster than
Moore’s law
If we started only 1
OOM behind, we are
now 5 OOM away from
user demand for data
Data Exhaust
58
• <Store, Item, Week>
• <Store, Item, Day>
– Simple aggregations
• Market Basket
– Affinity
– Link to person, demographics, HR
• Inventory by SKU by store
– Temporal, time series, forecasting
– Link to product, marketing, market basket
Retail Plateaus
2B records total
for 9 quarters
2B records per
day, keep 9
quarters
59
• Web Logs and traffic
– Behavioral patterns – eg path linked to person, offers, other channels
– Operations of the web site
• Supply chain sensors – sampled at major event
– Activity Based Costing
– Link to customer, product, HR, planning
• Social Media
– Text analysis, Filtering, languages
– Link to customer, sales, other channel interactions
• Supply chain sensors – sampled at minutes or seconds
– Telematics
– Real time, Event detection, trending, static and dynamic rules
– Link to HR, thresholds, forecasts, routing, planning
Retail Plateaus
30B records per day
60
FINANCE
Revenue
Expenses
Customers
CUSTOMER
CARE
Customer
Products
Orders
Case History
SALES
Orders
Customers
Products
MARKETING
Customers
Orders
Campaign History
OPERATIONS
Inventory
Returns
Manufacturing
Supply Chain
How many
batteries are
in inventory by
plant?
What is the
trend of
warranty
costs?
How many
people made
a warranty
claim last
week?
How many
sales have
been made
quarter to
date?
Which
customers
should get a
communicatio
n on extended
warranties?
54 32 29 49 66
61
2954 32 49 41
62
2855
Given the rise in warranty costs, isolate the problem to be a plant, then to a battery lot.
Communicate with affected customers, who have not made a warranty claim on batteries,
through Marketing and Customer Service channels to recall cars with affected batteries.
Inventory
Returns
Manufacturing
Supply Chain
Customer Service
Orders
Revenue
Expenses
Case History
Customers
Products
Pipeline
Customers
Campaign History
FINANCE
SALESMARKETING
OPERATIONS
CUSTOMER EXPERIENCE
2855
63
Manufacturing: Data Overlap Analysis
New Business Improvement Opportunities through Data Leverage
Sales Force Profitability Analysis 100% 80% 66% 24% 41% 66% 0% 24% 0% 24%
Transportation Planning 11% 100% 87% 28% 64% 56% 22% 34% 13% 45%
Production Planning 6% 57% 100% 19% 83% 35% 17% 28% 9% 40%
Vendor Managed Inventory 12% 100% 100% 100% 78% 100% 61% 100% 39% 100%
Global Pricing Rationalization 4% 50% 100% 17% 100% 27% 15% 29% 11% 41%
Fulfillment (Perfect Order) 16% 94% 90% 48% 58% 100% 35% 53% 19% 56%
Manufacturing Quality Optimization 0% 76% 88% 60% 66% 71% 100% 79% 43% 73%
Preventative Maintenance Analysis 7% 75% 93% 61% 80% 68% 50% 100% 27% 82%
Warranty Claims Analysis 0% 94% 100% 83% 100% 83% 94% 93% 100% 98%
Quality Life Cycle Improvement 4% 57% 75% 35% 64% 41% 26% 47% 16% 100%
SalesForce
ProfitabilityAnalysis
ProductionPlanning
VendorManaged
Inventory
GlobalPricing
Rationalization
TransportationPlanning
Fulfillment
(PerfectOrder)
ManufacturingQuality
Optimization
Preventative
Maintenance
Analysis
WarrantyClaims
Analysis
QualityLifeCycle
Improvement
If
Then
64
65
PRODUCT SENSOR SOCIAL MEDIA CUSTOMER
CARE AUDIO
RECORDINGS
DIGITAL
ADVERTISING
CLICKSTREAM
65 41 32 19 28
How many
visitors did we
have to our
hybrid cars
microsite
yesterday?
What are the
temperature
readings for
batteries by
Manufacturer?
What is the
sentiment
towards line of
hybrid
vehicles?
Which
customers likely
expressed
anger with
customer
care?
Which ad
creative
generated the
most clicks?
66
2855
High, Well
Known Quality
Directional
Quality
Unknown/Low
quality
Well Defined
Data Model
Curated
JSON, XML,…
Extracted
Attributes
Curation Required
67
2855
SENSOR
DIGITAL ADVERTISING
CLICKSTREAM INTERACTIONS
RATINGS & REVIEWS
CUSTOMER PORTAL
INTERACTIONS
EXTERNAL INTERACTIONS
SOCIAL MEDIA
IVR Routing
RFID
ELECTRONIC COMMERCE
FINANCE
SALES
MARKETING
Inventory
Returns
Manufacturing
Supply Chain
Customer Service
Orders
Revenue
Expenses
Case History
Customers
Products
Pipeline
Customers
Campaign History
OPERATIONS
CUSTOMER CARE –
AUDIO RECORDINGS
Maps
Telemetry
SERVER LOGS
CUSTOMER EXPERIENCE
Enterprise Data
68
2855
Schema on
Write
Evolving
Schema
Schema on
Read
Enterprise
Data
Evolving
Data
New Data
Sources
Schema?
69
2855
Out of Deviation
Sensor Readings
Fraud
Events
CUSTOMER EXPERIENCE
SALES
MARKETING
OPERATIONS
Abandoned
Carts
Online Price
Quotes
Social Media
Influencers
FINANCE
11234
Minimum Viable
Curation
Minimum Viable
Data Quality
70
2855
Out of Deviation
Sensor Readings
Fraud
Events
CUSTOMER EXPERIENCE
SALES
MARKETING
OPERATIONS
Abandoned
Carts
Online Price
Quotes
Social Media
Influencers
FINANCE
11234Sensor data
formatting
Unit
Normalization
Vehicle/version
normalization
Make/model/year
selection
Data Comprehension,
Pipelines
71
2855
Enterprise
Wide Use
10s to 100s
of Users
10 Users
Super Users
Business
Analysts
Action list, Report,
Dashboard Users
Share across
Enterprise
Share in
department
Share over
cube wall
User Base and
Sharing
72
2855
Enterprise
production
analytics
Departmental
analytics
Exploratory
analytics
Repeatable,
auditable
results
Evolving
Consumption
Ad-Hoc
Query, Self
Service
Data Labs
73
2855
Production tools,
curated data,
integrated across
business areas
Wide variety
of tools and
data forms
Targeted data access,
many applications,
response time SLAs
Evolving
Consumption
Requirements
High CPU,
moderate to
low IO
Bulk scans, large
computation,
transformation,
data access,
specific
integration
Moderate
CPU, High IO,
Resource
management
74
2855
Relational store
more appropriate
Hybrid
File Store more
appropriate
Technology,
Capacity
100 TB
1 PB
10 TB
75
2855
MANUFACTURING
CAMPAIGN HISTORY
COSTS
PRODUCTS
FINANCE
SALES
MARKETING
OPERATIONS
CUSTOMER EXPERIENCE
Given the rise in warranty
costs, isolate the problem to
be a plant and the specific
lot.
Exclude 2/3rd of the batteries
from the lot that are fine.
Communicate with affected
customers, who have not
made a warranty claim,
through Marketing and
Customer Service channels
to recall cars with affected
batteries.
CUSTOMERS
CASE HISTORY
SENSOR
Access Wide Variety of Data
to Answer a Question
Copyright Third Nature, Inc.
An event contains mainly IDs…that reference other data
2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a
13ae51a6d092 301212631165031 590387153892659 DNIS5555685981-
UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s
m100109_44IOJ1-Id=105543CD1A7B8322
timestamp
user ID
device ID
game ID event details
IP adr
log ID
Log de-referencing and enrichment is difficult since you can’t
enforce integrity like you can in a DB.
What’s the glue that holds it together?
It’s just keys to other data.
e.g. remember that device ID 0 problem?
Copyright Third Nature, Inc.© Third Nature Inc.
It’s not just the log data, it’s big data + small data
Device
Event logs
Event logs
Event logs
A product has a complex set of master
data and metadata and this needs to
be added back to the bare events.
Copyright Third Nature, Inc.© Third Nature Inc.
Where does the reference data come from?
The keys come from somewhere. You don’t just make up
system-wide unique identifiers in your code.
The lack of local lookup data at event generation leads to
development practices that lead to inconsistencies.
Event logs
Event logs
Event logs
Problem we had: product
identifiers that didn’t match
any known products
Had to fix by analyzing each
log of bad-ID events.
Because developers used
config files they copied from
the PMS and put in the code
Break here and
you have nothing
Copyright Third Nature, Inc.
MDM again
If you want to link datasets then you must manage the keys
You need canonical forms for common data (in code too)
user ID
IP adr
2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a
13ae51a6d092 301212631165031 590387153892659 DNIS5555685981-
UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s
m100109_44IOJ1-Id=105543CD1A7B8322
UID CID Email City State Country
590387153892659 10098213 barry.dylan@odin.com Paris Île-de-FranceFrance
2010.01.10.14:26:2468 67.189.110.179 10098213 5046876319474403 MOZILLA/4.0
(COMPATIBLE; TRIDENT/4.0; GTB6; .NET CLR 1.1.4322) https://www.phisherking.com/
gifts/store/LogonForm?mmc=link-src-email_m100109 http://www.google.com/search?
sourceid=navclient&aq=0h&oq=Italian&ie=UTF8&pid=1T4ACGW_13ae51a6d092&q=ita
lian+rose&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s
game ID
date
IP adr
customer ID
Event
Click
Cust-
user
What you think you have
Copyright Third Nature, Inc.
What you really have
This is called “self-service.” Analysts will love it…
Copyright Third Nature, Inc.
The solution to our problems isn’t
technology, it’s architecture.
Copyright Third Nature, Inc.
History: This is how BI was done through the 80s
First there were files and reporting programs.
Application files feed through a data processing pipeline to
generate an output file. The file is used by a report formatter
for print/screen. Files are largely single-purpose use.
Every report is a program written by a developer.
Data pipeline
code
Copyright Third Nature, Inc.
History: This is how BI ended the 80s
The inevitable situation was...
Data pipeline
code
Copyright Third Nature, Inc.
History: This is how we started the 90s
Collect data in a database. Queries replaced a LOT of
application code because much was just joins. We
learned about “dead code”
SQL
SQL
SQL
SQL
SQL
Copyright Third Nature, Inc.
Pragmatism and Data
Lessons learned during
the ad-hoc SQL era of the
DW market:
When the technology is
awkward for the users, the
users will stop trying to use
it.
Even “simple” schemas
weren’t enough for anyone
other than analysts and
their Brio…
Led to the evolution of
metadata-driven SQL-
generating BI tools, ETL
tools.
Copyright Third Nature, Inc.
BI evolved to hiding query generation for end users
With more regular schema models, in particular
dimensional models that didn’t contain cyclic join paths, it
was possible to automate SQL generation via semantic
mapping layers.
We developed data pipeline building tools (ETL).
Query via business terms made BI usable by non-technical
people.
ETL
SQL
Life got much easier…for a while
Copyright Third Nature, Inc.
Today’s model: Lake + self-service, looks familiar…
The Lake with data pipelines to files or Hive tables
is exactly the same pattern as the COBOL batch..
Dataflow
code
We already know that people don’t scale…
Copyright Third Nature, Inc.
DATA ARCHITECTURE
We’re so focused on the light switch that we’re not
talking about the light
Copyright Third Nature, Inc.
Decouple the Architecture
The core of the data warehouse isn’t the
database, it’s the data architecture that the
database and tools implement.
We need a data architecture that is not limiting:
▪ Deals with change more easily and at scale
▪ Does not enforce requirements and models up front
▪ Does not limit the format or structure of data
▪ Assumes the range of data latencies in and out, from
streaming to one-time bulk
▪ Allows both reading and writing of data from outside
Copyright Third Nature, Inc.
The architecture from 1988 we SHOULD HAVE BEEN USING
The general concept of a
separate architecture for BI
has been around longer, but
this paper by Devlin and
Murphy is the first formal
data warehouse architecture
and definition published.
91
“An architecture for a business and
information system”, B. A. Devlin,
P. T. Murphy, IBM Systems Journal,
Vol.27, No. 1, (1988)
Slide 91Copyright Third Nature, Inc.
Copyright Third Nature, Inc.
The goal is to decouple: solve the application and
infrastructure problems separately
Platform Services
Data Access
Deliver & Use
Data storage
This separates BI from other uses of data, allowing each type of
use to structure the data specific to its own requirements.
Data access is already
somewhat separate
today. Make the
separation of different
access methods a formal
part of the architecture.
Don’t force one model.
Copyright Third Nature, Inc.
The goal is to decouple: solve the application and
infrastructure problems separately
Platform Services
Data Management
Process & Integrate
Data storage
Data management should not be subject to the constraints of a single use
Data management
was blended with both
data acquisition and
structuring data for
client tools. It should
be an independent
function.
Copyright Third Nature, Inc.
The goal is to decouple: solve the application and
infrastructure problems separately
Data Acquisition
Collect & Store
Incremental
Batch
One-time copy
Real time
Platform Services
Data storage
Data arrives in many latencies, from real-time to one-time. Acquisition
can’t be limited by the management or consumption layers.
Data acquisition should not be
directly tied to the needs of
consumption. It must operate
independently of data use.
Copyright Third Nature, Inc.
The full analytic environment subsumes all the functions
of the data warehouse, and extends them
Data Acquisition
Collect & Store
Incremental
Batch
One-time copy
Real time
Platform Services
Data Management
Process & Integrate
Data Access
Deliver & Use
Data storage
The platform has to do more than serve queries; it has to be read-write.
Copyright Third Nature, Inc.
The data architecture must align with system components
because each of them addresses different data needs
Incremental
Collect
Batch
One-time copy
Real time
Manage & Integrate
Data Acquisition
Collect & Store
Data Management
Process & Integrate
Data Access
Deliver & Use
Separating concerns is part of the mechanism for change isolation
Copyright Third Nature, Inc.
Zone 1: Acquistion
▪ Focus is only on collecting and tracking data
▪ Direct recording of data from sources
▪ Each dataset has as much schema as the source
provides, could be explicit or implicit or none
▪ Foundation layer for all subsequent use
Zone 1
Transactions Events Objects
Copyright Third Nature, Inc.
Zone 2
Zone 2: Integration - Standardized
▪ Parsed and cataloged – all structure and form are
made explicit so data can be accessed
▪ Namespace (e.g. keys) managed
▪ Common elements are standardized
▪ Datasets are profiled, indexed, available
Zone 1
Transactions Events Objects
Copyright Third Nature, Inc.
Zone 2
Zone 2
Zone 2: Integration - Enhanced
▪ Common shared KPIs, master datasets
▪ Extracted and derived data available, e.g. NEE output
▪ Data is linkable, labeled, possibly cleaned
Zone 1
Transactions Events Objects
Copyright Third Nature, Inc.
Layer 1
Layer 2
Layer 0
Zone 3
Zone 3: Access
▪ Data structured to suit the workload
▪ Integrated data for a specific purpose
▪ Business rules applied, e.g. filters, controls, etc.
▪ Designed, explicit structures
▪ Generally repeatable use
Copyright Third Nature, Inc.
Layer 0
Layer 1
Layer 2
Zone 3
Zone 4
Zone 4: Transient
▪ Under business / developer control
▪ The data for one-off projects of unknown value
or repeatability, integrated from other layers
▪ Place for ephemeral analytics output
▪ The sandbox
Copyright Third Nature, Inc.
Zone 1
Raw
Zone 2
Standardized
Zone 2
Enhanced
Zone 3
Managed
Zone 4
Transient
Decoupled data architecture
Copyright Third Nature, Inc.
Food supply chain: an analogy for analytic data
Multiple contexts of use, differing quality levels
You need to keep the original because just like baking,
you can’t unmake dough once it’s mixed.
Copyright Third Nature, Inc.
The design focus is different in each area
Ingredients
Goal: available
User needs a recipe
in order to make
use of the data.
Pre-mixed
Goal: discoverable
and integrateable
User needs a menu
to choose from the
data available
Meals
Goal: usable
User needs utensils
but is given a
finished meal
Copyright Third Nature, Inc.
The data is in zones of management, not isolating layers
Raw data in an immutable
storage area
Standardized or
enhanced data
Common or
usage-
specific
data
Transient data
Relax control to enable self-service while avoiding a mess.
Do not constrain access to one zone or to a single tool.
Focus on visibility of data use, not control of data.
Copyright Third Nature, Inc.
This data architecture resolves rate of change problems
Raw data in an immutable
storage area
Standardized or
enhanced data
Common or
usage-
specific data
Transient data
New data of
unknown value,
simple requests for
new data can land
here first, with little
work by IT.
More effort applied to
management, slower.
Optimized for
specific uses /
workloads. Generally
the slowest change.
Not fast vs slow:
fast vs right
Not flexibility vs
control: flexibility
vs repeatability
Agile for
structure change
vs agile for
questions / use
Copyright Third Nature, Inc.
SQLServer
RDBMS
Example: data environment, mid-size retailer
Replication, batch ETL
to collect data
Web Application
Standard retail /
wholesale data
warehouse model.
Nightly batch model
from stage, contains
~ 1 TB of data
DW
New requirement:
Load all the online marketing and web activity for
use in analytic models (not simple web analytics)
Will add ~10 TB of data, continuous or hourly loads
Copyright Third Nature, Inc.
SQLServer
RDBMS
Example: data environment, mid-size retailer
Replication, batch ETL
to collect data
Hadoop
Web Application
Persistent store,
data warehouse on
same DB in
different schemas
DW
Log file fetch & load for clickstream,
summaries sent to reporting env
Discovery work usually
done in Hadoop,
sometimes in database.
Copyright Third Nature, Inc.
SQLServer
This data architecture uses the 3 zone pattern
Hadoop
Web Application
RDBMS
DW
Raw data in an immutable
storage area
Standardized or
enhanced data
Common or
usage-
specific data
Transient data
All reporting data
and analytics
output is sent to
the DW where it is
can be accessed.
Cleaned,
summarized or
derived data is
stored in DB
Raw data is stored
in two places:
relational and small
sets in DB, log
collection in Hadoop
Copyright Third Nature, Inc.
The concept of a zone is not a physical system. It’s data architecture
Apps
DW
The biggest decision is to separate all data collection from
the data integration from consumption.
Physical system/technology overlays are separate, depend
on the specific use cases and needs of the organization.
Dist FS
RDBMSs
Decompress &
process device logs
RDBMS
RDBMS
RDBMS
Discovery
platform
Copyright Third Nature, Inc.
What about the technology? Do I need an <X>?
112
Blended Architectures Are a Requirement,
Not an Option
Data Warehouse + Data Lake
On Premise + Cloud
RDBMS + S3 + HDFS
Commercial + Open Source
Copyright Third Nature, Inc.
Bricks are not buildings
We don’t think this is equivalent to this
Architecture is not technology. It’s not a product you can buy.
Copyright Third Nature, Inc.Copyright Third Nature, Inc.
New tools and tool architectures are required
One tool for all jobs doesn’t work any more*
* it never did, we just had fewer jobs
Copyright Third Nature, Inc.
Beware overengineering: do you really need special
new technology infrastructure?
Numberofpeople
The distribution of data
size is about normal, yet
these guys set the tone of
the market today.
Bigness of data
Copyright Third Nature, Inc.
Analytics: This is really raw data under storageNumberofjobs
Microsoft study of 174,000 analytic
jobs in their petabyte scale cluster:
median size = ???
Bigness of data
Copyright Third Nature, Inc.
Copyright Third Nature, Inc.
Working data for analytics most often not bigNumberofjobs
14 GB
Smallness of data
Copyright Third Nature, Inc.
The usable size distribution
Copyright Third Nature, Inc.
TANSTAAFL
When replacing the old
with the new (or ignoring
the new over the old) you
always make tradeoffs,
and usually you won’t see
them for a long time.
Technologies are not
perfect replacements for
one another. Often not
better, only different.
© Third Nature Inc.© Third Nature Inc.
1.It’s not about storing data, it’s about using it
2.Use drives architecture. Understand the uses,
what you are designing for, to drive decisions.
3.Put data at the center, not technology. Don’t let
the tech define what you can do or how you do
it.
4.The death star is not the answer. The data model
is not a flat earth. You are not building a
monolith.
5.Know your history. Avoiding wheel reinvention
saves time, money, careers.
Conclusion
© Third Nature Inc.
Manage your data
(or it will manage you)
Data management is where
developers are weakest.
Modern engineering
practices are where data
management is weakest.
You need to bridge these
groups and practices in the
organization if you want to
do meaningful work with
event stream data.
© Third Nature Inc.
“Now is not the end.
It is not even the
beginning of the end.
But it is, perhaps,
the end of the beginning.”
Winston Churchill
Mark Madsen is the global head of
architecture at Teradata, Prior to that
he was president of Third Nature, a
research and consulting firm focused
on analytics, data integration and data
management. Mark is an award-
winning author, architect and CTO
whose work has been featured in
numerous industry publications. Over
the past ten years Mark received
awards for his work from the American
Productivity & Quality Center, TDWI,
and the Smithsonian Institute. He is an
international speaker, chairs several
conferences, and is on the O’Reilly
Strata program committee. For more
information or to contact Mark, follow
@markmadsen on Twitter or visit
http://ThirdNature.net
About the Presenter
Todd Walter
Chief Technologist - Teradata
• Chief Technologist for Teradata
• A pragmatic visionary, Walter helps business leaders,
analysts and technologists better understand all of the
astonishing possibilities of big data and analytics
• Works with organizations of all sizes and levels of
experience at the leading edge of adopting big data, data
warehouse and analytics technologies
• With Teradata for more than 30 years and served for more
than 10 years as CTO of Teradata Labs, contributing
significantly to Teradata’s unique design features and
functionality
• Holds more than a dozen Teradata patents and is a
Teradata Fellow in recognition of his long record of
technical innovation and contribution to the company

Más contenido relacionado

La actualidad más candente

Assumptions about Data and Analysis: Briefing room webcast slides
Assumptions about Data and Analysis: Briefing room webcast slidesAssumptions about Data and Analysis: Briefing room webcast slides
Assumptions about Data and Analysis: Briefing room webcast slidesmark madsen
 
Strata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
Strata Data Conference 2019 : Scaling Visualization for Big Data in the CloudStrata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
Strata Data Conference 2019 : Scaling Visualization for Big Data in the CloudJaipaul Agonus
 
Everything Has Changed Except Us: Modernizing the Data Warehouse
Everything Has Changed Except Us: Modernizing the Data WarehouseEverything Has Changed Except Us: Modernizing the Data Warehouse
Everything Has Changed Except Us: Modernizing the Data Warehousemark madsen
 
Analytics 3.0 Measurable business impact from analytics & big data
Analytics 3.0 Measurable business impact from analytics & big dataAnalytics 3.0 Measurable business impact from analytics & big data
Analytics 3.0 Measurable business impact from analytics & big dataMicrosoft
 
How to Build Data Science Teams
How to Build Data Science TeamsHow to Build Data Science Teams
How to Build Data Science TeamsGanes Kesari
 
Building Data Science Teams
Building Data Science TeamsBuilding Data Science Teams
Building Data Science TeamsEMC
 
Managing Data Science | Lessons from the Field
Managing Data Science | Lessons from the Field Managing Data Science | Lessons from the Field
Managing Data Science | Lessons from the Field Domino Data Lab
 
Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?mark madsen
 
Intro to Data Science for Non-Data Scientists
Intro to Data Science for Non-Data ScientistsIntro to Data Science for Non-Data Scientists
Intro to Data Science for Non-Data ScientistsSri Ambati
 
Embracing data science
Embracing data scienceEmbracing data science
Embracing data scienceVipul Kalamkar
 
Wake up and smell the data
Wake up and smell the dataWake up and smell the data
Wake up and smell the datamark madsen
 
Bi isn't big data and big data isn't BI (updated)
Bi isn't big data and big data isn't BI (updated)Bi isn't big data and big data isn't BI (updated)
Bi isn't big data and big data isn't BI (updated)mark madsen
 
Big dataplatform operationalstrategy
Big dataplatform operationalstrategyBig dataplatform operationalstrategy
Big dataplatform operationalstrategyHimanshu Bari
 
O'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
O'Reilly ebook: Machine Learning at Enterprise Scale | QuboleO'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
O'Reilly ebook: Machine Learning at Enterprise Scale | QuboleVasu S
 
Briefing room: An alternative for streaming data collection
Briefing room: An alternative for streaming data collectionBriefing room: An alternative for streaming data collection
Briefing room: An alternative for streaming data collectionmark madsen
 
Building Data Science Teams: A Moneyball Approach
Building Data Science Teams: A Moneyball ApproachBuilding Data Science Teams: A Moneyball Approach
Building Data Science Teams: A Moneyball Approachjoshwills
 
Everything has changed except us
Everything has changed except usEverything has changed except us
Everything has changed except usmark madsen
 

La actualidad más candente (20)

Assumptions about Data and Analysis: Briefing room webcast slides
Assumptions about Data and Analysis: Briefing room webcast slidesAssumptions about Data and Analysis: Briefing room webcast slides
Assumptions about Data and Analysis: Briefing room webcast slides
 
Strata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
Strata Data Conference 2019 : Scaling Visualization for Big Data in the CloudStrata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
Strata Data Conference 2019 : Scaling Visualization for Big Data in the Cloud
 
Everything Has Changed Except Us: Modernizing the Data Warehouse
Everything Has Changed Except Us: Modernizing the Data WarehouseEverything Has Changed Except Us: Modernizing the Data Warehouse
Everything Has Changed Except Us: Modernizing the Data Warehouse
 
Analytics 3.0 Measurable business impact from analytics & big data
Analytics 3.0 Measurable business impact from analytics & big dataAnalytics 3.0 Measurable business impact from analytics & big data
Analytics 3.0 Measurable business impact from analytics & big data
 
How to Build Data Science Teams
How to Build Data Science TeamsHow to Build Data Science Teams
How to Build Data Science Teams
 
Building Data Science Teams
Building Data Science TeamsBuilding Data Science Teams
Building Data Science Teams
 
Managing Data Science | Lessons from the Field
Managing Data Science | Lessons from the Field Managing Data Science | Lessons from the Field
Managing Data Science | Lessons from the Field
 
Machine Learning in Big Data
Machine Learning in Big DataMachine Learning in Big Data
Machine Learning in Big Data
 
Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?Disruptive Innovation: how do you use these theories to manage your IT?
Disruptive Innovation: how do you use these theories to manage your IT?
 
Intro to Data Science for Non-Data Scientists
Intro to Data Science for Non-Data ScientistsIntro to Data Science for Non-Data Scientists
Intro to Data Science for Non-Data Scientists
 
Embracing data science
Embracing data scienceEmbracing data science
Embracing data science
 
Wake up and smell the data
Wake up and smell the dataWake up and smell the data
Wake up and smell the data
 
Bi isn't big data and big data isn't BI (updated)
Bi isn't big data and big data isn't BI (updated)Bi isn't big data and big data isn't BI (updated)
Bi isn't big data and big data isn't BI (updated)
 
Big dataplatform operationalstrategy
Big dataplatform operationalstrategyBig dataplatform operationalstrategy
Big dataplatform operationalstrategy
 
O'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
O'Reilly ebook: Machine Learning at Enterprise Scale | QuboleO'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
O'Reilly ebook: Machine Learning at Enterprise Scale | Qubole
 
Briefing room: An alternative for streaming data collection
Briefing room: An alternative for streaming data collectionBriefing room: An alternative for streaming data collection
Briefing room: An alternative for streaming data collection
 
Building Data Science Teams: A Moneyball Approach
Building Data Science Teams: A Moneyball ApproachBuilding Data Science Teams: A Moneyball Approach
Building Data Science Teams: A Moneyball Approach
 
Keynote Dubai
Keynote DubaiKeynote Dubai
Keynote Dubai
 
Everything has changed except us
Everything has changed except usEverything has changed except us
Everything has changed except us
 
Analytics3.0 e book
Analytics3.0 e bookAnalytics3.0 e book
Analytics3.0 e book
 

Similar a Architecting a Platform for Enterprise Use - Strata London 2018

The Role of Data Wrangling in Driving Hadoop Adoption
The Role of Data Wrangling in Driving Hadoop AdoptionThe Role of Data Wrangling in Driving Hadoop Adoption
The Role of Data Wrangling in Driving Hadoop AdoptionInside Analysis
 
Expert Big Data Tips
Expert Big Data TipsExpert Big Data Tips
Expert Big Data TipsQubole
 
Analytics and Big Data Analytics
Analytics and Big Data AnalyticsAnalytics and Big Data Analytics
Analytics and Big Data AnalyticsInside Analysis
 
Putting data science in your business a first utility feedback
Putting data science in your business a first utility feedbackPutting data science in your business a first utility feedback
Putting data science in your business a first utility feedbackPeculium Crypto
 
From Lab to Factory: Or how to turn data into value
From Lab to Factory: Or how to turn data into valueFrom Lab to Factory: Or how to turn data into value
From Lab to Factory: Or how to turn data into valuePeadar Coyle
 
Big Data
Big DataBig Data
Big DataNGDATA
 
Best Practices for Scaling Data Science Across the Organization
Best Practices for Scaling Data Science Across the OrganizationBest Practices for Scaling Data Science Across the Organization
Best Practices for Scaling Data Science Across the OrganizationChasity Gibson
 
The Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value ThereafterThe Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value ThereafterInside Analysis
 
How to make your data scientists happy
How to make your data scientists happy How to make your data scientists happy
How to make your data scientists happy Hussain Sultan
 
How to Prepare for a Career in Data Science
How to Prepare for a Career in Data ScienceHow to Prepare for a Career in Data Science
How to Prepare for a Career in Data ScienceJuuso Parkkinen
 
SDD2017 - 03 Abed Ajraou - putting data science in your business a first uti...
SDD2017 - 03 Abed Ajraou  - putting data science in your business a first uti...SDD2017 - 03 Abed Ajraou  - putting data science in your business a first uti...
SDD2017 - 03 Abed Ajraou - putting data science in your business a first uti...Dario Mangano
 
The 3 Key Barriers Keeping Companies from Deploying Data Products
The 3 Key Barriers Keeping Companies from Deploying Data Products The 3 Key Barriers Keeping Companies from Deploying Data Products
The 3 Key Barriers Keeping Companies from Deploying Data Products Dataiku
 
Python's Role in the Future of Data Analysis
Python's Role in the Future of Data AnalysisPython's Role in the Future of Data Analysis
Python's Role in the Future of Data AnalysisPeter Wang
 
Data Mining and Data Warehouse
Data Mining and Data WarehouseData Mining and Data Warehouse
Data Mining and Data WarehouseAnupam Sharma
 
The Python ecosystem for data science - Landscape Overview
The Python ecosystem for data science - Landscape OverviewThe Python ecosystem for data science - Landscape Overview
The Python ecosystem for data science - Landscape OverviewDr. Ananth Krishnamoorthy
 
From Rocket Science to Data Science
From Rocket Science to Data ScienceFrom Rocket Science to Data Science
From Rocket Science to Data ScienceSanghamitra Deb
 
Data science presentation
Data science presentationData science presentation
Data science presentationMSDEVMTL
 
Overview of mit sloan case study on ge data and analytics initiative titled g...
Overview of mit sloan case study on ge data and analytics initiative titled g...Overview of mit sloan case study on ge data and analytics initiative titled g...
Overview of mit sloan case study on ge data and analytics initiative titled g...Gregg Barrett
 

Similar a Architecting a Platform for Enterprise Use - Strata London 2018 (20)

The Role of Data Wrangling in Driving Hadoop Adoption
The Role of Data Wrangling in Driving Hadoop AdoptionThe Role of Data Wrangling in Driving Hadoop Adoption
The Role of Data Wrangling in Driving Hadoop Adoption
 
Expert Big Data Tips
Expert Big Data TipsExpert Big Data Tips
Expert Big Data Tips
 
Analytics and Big Data Analytics
Analytics and Big Data AnalyticsAnalytics and Big Data Analytics
Analytics and Big Data Analytics
 
Putting data science in your business a first utility feedback
Putting data science in your business a first utility feedbackPutting data science in your business a first utility feedback
Putting data science in your business a first utility feedback
 
From Lab to Factory: Or how to turn data into value
From Lab to Factory: Or how to turn data into valueFrom Lab to Factory: Or how to turn data into value
From Lab to Factory: Or how to turn data into value
 
Big Data
Big DataBig Data
Big Data
 
Best Practices for Scaling Data Science Across the Organization
Best Practices for Scaling Data Science Across the OrganizationBest Practices for Scaling Data Science Across the Organization
Best Practices for Scaling Data Science Across the Organization
 
Proposed Talk Outline for Pycon2017
Proposed Talk Outline for Pycon2017 Proposed Talk Outline for Pycon2017
Proposed Talk Outline for Pycon2017
 
The Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value ThereafterThe Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value Thereafter
 
How to make your data scientists happy
How to make your data scientists happy How to make your data scientists happy
How to make your data scientists happy
 
How to Prepare for a Career in Data Science
How to Prepare for a Career in Data ScienceHow to Prepare for a Career in Data Science
How to Prepare for a Career in Data Science
 
SDD2017 - 03 Abed Ajraou - putting data science in your business a first uti...
SDD2017 - 03 Abed Ajraou  - putting data science in your business a first uti...SDD2017 - 03 Abed Ajraou  - putting data science in your business a first uti...
SDD2017 - 03 Abed Ajraou - putting data science in your business a first uti...
 
The 3 Key Barriers Keeping Companies from Deploying Data Products
The 3 Key Barriers Keeping Companies from Deploying Data Products The 3 Key Barriers Keeping Companies from Deploying Data Products
The 3 Key Barriers Keeping Companies from Deploying Data Products
 
The REAL face of Big Data
The REAL face of Big DataThe REAL face of Big Data
The REAL face of Big Data
 
Python's Role in the Future of Data Analysis
Python's Role in the Future of Data AnalysisPython's Role in the Future of Data Analysis
Python's Role in the Future of Data Analysis
 
Data Mining and Data Warehouse
Data Mining and Data WarehouseData Mining and Data Warehouse
Data Mining and Data Warehouse
 
The Python ecosystem for data science - Landscape Overview
The Python ecosystem for data science - Landscape OverviewThe Python ecosystem for data science - Landscape Overview
The Python ecosystem for data science - Landscape Overview
 
From Rocket Science to Data Science
From Rocket Science to Data ScienceFrom Rocket Science to Data Science
From Rocket Science to Data Science
 
Data science presentation
Data science presentationData science presentation
Data science presentation
 
Overview of mit sloan case study on ge data and analytics initiative titled g...
Overview of mit sloan case study on ge data and analytics initiative titled g...Overview of mit sloan case study on ge data and analytics initiative titled g...
Overview of mit sloan case study on ge data and analytics initiative titled g...
 

Más de mark madsen

A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou RangeA Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Rangemark madsen
 
A Pragmatic Approach to Analyzing Customers
A Pragmatic Approach to Analyzing CustomersA Pragmatic Approach to Analyzing Customers
A Pragmatic Approach to Analyzing Customersmark madsen
 
Building the Enterprise Data Lake: A look at architecture
Building the Enterprise Data Lake: A look at architectureBuilding the Enterprise Data Lake: A look at architecture
Building the Enterprise Data Lake: A look at architecturemark madsen
 
Briefing Room analyst comments - streaming analytics
Briefing Room analyst comments - streaming analyticsBriefing Room analyst comments - streaming analytics
Briefing Room analyst comments - streaming analyticsmark madsen
 
On the edge: analytics for the modern enterprise (analyst comments)
On the edge: analytics for the modern enterprise (analyst comments)On the edge: analytics for the modern enterprise (analyst comments)
On the edge: analytics for the modern enterprise (analyst comments)mark madsen
 
Crossing the chasm with a high performance dynamically scalable open source p...
Crossing the chasm with a high performance dynamically scalable open source p...Crossing the chasm with a high performance dynamically scalable open source p...
Crossing the chasm with a high performance dynamically scalable open source p...mark madsen
 
Don't let data get in the way of a good story
Don't let data get in the way of a good storyDon't let data get in the way of a good story
Don't let data get in the way of a good storymark madsen
 
Big Data and Bad Analogies
Big Data and Bad AnalogiesBig Data and Bad Analogies
Big Data and Bad Analogiesmark madsen
 
Don't follow the followers
Don't follow the followersDon't follow the followers
Don't follow the followersmark madsen
 
Exploring cloud for data warehousing
Exploring cloud for data warehousingExploring cloud for data warehousing
Exploring cloud for data warehousingmark madsen
 
Open Data: Free Data Isn't the Same as Freeing Data
Open Data: Free Data Isn't the Same as Freeing DataOpen Data: Free Data Isn't the Same as Freeing Data
Open Data: Free Data Isn't the Same as Freeing Datamark madsen
 
Exploring cloud for data warehousing
Exploring cloud for data warehousingExploring cloud for data warehousing
Exploring cloud for data warehousingmark madsen
 
Big Data Wonderland: Two Views on the Big Data Revolution
Big Data Wonderland: Two Views on the Big Data RevolutionBig Data Wonderland: Two Views on the Big Data Revolution
Big Data Wonderland: Two Views on the Big Data Revolutionmark madsen
 
Using Data Virtualization to Integrate With Big Data
Using Data Virtualization to Integrate With Big DataUsing Data Virtualization to Integrate With Big Data
Using Data Virtualization to Integrate With Big Datamark madsen
 
One Size Doesn't Fit All: The New Database Revolution
One Size Doesn't Fit All: The New Database RevolutionOne Size Doesn't Fit All: The New Database Revolution
One Size Doesn't Fit All: The New Database Revolutionmark madsen
 

Más de mark madsen (15)

A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou RangeA Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
A Brief Tour through the Geology & Endemic Botany of the Klamath-Siskiyou Range
 
A Pragmatic Approach to Analyzing Customers
A Pragmatic Approach to Analyzing CustomersA Pragmatic Approach to Analyzing Customers
A Pragmatic Approach to Analyzing Customers
 
Building the Enterprise Data Lake: A look at architecture
Building the Enterprise Data Lake: A look at architectureBuilding the Enterprise Data Lake: A look at architecture
Building the Enterprise Data Lake: A look at architecture
 
Briefing Room analyst comments - streaming analytics
Briefing Room analyst comments - streaming analyticsBriefing Room analyst comments - streaming analytics
Briefing Room analyst comments - streaming analytics
 
On the edge: analytics for the modern enterprise (analyst comments)
On the edge: analytics for the modern enterprise (analyst comments)On the edge: analytics for the modern enterprise (analyst comments)
On the edge: analytics for the modern enterprise (analyst comments)
 
Crossing the chasm with a high performance dynamically scalable open source p...
Crossing the chasm with a high performance dynamically scalable open source p...Crossing the chasm with a high performance dynamically scalable open source p...
Crossing the chasm with a high performance dynamically scalable open source p...
 
Don't let data get in the way of a good story
Don't let data get in the way of a good storyDon't let data get in the way of a good story
Don't let data get in the way of a good story
 
Big Data and Bad Analogies
Big Data and Bad AnalogiesBig Data and Bad Analogies
Big Data and Bad Analogies
 
Don't follow the followers
Don't follow the followersDon't follow the followers
Don't follow the followers
 
Exploring cloud for data warehousing
Exploring cloud for data warehousingExploring cloud for data warehousing
Exploring cloud for data warehousing
 
Open Data: Free Data Isn't the Same as Freeing Data
Open Data: Free Data Isn't the Same as Freeing DataOpen Data: Free Data Isn't the Same as Freeing Data
Open Data: Free Data Isn't the Same as Freeing Data
 
Exploring cloud for data warehousing
Exploring cloud for data warehousingExploring cloud for data warehousing
Exploring cloud for data warehousing
 
Big Data Wonderland: Two Views on the Big Data Revolution
Big Data Wonderland: Two Views on the Big Data RevolutionBig Data Wonderland: Two Views on the Big Data Revolution
Big Data Wonderland: Two Views on the Big Data Revolution
 
Using Data Virtualization to Integrate With Big Data
Using Data Virtualization to Integrate With Big DataUsing Data Virtualization to Integrate With Big Data
Using Data Virtualization to Integrate With Big Data
 
One Size Doesn't Fit All: The New Database Revolution
One Size Doesn't Fit All: The New Database RevolutionOne Size Doesn't Fit All: The New Database Revolution
One Size Doesn't Fit All: The New Database Revolution
 

Último

Brain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptxBrain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptxShammiRai3
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-ProfitsTimothy Spann
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsNeo4j
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptxFurkanTasci3
 
Using DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data WarehouseUsing DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data WarehouseThinkInnovation
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingMarketingTrips
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTimothy Spann
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe321k
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxEmmanuel Dauda
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptxFurkanTasci3
 
Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media PlatformsMahmoud Yasser
 
Stochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxStochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxjkmrshll88
 
Understanding the Impact of video length on student performance
Understanding the Impact of video length on student performanceUnderstanding the Impact of video length on student performance
Understanding the Impact of video length on student performancePrithaVashisht1
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfmxlos0
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfJasonBoboKyaw
 
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsGain Insights
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Neo4j
 

Último (20)

Brain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptxBrain Tumor Detection with Machine Learning.pptx
Brain Tumor Detection with Machine Learning.pptx
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge Graphs
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
 
Using DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data WarehouseUsing DAX & Time-based Analysis in Data Warehouse
Using DAX & Time-based Analysis in Data Warehouse
 
Báo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân MarketingBáo cáo Social Media Benchmark 2024 cho dân Marketing
Báo cáo Social Media Benchmark 2024 cho dân Marketing
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
 
The market for cross-border mortgages in Europe
The market for cross-border mortgages in EuropeThe market for cross-border mortgages in Europe
The market for cross-border mortgages in Europe
 
Data Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potxData Analytics Fundamentals: data analytics types.potx
Data Analytics Fundamentals: data analytics types.potx
 
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptxSTOCK PRICE ANALYSIS  Furkan Ali TASCI --.pptx
STOCK PRICE ANALYSIS Furkan Ali TASCI --.pptx
 
Data Collection from Social Media Platforms
Data Collection from Social Media PlatformsData Collection from Social Media Platforms
Data Collection from Social Media Platforms
 
Stochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptxStochastic Dynamic Programming and You.pptx
Stochastic Dynamic Programming and You.pptx
 
Understanding the Impact of video length on student performance
Understanding the Impact of video length on student performanceUnderstanding the Impact of video length on student performance
Understanding the Impact of video length on student performance
 
Microeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdfMicroeconomic Group Presentation Apple.pdf
Microeconomic Group Presentation Apple.pdf
 
Air Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdfAir Con Energy Rating Info411 Presentation.pdf
Air Con Energy Rating Info411 Presentation.pdf
 
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
 
Target_Company_Data_breach_2013_110million
Target_Company_Data_breach_2013_110millionTarget_Company_Data_breach_2013_110million
Target_Company_Data_breach_2013_110million
 
Empowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded AnalyticsEmpowering Decisions A Guide to Embedded Analytics
Empowering Decisions A Guide to Embedded Analytics
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
 

Architecting a Platform for Enterprise Use - Strata London 2018

  • 1. Copyright Third Nature, Inc. Architecting a Data Platform For Enterprise Use May 2018 Mark Madsen Todd Walter
  • 2. Copyright Third Nature, Inc. The business complaints about data and analytics You really should fire IT.
  • 3. Copyright Third Nature, Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio
  • 4. Copyright Third Nature, Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes How business responds How IT responsds Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency SaaS / Cloud BI Consultants Self-service BI Shadow BI Workarounds & spreadmarts Hidden costs Loss of control Loss of visibility Loss of knowledge Security risks Bad data risks 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio
  • 5. Copyright Third Nature, Inc.Copyright Third Nature, Inc. Data deficient Takes too long Costs too much Function deficient IT root causes IT proximate causes What is said in disputes How business responds How IT responds Lack of agility People & vendor cost basis Client-server infrastructure Lack of “good enough” competency SaaS / Cloud BI Consultants Self-service BI Shadow BI Workarounds & spreadmarts Hidden costs Loss of control Loss of visibility Loss of knowledge Security risks Bad data risks 1980s-era methods Inappropriate technology Data hygiene fetishes Vendor lock 1990s-era procurement IT skills deficit Dysfunctional OLTP portfolio This is not a technology problem – it is an architecture problem.
  • 6. Copyright Third Nature, Inc.Copyright Third Nature, Inc. What do we hear in the field? "I want to do analytics, beyond what I can do with BI tools" "I need an analytics strategy" "I need an analytics roadmap" "I have a specific analytics project of type..." “We have a data lake. What should we do with it?” “Technology Y replaces technology X” "I want to modernize my DW" aka "I want to speed up my DW" “Don’t need schema, curation, ETL, governance … anymore” Good judgement is the result of experience. Experience is the result of bad judgement. —Fred Brooks
  • 7. Copyright Third Nature, Inc. “Build a data lake and put all the data there first”
  • 8. Copyright Third Nature, Inc. DW: Centralize, that solves all problems! Creates bottlenecks Causes scale problems Availability?
  • 9. Copyright Third Nature, Inc. The data lake solution: no central authority wtf, it was fully operational!
  • 10. Copyright Third Nature, Inc. The data lake solution? There’s a problem: as the lake is envisioned, it is still a centralized data architecture, but this time there is no single global model. Instead it’s files and not modeled. It can be operational while under construction. It’s still a death star.
  • 11. Copyright Third Nature, Inc. Eventually we run into the same problems Seriously, wtf? It was agile and operational Rising complexity and scale break centralized models
  • 12. Copyright Third Nature, Inc. “Standardize on one tech, it’s simpler for everyone” It’s called stack think. Pick your vendor. Cede all architecture.
  • 13. Copyright Third Nature, Inc. Agile your way into the Analytics Shantytown
  • 14. Copyright Third Nature, Inc.Copyright Third Nature, Inc. “Infrastructure one PoC at a time” • Use case driven • Leverage (any) new technology • Re-use of the technology stack • Data is captive to the application or to the technology stack • Developers tend to think “function first” • Analytics people also think “function first”, but generally require integrated data
  • 15. Copyright Third Nature, Inc.Copyright Third Nature, Inc. A key aspect of operational analytics is “end-to-end” E2E does not allow you to take a random walk through technologies and BYOT because the complexity of integration leads to a mountain of technical debt. • Most data applications are organized around a process, not a task or function. • Real-time analytics systems pass their data over the network, but the dependencies can cross application boundaries, as well as requiring persistence. • Developers are being forced to think of data outside the local context.
  • 16. Copyright Third Nature, Inc. “Start with the platform. The rest will follow.”
  • 17. Copyright Third Nature, Inc.Copyright Third Nature, Inc. Big data reality: What users seeBig data promise: What you see
  • 18. Copyright Third Nature, Inc. Persisting data is not the end of the line. If you stop here you win the battle and lose the war
  • 19. Copyright Third Nature, Inc. We don’t have an analytics problem, just like we didn’t have a BI problem The origin of analytics as “business intelligence” was stated well in 1958: …the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal. ~ H. P. Luhn “A Business Intelligence System”, http://altaplana.com/ibmrd0204H.pdf ” “ Our goal is analytics as a capability, not a technology
  • 20. Copyright Third Nature, Inc. Three constituencies Stakeholder Analyst Builder aka the recipient aka the data scientist aka the engineer
  • 21. Copyright Third Nature, Inc. Starting points for analytics strategy Many organizations choose to start with the analysts. Create a data science team. Turn them loose to find a problem. Many more start with builders: technology solutions looking for problems, e.g. 65% of the IT driven Hadoop and Spark projects over the last five years. The right place to start? Stakeholders. The goal to achieve, the problem to solve.
  • 22. Copyright Third Nature, Inc. NATURE OF THE PROBLEM FROM THE STAKEHOLDER’S PERSPECTIVE Each constituency has their own set of problems to deal with
  • 23. BI and Analysis: Repeatability and Discoverability Discover Explore Analyze Model Consume Promote Focus on repeatability Application cycle time 90% of users Focus on discoverability Analyst cycle time 9% analysts, 1% data scientists 80% of data use 80% of analysis
  • 24. The myth that still drives analytics – analytic gold All we need is a fat pipe and pans working in parallel…
  • 25. Analytic insights that result in no action are expensive trivia. It’s not the insight, but what you do with it, that matters
  • 26. Applying analytics is not an analytics problem Applying analytics is not in the analyst’s control. It’s not in the engineer’s control. It’s in the control of the people involved in the process. Failures are often in execution, not in analytics development. For example, we saw unexpectedly poor performance in a number of geographies. Was it the new analytics we tried? Was it a data problem? No, it was a simple compliance problem.
  • 27. NATURE OF THE PROBLEM FROM THE ANALYST’S PERSPECTIVE
  • 28. The analytics process at a high level Diagram: Kate Matsudaira
  • 29. The nature of analytics problems is researching the unknown rather than accessing the known. Repeat for each new problem Diagram: Kate Matsudaira
  • 30. The main hurdle: just getting the data Do you know where to find it? Because it’s may not be in the data warehouse. Do you have access to it? Is access fast enough? Because DWs are for QRD, not for moving huge piles of data. And ERP systems and SaaS apps are right out. These are why developers jump on “data lake” as the solution. But that’s a small, easy part.
  • 31. Do you have the right data? Many machine learning techniques require labeled (known good) training data: Supervised learning: a person has to define the correct output for some portion of the data. Data is divided into training sets used for model building and test sets for validating the results. • What is spam and what isn’t? • What does a fraudulent transaction look like 36
  • 32. Do you have enough of the right data? ML needs a lot, you may be disappointed in your efforts
  • 33. Where do analysts spend their time? mostly data work Slide 38 Define the business problem Translate the problem into an analytic context Select appropriate data Learn the data Create a model set Fix problems with data Transform data Build models Assess models Deploy models Assess results % of time spent 70% 30% Source: Michael Berry, Data Miners Inc.
  • 34. The analyst’s workspace in BI is relatively spare
  • 35. The analyst’s workspace needs to be more like a kitchen than like BI vending machines
  • 36. Copyright Third Nature, Inc.Copyright Third Nature, Inc. Schema control, space limitations, resource limitations, production lockdown Analytics work is a production workload, but…
  • 37. NATURE OF THE PROBLEM FROM THE BUILDER’S PERSPECTIVE
  • 38. IT and Ops people want to know “what to build?” Giant data platform? Self service tools?
  • 39. Analytics has different processes and workloads None of this analytics work is the same as what IT considered “analysis” to be, which is usually equated with BI or ad-hoc query. Ad-hoc analysis = Exploratory data analysis = Batch analytics = Real-time analytics A real analytics production workflow Hatch, CIKM ‘11 Slide 44
  • 40. The world changes, do the models? In BI you maintain ETL and schemas, in ML you maintain models. “Model decay” happens as the assumptions around which a model is built change, e.g. spam techniques change. When you adjust the model you need to know it is normal again ▪ Better save the data used to build the model ▪ Better save the model ▪ Baseline and measurements
  • 41. THREE PERSPECTIVES, ONE SOLUTION? There are requirements from all constituents. You need to put them together to have a complete picture of what’s needed.
  • 42. The missing stakeholder There is another stakeholder: analytics management - the CAO, CDO, VP of analytics, aka “your boss” if you’re a data scientist. This is the perspective and problems of the person responsible for oversight of the team and efforts is across the organization and across multiple projects
  • 43. Job #1 - Repeatability
  • 44. Job # 2 - Operational predictability
  • 45. Job #3 - Reproducibility
  • 46. You need a system of record for analytics
  • 47. There is an extensive list of requirements to support Primary requirements needed by constituents S D E Data catalog and ability to search it for datasets X X Self-service access to curated data X Self-service access to uncurated (unknown, new) data X X Temporary storage for working with data X Data integration, cleaning, transformation, preparation tools and environment X X Persistent storage for source data used by production models X X Persistent storage for training, testing, production data used by models X X Storage and management of models X X Deployment, monitoring, decommissioning models X Lineage, traceability of changes made for data used by models X X Lineage, traceability for model changes X X X Managing baseline data / metrics for comparing model performance X X X Managing ongoing data / metrics for tracking ongoing model performance X X X S = stakeholder, user, D = data scientist, analyst, E = engineer, developer
  • 48. © Third Nature Inc. How did we get to this point with BI & analytics? There’s a difference between having no past and actively rejecting it.
  • 49. 54 100 1,000 10,000 1 2 3 4 5 6 7 8 9 10 Customer Data Plateaus Add users, accumulate history, add attributes, add dimensions Entirely new data set enabled by new technology at new price point – value has to exceed cost (ROI). Earliest adopters have vision ahead of ROI
  • 50. 55 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 1 2 3 4 5 6 7 8 9 10 11 12 13 User Adoption of New Data Customer Data Users - Data Set 1 Users - Data Set 2 Users of Integrated Data User adoption of new data sets starts over. Very small number of experts growing to wider audience, sophisticated users moving to business analysts, then business users, B2B customers and even to consumers Greater value is derived when data sets are linked – see bigger picture (eg who buys what, when). Comes after initial extraction of easy value from standalone data
  • 51. 56 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1,000,000,000 10,000,000,000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 Data Size Plateaus over Time Lather, Rinse, Repeat. Trend line is roughly Moore’s Law. Delay or skip a generation if new data set is two orders of magnitude instead of one.
  • 52. 57 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 100,000,000 1,000,000,000 10,000,000,000 100,000,000,000 1,000,000,000,000 10,000,000,000,000 100,000,000,000,000 1,000,000,000,000,000 10,000,000,000,000,000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 Customer Data Size vs. 2x per 12mo Customer Data Moore's Law - 2x/18mo 2x/12 mo But all studies show demand/creation of data increasing significantly faster than Moore’s law If we started only 1 OOM behind, we are now 5 OOM away from user demand for data Data Exhaust
  • 53. 58 • <Store, Item, Week> • <Store, Item, Day> – Simple aggregations • Market Basket – Affinity – Link to person, demographics, HR • Inventory by SKU by store – Temporal, time series, forecasting – Link to product, marketing, market basket Retail Plateaus 2B records total for 9 quarters 2B records per day, keep 9 quarters
  • 54. 59 • Web Logs and traffic – Behavioral patterns – eg path linked to person, offers, other channels – Operations of the web site • Supply chain sensors – sampled at major event – Activity Based Costing – Link to customer, product, HR, planning • Social Media – Text analysis, Filtering, languages – Link to customer, sales, other channel interactions • Supply chain sensors – sampled at minutes or seconds – Telematics – Real time, Event detection, trending, static and dynamic rules – Link to HR, thresholds, forecasts, routing, planning Retail Plateaus 30B records per day
  • 55. 60 FINANCE Revenue Expenses Customers CUSTOMER CARE Customer Products Orders Case History SALES Orders Customers Products MARKETING Customers Orders Campaign History OPERATIONS Inventory Returns Manufacturing Supply Chain How many batteries are in inventory by plant? What is the trend of warranty costs? How many people made a warranty claim last week? How many sales have been made quarter to date? Which customers should get a communicatio n on extended warranties? 54 32 29 49 66
  • 57. 62 2855 Given the rise in warranty costs, isolate the problem to be a plant, then to a battery lot. Communicate with affected customers, who have not made a warranty claim on batteries, through Marketing and Customer Service channels to recall cars with affected batteries. Inventory Returns Manufacturing Supply Chain Customer Service Orders Revenue Expenses Case History Customers Products Pipeline Customers Campaign History FINANCE SALESMARKETING OPERATIONS CUSTOMER EXPERIENCE 2855
  • 58. 63 Manufacturing: Data Overlap Analysis New Business Improvement Opportunities through Data Leverage Sales Force Profitability Analysis 100% 80% 66% 24% 41% 66% 0% 24% 0% 24% Transportation Planning 11% 100% 87% 28% 64% 56% 22% 34% 13% 45% Production Planning 6% 57% 100% 19% 83% 35% 17% 28% 9% 40% Vendor Managed Inventory 12% 100% 100% 100% 78% 100% 61% 100% 39% 100% Global Pricing Rationalization 4% 50% 100% 17% 100% 27% 15% 29% 11% 41% Fulfillment (Perfect Order) 16% 94% 90% 48% 58% 100% 35% 53% 19% 56% Manufacturing Quality Optimization 0% 76% 88% 60% 66% 71% 100% 79% 43% 73% Preventative Maintenance Analysis 7% 75% 93% 61% 80% 68% 50% 100% 27% 82% Warranty Claims Analysis 0% 94% 100% 83% 100% 83% 94% 93% 100% 98% Quality Life Cycle Improvement 4% 57% 75% 35% 64% 41% 26% 47% 16% 100% SalesForce ProfitabilityAnalysis ProductionPlanning VendorManaged Inventory GlobalPricing Rationalization TransportationPlanning Fulfillment (PerfectOrder) ManufacturingQuality Optimization Preventative Maintenance Analysis WarrantyClaims Analysis QualityLifeCycle Improvement If Then
  • 59. 64
  • 60. 65 PRODUCT SENSOR SOCIAL MEDIA CUSTOMER CARE AUDIO RECORDINGS DIGITAL ADVERTISING CLICKSTREAM 65 41 32 19 28 How many visitors did we have to our hybrid cars microsite yesterday? What are the temperature readings for batteries by Manufacturer? What is the sentiment towards line of hybrid vehicles? Which customers likely expressed anger with customer care? Which ad creative generated the most clicks?
  • 61. 66 2855 High, Well Known Quality Directional Quality Unknown/Low quality Well Defined Data Model Curated JSON, XML,… Extracted Attributes Curation Required
  • 62. 67 2855 SENSOR DIGITAL ADVERTISING CLICKSTREAM INTERACTIONS RATINGS & REVIEWS CUSTOMER PORTAL INTERACTIONS EXTERNAL INTERACTIONS SOCIAL MEDIA IVR Routing RFID ELECTRONIC COMMERCE FINANCE SALES MARKETING Inventory Returns Manufacturing Supply Chain Customer Service Orders Revenue Expenses Case History Customers Products Pipeline Customers Campaign History OPERATIONS CUSTOMER CARE – AUDIO RECORDINGS Maps Telemetry SERVER LOGS CUSTOMER EXPERIENCE Enterprise Data
  • 64. 69 2855 Out of Deviation Sensor Readings Fraud Events CUSTOMER EXPERIENCE SALES MARKETING OPERATIONS Abandoned Carts Online Price Quotes Social Media Influencers FINANCE 11234 Minimum Viable Curation Minimum Viable Data Quality
  • 65. 70 2855 Out of Deviation Sensor Readings Fraud Events CUSTOMER EXPERIENCE SALES MARKETING OPERATIONS Abandoned Carts Online Price Quotes Social Media Influencers FINANCE 11234Sensor data formatting Unit Normalization Vehicle/version normalization Make/model/year selection Data Comprehension, Pipelines
  • 66. 71 2855 Enterprise Wide Use 10s to 100s of Users 10 Users Super Users Business Analysts Action list, Report, Dashboard Users Share across Enterprise Share in department Share over cube wall User Base and Sharing
  • 68. 73 2855 Production tools, curated data, integrated across business areas Wide variety of tools and data forms Targeted data access, many applications, response time SLAs Evolving Consumption Requirements High CPU, moderate to low IO Bulk scans, large computation, transformation, data access, specific integration Moderate CPU, High IO, Resource management
  • 69. 74 2855 Relational store more appropriate Hybrid File Store more appropriate Technology, Capacity 100 TB 1 PB 10 TB
  • 70. 75 2855 MANUFACTURING CAMPAIGN HISTORY COSTS PRODUCTS FINANCE SALES MARKETING OPERATIONS CUSTOMER EXPERIENCE Given the rise in warranty costs, isolate the problem to be a plant and the specific lot. Exclude 2/3rd of the batteries from the lot that are fine. Communicate with affected customers, who have not made a warranty claim, through Marketing and Customer Service channels to recall cars with affected batteries. CUSTOMERS CASE HISTORY SENSOR Access Wide Variety of Data to Answer a Question
  • 71. Copyright Third Nature, Inc. An event contains mainly IDs…that reference other data 2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a 13ae51a6d092 301212631165031 590387153892659 DNIS5555685981- UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s m100109_44IOJ1-Id=105543CD1A7B8322 timestamp user ID device ID game ID event details IP adr log ID Log de-referencing and enrichment is difficult since you can’t enforce integrity like you can in a DB. What’s the glue that holds it together? It’s just keys to other data. e.g. remember that device ID 0 problem?
  • 72. Copyright Third Nature, Inc.© Third Nature Inc. It’s not just the log data, it’s big data + small data Device Event logs Event logs Event logs A product has a complex set of master data and metadata and this needs to be added back to the bare events.
  • 73. Copyright Third Nature, Inc.© Third Nature Inc. Where does the reference data come from? The keys come from somewhere. You don’t just make up system-wide unique identifiers in your code. The lack of local lookup data at event generation leads to development practices that lead to inconsistencies. Event logs Event logs Event logs Problem we had: product identifiers that didn’t match any known products Had to fix by analyzing each log of bad-ID events. Because developers used config files they copied from the PMS and put in the code Break here and you have nothing
  • 74. Copyright Third Nature, Inc. MDM again If you want to link datasets then you must manage the keys You need canonical forms for common data (in code too) user ID IP adr 2010.01.10.01:41:4468 67.189.110.179 Ser40489a07-7f6e4251801a 13ae51a6d092 301212631165031 590387153892659 DNIS5555685981- UTF8&1T4ACGW_enUS386US387&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s m100109_44IOJ1-Id=105543CD1A7B8322 UID CID Email City State Country 590387153892659 10098213 barry.dylan@odin.com Paris Île-de-FranceFrance 2010.01.10.14:26:2468 67.189.110.179 10098213 5046876319474403 MOZILLA/4.0 (COMPATIBLE; TRIDENT/4.0; GTB6; .NET CLR 1.1.4322) https://www.phisherking.com/ gifts/store/LogonForm?mmc=link-src-email_m100109 http://www.google.com/search? sourceid=navclient&aq=0h&oq=Italian&ie=UTF8&pid=1T4ACGW_13ae51a6d092&q=ita lian+rose&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s game ID date IP adr customer ID Event Click Cust- user
  • 75. What you think you have
  • 76. Copyright Third Nature, Inc. What you really have This is called “self-service.” Analysts will love it…
  • 77. Copyright Third Nature, Inc. The solution to our problems isn’t technology, it’s architecture.
  • 78. Copyright Third Nature, Inc. History: This is how BI was done through the 80s First there were files and reporting programs. Application files feed through a data processing pipeline to generate an output file. The file is used by a report formatter for print/screen. Files are largely single-purpose use. Every report is a program written by a developer. Data pipeline code
  • 79. Copyright Third Nature, Inc. History: This is how BI ended the 80s The inevitable situation was... Data pipeline code
  • 80. Copyright Third Nature, Inc. History: This is how we started the 90s Collect data in a database. Queries replaced a LOT of application code because much was just joins. We learned about “dead code” SQL SQL SQL SQL SQL
  • 81. Copyright Third Nature, Inc. Pragmatism and Data Lessons learned during the ad-hoc SQL era of the DW market: When the technology is awkward for the users, the users will stop trying to use it. Even “simple” schemas weren’t enough for anyone other than analysts and their Brio… Led to the evolution of metadata-driven SQL- generating BI tools, ETL tools.
  • 82. Copyright Third Nature, Inc. BI evolved to hiding query generation for end users With more regular schema models, in particular dimensional models that didn’t contain cyclic join paths, it was possible to automate SQL generation via semantic mapping layers. We developed data pipeline building tools (ETL). Query via business terms made BI usable by non-technical people. ETL SQL Life got much easier…for a while
  • 83. Copyright Third Nature, Inc. Today’s model: Lake + self-service, looks familiar… The Lake with data pipelines to files or Hive tables is exactly the same pattern as the COBOL batch.. Dataflow code We already know that people don’t scale…
  • 84. Copyright Third Nature, Inc. DATA ARCHITECTURE We’re so focused on the light switch that we’re not talking about the light
  • 85. Copyright Third Nature, Inc. Decouple the Architecture The core of the data warehouse isn’t the database, it’s the data architecture that the database and tools implement. We need a data architecture that is not limiting: ▪ Deals with change more easily and at scale ▪ Does not enforce requirements and models up front ▪ Does not limit the format or structure of data ▪ Assumes the range of data latencies in and out, from streaming to one-time bulk ▪ Allows both reading and writing of data from outside
  • 86. Copyright Third Nature, Inc. The architecture from 1988 we SHOULD HAVE BEEN USING The general concept of a separate architecture for BI has been around longer, but this paper by Devlin and Murphy is the first formal data warehouse architecture and definition published. 91 “An architecture for a business and information system”, B. A. Devlin, P. T. Murphy, IBM Systems Journal, Vol.27, No. 1, (1988) Slide 91Copyright Third Nature, Inc.
  • 87. Copyright Third Nature, Inc. The goal is to decouple: solve the application and infrastructure problems separately Platform Services Data Access Deliver & Use Data storage This separates BI from other uses of data, allowing each type of use to structure the data specific to its own requirements. Data access is already somewhat separate today. Make the separation of different access methods a formal part of the architecture. Don’t force one model.
  • 88. Copyright Third Nature, Inc. The goal is to decouple: solve the application and infrastructure problems separately Platform Services Data Management Process & Integrate Data storage Data management should not be subject to the constraints of a single use Data management was blended with both data acquisition and structuring data for client tools. It should be an independent function.
  • 89. Copyright Third Nature, Inc. The goal is to decouple: solve the application and infrastructure problems separately Data Acquisition Collect & Store Incremental Batch One-time copy Real time Platform Services Data storage Data arrives in many latencies, from real-time to one-time. Acquisition can’t be limited by the management or consumption layers. Data acquisition should not be directly tied to the needs of consumption. It must operate independently of data use.
  • 90. Copyright Third Nature, Inc. The full analytic environment subsumes all the functions of the data warehouse, and extends them Data Acquisition Collect & Store Incremental Batch One-time copy Real time Platform Services Data Management Process & Integrate Data Access Deliver & Use Data storage The platform has to do more than serve queries; it has to be read-write.
  • 91. Copyright Third Nature, Inc. The data architecture must align with system components because each of them addresses different data needs Incremental Collect Batch One-time copy Real time Manage & Integrate Data Acquisition Collect & Store Data Management Process & Integrate Data Access Deliver & Use Separating concerns is part of the mechanism for change isolation
  • 92. Copyright Third Nature, Inc. Zone 1: Acquistion ▪ Focus is only on collecting and tracking data ▪ Direct recording of data from sources ▪ Each dataset has as much schema as the source provides, could be explicit or implicit or none ▪ Foundation layer for all subsequent use Zone 1 Transactions Events Objects
  • 93. Copyright Third Nature, Inc. Zone 2 Zone 2: Integration - Standardized ▪ Parsed and cataloged – all structure and form are made explicit so data can be accessed ▪ Namespace (e.g. keys) managed ▪ Common elements are standardized ▪ Datasets are profiled, indexed, available Zone 1 Transactions Events Objects
  • 94. Copyright Third Nature, Inc. Zone 2 Zone 2 Zone 2: Integration - Enhanced ▪ Common shared KPIs, master datasets ▪ Extracted and derived data available, e.g. NEE output ▪ Data is linkable, labeled, possibly cleaned Zone 1 Transactions Events Objects
  • 95. Copyright Third Nature, Inc. Layer 1 Layer 2 Layer 0 Zone 3 Zone 3: Access ▪ Data structured to suit the workload ▪ Integrated data for a specific purpose ▪ Business rules applied, e.g. filters, controls, etc. ▪ Designed, explicit structures ▪ Generally repeatable use
  • 96. Copyright Third Nature, Inc. Layer 0 Layer 1 Layer 2 Zone 3 Zone 4 Zone 4: Transient ▪ Under business / developer control ▪ The data for one-off projects of unknown value or repeatability, integrated from other layers ▪ Place for ephemeral analytics output ▪ The sandbox
  • 97. Copyright Third Nature, Inc. Zone 1 Raw Zone 2 Standardized Zone 2 Enhanced Zone 3 Managed Zone 4 Transient Decoupled data architecture
  • 98. Copyright Third Nature, Inc. Food supply chain: an analogy for analytic data Multiple contexts of use, differing quality levels You need to keep the original because just like baking, you can’t unmake dough once it’s mixed.
  • 99. Copyright Third Nature, Inc. The design focus is different in each area Ingredients Goal: available User needs a recipe in order to make use of the data. Pre-mixed Goal: discoverable and integrateable User needs a menu to choose from the data available Meals Goal: usable User needs utensils but is given a finished meal
  • 100. Copyright Third Nature, Inc. The data is in zones of management, not isolating layers Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data Relax control to enable self-service while avoiding a mess. Do not constrain access to one zone or to a single tool. Focus on visibility of data use, not control of data.
  • 101. Copyright Third Nature, Inc. This data architecture resolves rate of change problems Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data New data of unknown value, simple requests for new data can land here first, with little work by IT. More effort applied to management, slower. Optimized for specific uses / workloads. Generally the slowest change. Not fast vs slow: fast vs right Not flexibility vs control: flexibility vs repeatability Agile for structure change vs agile for questions / use
  • 102. Copyright Third Nature, Inc. SQLServer RDBMS Example: data environment, mid-size retailer Replication, batch ETL to collect data Web Application Standard retail / wholesale data warehouse model. Nightly batch model from stage, contains ~ 1 TB of data DW New requirement: Load all the online marketing and web activity for use in analytic models (not simple web analytics) Will add ~10 TB of data, continuous or hourly loads
  • 103. Copyright Third Nature, Inc. SQLServer RDBMS Example: data environment, mid-size retailer Replication, batch ETL to collect data Hadoop Web Application Persistent store, data warehouse on same DB in different schemas DW Log file fetch & load for clickstream, summaries sent to reporting env Discovery work usually done in Hadoop, sometimes in database.
  • 104. Copyright Third Nature, Inc. SQLServer This data architecture uses the 3 zone pattern Hadoop Web Application RDBMS DW Raw data in an immutable storage area Standardized or enhanced data Common or usage- specific data Transient data All reporting data and analytics output is sent to the DW where it is can be accessed. Cleaned, summarized or derived data is stored in DB Raw data is stored in two places: relational and small sets in DB, log collection in Hadoop
  • 105. Copyright Third Nature, Inc. The concept of a zone is not a physical system. It’s data architecture Apps DW The biggest decision is to separate all data collection from the data integration from consumption. Physical system/technology overlays are separate, depend on the specific use cases and needs of the organization. Dist FS RDBMSs Decompress & process device logs RDBMS RDBMS RDBMS Discovery platform
  • 106. Copyright Third Nature, Inc. What about the technology? Do I need an <X>?
  • 107. 112 Blended Architectures Are a Requirement, Not an Option Data Warehouse + Data Lake On Premise + Cloud RDBMS + S3 + HDFS Commercial + Open Source
  • 108. Copyright Third Nature, Inc. Bricks are not buildings We don’t think this is equivalent to this Architecture is not technology. It’s not a product you can buy.
  • 109. Copyright Third Nature, Inc.Copyright Third Nature, Inc. New tools and tool architectures are required One tool for all jobs doesn’t work any more* * it never did, we just had fewer jobs
  • 110. Copyright Third Nature, Inc. Beware overengineering: do you really need special new technology infrastructure? Numberofpeople The distribution of data size is about normal, yet these guys set the tone of the market today. Bigness of data
  • 111. Copyright Third Nature, Inc. Analytics: This is really raw data under storageNumberofjobs Microsoft study of 174,000 analytic jobs in their petabyte scale cluster: median size = ??? Bigness of data Copyright Third Nature, Inc.
  • 112. Copyright Third Nature, Inc. Working data for analytics most often not bigNumberofjobs 14 GB Smallness of data Copyright Third Nature, Inc. The usable size distribution
  • 113. Copyright Third Nature, Inc. TANSTAAFL When replacing the old with the new (or ignoring the new over the old) you always make tradeoffs, and usually you won’t see them for a long time. Technologies are not perfect replacements for one another. Often not better, only different.
  • 114. © Third Nature Inc.© Third Nature Inc. 1.It’s not about storing data, it’s about using it 2.Use drives architecture. Understand the uses, what you are designing for, to drive decisions. 3.Put data at the center, not technology. Don’t let the tech define what you can do or how you do it. 4.The death star is not the answer. The data model is not a flat earth. You are not building a monolith. 5.Know your history. Avoiding wheel reinvention saves time, money, careers. Conclusion
  • 115. © Third Nature Inc. Manage your data (or it will manage you) Data management is where developers are weakest. Modern engineering practices are where data management is weakest. You need to bridge these groups and practices in the organization if you want to do meaningful work with event stream data.
  • 116. © Third Nature Inc. “Now is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” Winston Churchill
  • 117. Mark Madsen is the global head of architecture at Teradata, Prior to that he was president of Third Nature, a research and consulting firm focused on analytics, data integration and data management. Mark is an award- winning author, architect and CTO whose work has been featured in numerous industry publications. Over the past ten years Mark received awards for his work from the American Productivity & Quality Center, TDWI, and the Smithsonian Institute. He is an international speaker, chairs several conferences, and is on the O’Reilly Strata program committee. For more information or to contact Mark, follow @markmadsen on Twitter or visit http://ThirdNature.net About the Presenter
  • 118. Todd Walter Chief Technologist - Teradata • Chief Technologist for Teradata • A pragmatic visionary, Walter helps business leaders, analysts and technologists better understand all of the astonishing possibilities of big data and analytics • Works with organizations of all sizes and levels of experience at the leading edge of adopting big data, data warehouse and analytics technologies • With Teradata for more than 30 years and served for more than 10 years as CTO of Teradata Labs, contributing significantly to Teradata’s unique design features and functionality • Holds more than a dozen Teradata patents and is a Teradata Fellow in recognition of his long record of technical innovation and contribution to the company