Analytics has been maturing for more than half a century, but in the last decade there's been a real revolution in how businesses are using data to drive business strategy. What is the history? Why is the current age of analytics different? What role is technology playing in the revolution? How can you engage in the revolution?!
This is a talk I presented to the business analytics program at the University of Central Missouri in Warrensburg, MO on October 18, 2018. https://www.ucmo.edu/academics/programs/majors/big-data-and-business-analytics-bsba-46-640/index.php
2. Innovation andValue
1. The purpose of analytics is to enable
agility in a way that creates value for
the business.
10/19/2018 2
3. Analytics as a Partnership
1. The purpose of analytics is to enable
agility in a way that creates value for
the business.
2. Analytics can only create value when it
is done in a cooperative partnership
that is intermittently lead by business
leaders and then technology leaders.
10/19/2018 3
4. Constancy of Change
1. The purpose of analytics is to enable
agility in a way that creates value for
the business.
2. Analytics can only create value when it
is done in a cooperative partnership
that is intermittently lead by business
leaders and then technology leaders.
3. The rate of change in business is
accelerating, accelerates the demand
for analytics, and is accelerated by
analytics.
10/19/2018 4
6. Early Analytics Concepts
10/19/2018 6
“Consider a future device … in which
an individual stores all his books,
records, and communications, and
which is mechanized so that it may be
consulted with exceeding speed and
flexibility. It is an enlarged intimate
supplement to his memory.”
Vannevar Bush 1945
1960’s Researchers at MIT, Carnegie Mellon, Harvard,
Dartmouth, and others begin work researching the
idea of computerized models to assist in decision
making.
1970’s
1980’s
1990’s
Business journals first begin reporting on the idea
of computerized decision support
systems. ACNeilson and IRI develop fact /
dimension model for retail sales.
Berry Devlin and Paul Murphy coin the term “data
warehouse.” Teradata releases first database
designed specifically for decision support.
Ralph Kimball and Bill Inmon publish seminal
books and articles on data warehousing and
business intelligence systems.
8. Business-Driven
• Business users identified the opportunity and need
• Business users defined the requirements
• Business users set the priority and the timeline
• Technologists established standards and repeatable patterns
10/19/2018 8
ETLDW Rpt XLS P
9. • Sophisticated business
users spend more time
making decisions and less
time crunching numbers.
• Technologists develop
business patterns for
building systems faster
and more reliably.
10/19/2018 9
• Business users set the
direction, specify the
requirements, and verify
the work is done correctly.
• Technologists build
systems.
• As soon as a developer is
done delivering one
product, the requirements
are sure to change.
• Users always have new
questions.
Innovation and
Value
Analytics as
Partnership
Constancy of
Change
$
10. Pushback During the
TeenageYears
10/19/2018 10
• Self-Service Reporting
• Master Data Management
• Data Stewardship
• Data Quality Management
• Data Governance
• Self-Service Data Prep
13. Making Life Easier forTechnologists
10/19/2018 13
1990 Dimensional modeling
Low-code ETL tools
Self-service reporting tools
Data warehouse appliances
On-line analytical processing
Wal-Mart andTeradata create the
world’s largest data warehouse at
24TB
1999
15. • Self-Service Reporting
enables easier drag-and-
drop reporting.
• Master Data Management
allows users to edit
business configuration
data.
• Data Governance
becomes a business
function.
10/19/2018 15
• Technology deploys more
sophisticated and user-
friendly tools.
• Business users begin
focusing more energy on
looking at information
rather than waiting on
developers.
• Vendors continuously leap
frog each other, leading
users to want to
continuously switch.
• Everyone builds their own
version of each report.
Innovation and
Value
Analytics as
Partnership
Constancy of
Change
$
16. Data and Analytics Maturity Model
10/19/2018 16
PRENATAL INFANT CHILD TEENAGE ADULT SAGE
Financial Systems Executive
Systems
Analytical
Systems
Monitoring
System
Strategic
System
Business Service
Static Reports Spreadsheets OLAP/Ad Hoc
Reports
Dashboards Scorecards /
Analytics
Customer /
Embedded
“Cost
Center”
“Inform Executives” “EmpowerWorkers” “Monitor
Performance”
“Drive the Business” “Drive the
Market”
18. Data and Analytics Methodology
10/19/2018 18
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Requirements Analysis
Data Warehouse
Data Profiling
Data Modeling
Data Mapping
ETL Development
ETL Testing
Workflow Scheduling
System Testing
Information Delivery
Report Design
Report Development
Report Validation
User Acceptance
Testing
Report Deployment
19. • Emergence of the Chief
Data Officer and Chief
Analytics Officer who
drive best practices and
efficiencies across an
organization.
• Ongoing refinement and
automation of data
warehousing and business
intelligence design
patterns.
10/19/2018 19
• Organizations create an
Analytics Center of
Excellence and shared
services models to provide
optimized delivery of
analytics.
• Relationships between
requestors and providers
of analytics become
formalized with service
level agreements.
• Incremental improvement
and planned change are
the standard practice.
• Analytics roadmaps have
three to five year time
horizons.
Innovation and
Value
Analytics as
Partnership
Constancy of
Change
$
21. Who’s On
First?
Business requests
Technology to do
what Technology is
enabling Business to
do so that Technology
doesn’t have to do
what Business wants
Technology to do at
the request of
Business.
10/19/2018 21
23. Time for a Revolution!
10/19/2018 23
2000’s Big Data
DataVisualization
Machine Learning &
Artificial Intelligence
2010’s Agile & DevOps
“The Cloud”
24. What is Big Data?
10/19/2018 24
New patient data in
the US would create
a stack of paper
1,000 miles high
every year.
1,000 miles
250 miles
International
Space Station
340 miles
HubbleTelescope
35,000 ft (6.6 miles)
CommercialAirplane
33. Traditional Programming vs Machine Learning
• In traditional programming, we write a program that takes input
data, applies a set of instructions, and produces output data.
10/19/2018 33
COMP
01001110
01101101
34. Traditional Programming vs Machine Learning
• In machine learning, we provide a computer with sample inputs and
outputs, and the computer produces a reusable program.
10/19/2018 34
ML
01001110
01101101
COMP
38. What does Agile look like?
10/19/2018 38
• Deployment plan
• Database server provisioning
• Database software installation
• Database configuration
• Database object creation
• BI server provisioning
• BI server software installation
• BI desktop software installation
• Setup BI database connections
• Integration server provisioning
• Integration software installation
• Integration software configuration
• Setup integration database connection
• Schedule data integration jobs
• Schedule BI reports
• Setup ad hoc reporting
> git clone data-analytics
> kubectl run data-analytics
Traditional Setup Steps Using Agile / DevOps Tools
40. …As A Service
• Infrastructure as a Service
• Database as a Service
• Platform as a Service
• Identity as a Service
• Enterprise Integration as a Service
• Machine Learning as a Service
• Internet ofThings as a Service
• Artificial Intelligence as a Service
• Labor as a Service
10/19/2018 40
42. Innovation andValue
1. The purpose of analytics is to enable agility in
a way that creates value for the business.
In order to create sustainable business agility, the
the processes and technology behind delivering analytics
delivering analytics must be even more agile.
Technology is cool and should be used to deliver
innovative solutions to business objectives.
• Machine learning and artificial intelligence
• Business-centric analytics
• Decomposition techniques
10/19/2018 42
Understand
43. Analytics as a Partnership
2. Analytics can only create value when it is
done in a cooperative partnership that is
intermittently lead by business leaders and
then technology leaders.
Sometimes business needs will drive technical choices
choices and at other times, the technology will open the
open the business to new insights, business models, and
models, and create entirely new markets.
Imagine what you can accomplish working together
and be open to collaboration!
• The Uber model
• Precision Medicine
• Smart Everything
10/19/2018 43
Embrace
Understand
44. Constancy of Change
3. The rate of change in business is accelerating,
accelerates the demand for analytics, and is
accelerated by analytics.
Consumers have come to expect change. Smarter
Smarter businesses adapt and create new competitive
competitive forces faster. Markets are being disrupted
disrupted regularly.
You need to embrace and be able to drive change!
• Agile software development
• DevOps and DataOps
• Cloud deployment and containers
• Flexible data and solutions architectures
10/19/2018 44
Embrace
Accept
Understand
For a very long time, even before human “computers” and tabulating machines we had the notion that the work of business decision-making could be supported by offloading time intensive work of collating data and tabulating numbers to non-decision-making individuals.
The origin of the analytics movement was the idea that businesses could fundamentally improve and accelerate the rate of improvement by learning from its past decisions and past performance. Business leaders had always been able to do this, and they were able to pass that down to subsequent generations through mentorship of family members and apprenticeships. What was new was the idea that a business could institutionalize and systematize this process of improvement. From this basic notion, we’ve had all kinds of great business theories emerge.
For many of these early years, the use of technology was mostly about the automation of business computations that were needed to compile data and calculate business metrics. This allowed decision makers to react to new information more quickly that competitors, to see trends more precisely, to understand complexity more easily. We weren’t making a significant transformational change in the kinds of analysis being done, though – just the difficulty and speed with which the analysis could be done.
The need, the insights, and the direction were primarily business driven.
As teenagers tend to do, the adolescent phase in the history of analytics was one of push back to business users. Data warehouse and BI developers got tired of doing the same old work over and over again and decided that they should build tools to let more businessy people write their own data integration jobs and reports. In came the era of self-service.
Data Governance
Master Data Management
Data Quality Management
Reference Data Management
By the way – this is what 24 TB can look like today.
As data analytics started to mature, the industry developed standard processes and maturity models that helped everyone feel good about what they were doing, even if they didn’t measure and see the business impact.
TODO: Update
With all of the talk about “maturity” you’d think we hit some kind of Zen.
We weren’t really achieving some kind of Zen, though.
The pile of business challenges was increasing and we were building ever larger and more complex technologies and processes to try to deal with those big challenges.
Creve Maples, a pioneer in advanced data visualization, talked in 2011 about work that he’d done with Goodyear Tire Company.
The engineers at Goodyear had been working for two years to figure out how to make their tires go faster without failing under the stress.
Dr. Maples provided a sophisticated and immersive environment to explore the data and within five minutes, the engineers literally “saw” the answer.
Every NBA game has six high definition cameras recording the game at 25 frames per second.
Each game produces about 54 GB per game.
Over a season, that’s more than 66 TB of data for the games… not to mention the tapes of practices.
NFL has chips inside shoulder pads that track their position, speed, and direction continuously during games.
They use that for tracking players but they’re also using that to train robotic tackle dummies that can be used to help reduce damage from repetitive brain injuries.
Now players are starting spend several hours training while sitting at a desk with VR goggles on. Running plays over and over to train their brain to see what the play should look like, which enhances their performance in learning the play physically and then executing it in games.
Amazon’s hiring algorithm has gender biases just like we do.
When I started, we were all reading “The Mythical Man Month” as a lesson about project management, estimation, and how long things really take to get done.
Now, the personal marketplace, Etsy, deploys new updates to it’s website thousands of times per day.
In my personal experience, the process of setting up a full BI stack of technology could take multiple days - for a test environment let alone a complex production environment.
Database – we had to install the database and set all of the company standard settings; then we had to change all of those settings to fit the BI vendor recommendations. Then we’d have to tweak them for performance as we learned more.
ETL Tools – usually at least two layers of servers plus the desktop development and administration tools plus the database configuration on all the developer desktops as well as on the servers. And make sure you have right version of the database drivers – sometimes both 32-bit and 64-bit.
BI Tools – also usually multiple servers. In fact, this was somewhere that we almost always have a cluster of servers. So, they have to be able to talk to each other as well to the databases and to the user web browser and all the desktop development tools.
Infrastructure as a service and virtualization… but the cloud is way more than that!
Understand: Modern analytics is about innovation and business value
Embrace: Business / Technology partnerships
Accept: Change used to happen in big steps, occasionally. Not it happens continuously and incrementally.
Agile
Dev Ops
Dimensional Modeling
Data Visualization
Statistics
Machine Learning
Business
Artificial Intelligence