Synthesis Webcast with Eric Kavanagh and Tamr DataOps is an emerging set of practices, processes, and technologies for building and automating data pipelines to meet business needs quickly. As these pipelines become more complex and development teams grow in size, organizations need better collaboration and development processes to govern the flow of data and code from one step of the data lifecycle to the next – from data ingestion and transformation to analysis and reporting. DataOps is not something that can be implemented all at once or in a short period of time. DataOps is a journey that requires a cultural shift. DataOps teams continuously search for new ways to cut waste, streamline steps, automate processes, increase output, and get it right the first time. The goal is to increase agility and cycle times, while reducing data defects, giving developers and business users greater confidence in data analytic output. This webcast examines how organizations adopt DataOps practices in the field. It will review results of an Eckerson Group survey that sheds light on the rate and scope of DataOps adoption. It will also describe case studies of organizations that have successfully implemented DataOps practices, the challenges they have encountered and benefits they’ve received. Tune into our webcast to learn: - User perceptions of DataOps - The rate of DataOps adoption by industry and other demographic variables - DataOps adoption by technique and component (i.e., agile, test automation, orchestration, continuous development/continuous integration) - Key challenges organizations face with DataOps - Key benefits organizations experience with DataOps - Best practices in doing DataOps - Case studies and anecdotes of DataOps at companies

1. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Best Practices in DataOps
How to Create Agile, Automated Data Pipelines
Wayne W. Eckerson
May 8, 2019

2. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
1. Your data team is flooded with minor request tickets and is burning out.
2. Business users don’t trust the data because it contains too many errors.
3. Source system changes keep breaking your ETL jobs and data pipelines.
4. Business users don’t understand why it takes so long to get data.
5. You have difficulty meeting service level agreements (SLAs).
6. Data analysts write the same jobs and reports with minor variations.
7. Data scientists wait for months for data and computing resources
8. Your company can’t discern the true cost of migrating to the cloud
9. Your data environment is too chaotic to implement predictive analytics
10. Your self-service initiative has spawned hundreds of data silos.
Bonus: Your data lake is more of a data swamp.
Bonus: It’s takes months to deploy a single predictive model.
10 Symptoms You Need DataOps

3. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
What is DataOps?
LeanTQM
Agile Dev/Ops
• Scrum, Kanban
• Business engagement
• Self-organizing teams
• Retrospectives
• Automation
• Orchestration
• Efficiency
• Simplicity
• Team-based development
• Version control
• Continuous integration/ delivery
• Test-driven development
• Performance management
• Performance metrics
• Continuous monitoring
• Benchmarking
DataOps
“A set of practices, processes, and technologies for building, operationalizing,
automating, and managing data pipelines from source to consumption.”
DataOps = Data Operations

4. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
DataOps History
DataOps applies rigor of software engineering to the
development and execution of data pipelines.
“Cowboy
Coders”
Team-based
Development
DevOps-based
Development
1960s 1970s 1980s 1990s 2000s 2010s 2020s
First DevOps
event (2009)
Manifesto for Agile Software
Development published (2001)
DevOps DataOps
KEY:
DataOps Manifesto
published (2017)
First DataOps
Event (2019)

5. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Primary Use Cases
Big Data Data Science Self Service
Data
Warehousing
Standardize and
reuse core data
pipeline components:
ingest, transform,
clean, etc.
Create data science
sandboxes on
demand; deploy
models automatically;
monitor data drift.
Centralize logic and
permissions to
facilitate data access
and analysis while
eliminating data silos
Speed development
by assigning agile
teams to business
groups to build end-
to-end solutions
Agile but ungoverned Governed but not agile
Reuse and
Collaboration
Self Service and
Automation
Governance and
Infrastructure
Speed and
Prioritization
Biggest
Needs

6. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Adoption
Yes
27%
Somewhat
30%
No
43%
DOES YOUR ORGANIZATION HAVE A
DATAOPS INITIATIVE?
Based on 175 respondents from an Eckerson Group
survey conducted in April, 2019.
32%
29%
10%
9%
7%
6%
5%
2%
1%
0%
IT OR BI DIRE C T OR OR MA NA GE R
IT OR BI A RC H IT E C T ,…
C ONS ULT A NT
BUS INE S S MA NA GE R - A NA LY T IC S
DA T A A NA LY S T OR S C IE NT IS T
BUS INE S S E X E C UT IVE OR …
DA T A E NGINE E R
A C A DE MIC
VE NDOR
DA T A OP S E NGINE E R
RESPONDENT ROLES
18%
15%
11%
29%
26%
VE RY S MA LL < 100 …
S MA LL <500 …
ME DIUM <1, 000 E MP …
LA RGE <10, 000
VE RY LA RGE > …
COMPANY SIZE

7. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Benefits
Faster cycle time
Fewer data defects
More scalability, reliability
Lower costs
More innovation
Happier customers
Continuous integration/delivery, reuse, automation
Test-driven development and execution
Team-based development, continuous monitoring
Higher development capacity, fewer errors
Focus efforts on value-add solutions and technologies
Get more for less with greater trust and alignment

8. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Benefits from Survey
60%
55%
50%
50%
48%
47%
47%
42%
FA S TE R CY CLE TI ME S
HA P P I E R B US I NE S S US E RS
DE LI V E R NE W A P P LI CA TI ONS MORE QUI CK LY
FE W E R DE FE CTS A ND E RRORS
I NGE S T NE W DA TA S OURCE S MORE RA P I DLY
FASTER CHANGE REQUESTS
I NCRE A S E D DE V E LOP ME NT CA P A CI TY
I MP ROV E D DA TA GOV E RNA NCE
BENEFITS OF DATAOPS

9. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Challenges
55%
53%
50%
50%
47%
42%
35%
34%
26%
23%
E S TA B LI S HING FORMA L P ROCE S S E S
ORCHE S TRA TI NG CODE A ND DA TA A CROS S TOOLS
STAFF CAPACI TY
MONI TORI NG THE E ND -T O-E ND E NV I RONME NT
BUI LDI NG RI GOROUS TESTS UPFRONT
LA CK OF A DE QUA TE A UTOMA TI ON TOOLS
GE TTI NG B US I NE S S US E RS TO B UY I NTO THE
P ROCE S S
A DOP TI NG A GI LE ME THODS A ND TE A MS
DA TA I S TOO HA RD TO FI ND
GETTI NG TECHNI CAL USERS TO BUY I N TO THE
P ROCE S S
DATAOPS CHALLENGES

10. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Components and Tools
58%
54%
53%
50%
50%
46%
46%
41%
38%
32%
28%
A GI LE DE V E LOP ME NT
CONTI NUOUS DE LI V E RY
COLLA B ORA TI ON A ND RE US E
CONTI NUOUS I NTE GRA TI ON
CODE RE P OS I TORY
DATA PI PELI NE ORCHESTRATI ON
P E RFORMA NCE A ND A P P LI CA TI ON MONI TORI NG
CONTI NUOUS TE S TI NG
W ORKFLOW MA NA GE ME NT
CHA NGE MA NA GE ME NT RE QUE S T
CONTA I NE RS A ND ORCHE S TRA TI ON TOOLS
RATE THE IMPORTANCE OF EACH DATAOPS
COMPONENT?
High

11. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Use Cases
66%
60%
56%
52%
39%
29%
34%
27%
DA T A W A RE H OUS E S A ND MA RT S
RE P ORT ING A ND DA S H BOA RDING
S E LF - S E RVIC E A NA LY S IS
DA T A S C IE NC E A ND MA C H INE LE A RNING
DA T A LA K E
OLA P C UBE S F OR RE P ORT ING A ND A NA LY S IS
C US T OME R- F A CING A P P LIC A T IONS
A UDIT , C OMP LIA NC E , S E C URIT Y
DATAOPS USE CASES

12. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Best Practices
Form a data department (with a CDO)
Map and assess your data environment
Educate your team about DataOps
Create cross-functional dev teams
Align the teams with business priorities
Continuously review and refine processes
The “Soft
Stuff”
If you don’t have one already Add a CDO for executive clout
Map data flows; assess waste, inefficiencies,
manual processes, error sources, dev capacity.
Expect resistance: ”Data is different!” “Don’t
slow us down!”
Stick with it; “You can’t drive fast w/o brakes.”
Self-organizing, cross-trained; collaborative, agile
teams that build end-to-end solutions
Align agile themes, initiatives, epics, and stories
with business goals; get cross-functional priorities
It’s a journey; benchmark performance and
continuously improve cycle times, capacity,
reuse, and other core objectives.
Pull ”data people” out of IT; unite data engineers,
data scientists, and SW engineers.

13. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Best Practices (cont)
Start small and build incrementally
Build for reuse
Segregate duties and environments
Test and monitor everything
Use DevOps and DataOps tools
Create a self-service infrastructure
Build for the enterprise
The “Hard
Stuff”
Standardize ingest, transforms, configurations,
code, data sets; use repositories & containers.
Use tools to migrate code from dev to test, to
production environments and segregate duties
Build tests before and after coding; use tests to
monitor and automate data pipelines.
Repositories for data, code, configurations; tools
for agile collaboration, CI/CD, testing, data
catalog, orchestration, data glossary, unification.
Centralize logic; apply permissions for data
access and functionality; automate report and
model deployment; serverless, Kubernetes,
Plan for security, governance, auditability,
scalability, reliability, portability, and continuous
monitoring.
Insist on business representation on the dev
team; get cross-functional priorities monthly

14. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Summary
DataOps puts
your data on a
solid foundation
• Speeds cycle time,
improves quality,
increases capacity,
reduces cost
Lets your data
team focus on
value-add
• Such as
predictive
analytics,
streaming data,
cloud computing
Increasing
customer
satisfaction and
business value
DataOps is light—
out, automated
data operations.

15. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Questions?
I’m listening!

16. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Wayne Eckerson
• 25+ year thought leader in data and analytics
• Sought-after speaker and consultant
• President, Eckerson Group
• Former director of research at TDWI
• Author of hundreds of articles and reports
Performance
Management
BI/Analytics

17. © Eckerson Group 2019 Twitter: @weckerson www.eckerson.com
Get More Value from
Data and Analytics