1.
CI/CD for Data –
Building Dev/Test Data
Environments With lakeFS
Vino SD, Developer Advocate

2.
About Me
■ I am Vino Duraisamy.
■ SWE -> Data/ML Engineer -> Developer Advocate
■ Open-source contributor @lakeFS
@vinodhini_sd @vinodhini-sd
vinodhini-sd.medium.com

3.
A Day in the Life of a Data Engineer!
Expectation Reality

4.
What Can Go Wrong?
■ EMR/Spark upgrade
■ Schema changes
■ Change in business logic
■ Infrastructure changes
■ Troubleshooting failed spark jobs
■ Backfilling historic data
■ Non-idempotent pipelines
■ Maintaining legacy DAGs/pipelines

5.
■ Unit Testing
■ Integration Testing
■ E2E Testing
■ Realistic Input Data
■ Mock data
■ Sampled production data
■ Copy all of Production data
Effective Testing Strategy

6.
CI/CD for Data–
Apply engineering best
practices to data
engineering

7.
Build
■ Place all data
assets under data
version control
Test
■ Create isolated
data environments
on-demand
■ Automate all your
tests
Deploy
■ Data quality checks
passed
■ Automatic rollbacks
are in place
CI/CD Best Practices

8.
Build
■ Place all data
assets under data
version control
Test (CI)
■ Create isolated
data environments
on-demand
■ Automate all your
tests
Deploy
■ Data quality checks
passed
■ Automatic rollbacks
are in place
CI/CD Best Practices

9.
Build
■ Place all data
assets under data
version control
Test (CI)
■ Create isolated
data environments
on-demand
■ Automate all your
tests
Deploy (CD)
■ Data quality checks
passed
■ Automatic rollbacks
are in place
CI/CD Best Practices

10.
Build – Version
Control

11.
Data Repo
Branches of the
whole repository
Data Versioning at
the object level
Place All Data Assets Under Version
Control

12.
Git for Data with lakeFS
s3://data-repo/collections/foo
lakefs://data-repo/main/collections/foo
lakectl branch create
lakefs://repo/experiment-1
--source lakefs://repo/main
# output:
# created branch 'experiment-1',
# pointing to commit ID: 'd1e9adc71c10a’

13.
How lakeFS Works

14.
Test - CI

15.
Ingestion
Experimentation
Safe CI: Create Isolated Data Environments
On-Demand

16.
Automated Tests Using lakeFS Hooks

17.
Demo Time

18.
Deploy - CD

19.
■ Conceptually similar to Github actions
■ Hooks run as a standalone webserver - performs tests when triggered
■ Run custom tests or use test suites like Great Expectations, etc.,
Automating Data Quality Checks Using
lakeFS Hooks

20.
■ post_merge event: Failed
Hook runs can
programmatically revert
the changes to a branch.
■ Reliability of data in Prod.
$lakectl revert main^1
CD: Automatic Rollbacks in Place

21.
Easily revert
corrupted
data.
Time travel
to debug a
snapshot
Safely test your
pipelines on an
isolated branch
from production
Production
identical
isolated
branch and
hooks to test
your data
Data Lakes with lakeFS

22.
.io

23.
Thank You