The talk consists of 4 parts: introduction in today’s data quality monitoring methods (hard-coded SQL rules), explanation of autoencoders and their use in making error detection more generalized and efficient, a small demonstration of the way autoencoders work on practical examples, and our findings and further enhancement approaches. The demo is made in R Shiny.

1. Automated Data Quality
Assurance
with Machine Learning
Milica Petrović
InCube Group

2. 2
Talk Outline
2
What’s Wrong with Data Quality?
3
Error Detection using ML
Demo & Features
1
4 Error Remediation using RPA

3. 3
2
What’s Wrong with Data Quality?
3
Error Detection using ML
Demo & Features
1
4 Error Remediation using RPA

4. 4
Data Quality Today
What’s wrong with it?

5. 5
Data Quality Today
Our Take at a Solution
Data Quality Today
 Manually coded SQL
rules
 Uni-/bi-variate checks
 Too much data
 Too many error types
 Too narrow focus
 Too late
 Automate: simultaneous error detection & faster process
 Reusability: tailored ML algorithms reused for fields of similar type
 Deep dive: discovery of new error types based on multivariate dependencies
 Unsupervised
 Capture multivariate
relationships
Challenges
Solutions with Machine Learning
Autoencoders

6. 6
2
What’s Wrong with Data Quality?
3
Error Detection using ML
Demo & Features
1
4 Error Remediation using RPA

7. 7
Autoencoders for Data Quality
Use and Architecture
Target: Reconstruct input
Bottleneck: Ensures network learns structure of input data
For good data only
INPUTINPUT OUTPUT
Differentiate between
good and bad data
Reconstruction: Good reconstruction only for data with structure similar to
training data

8. 8
Autoencoders for Data Quality
Assumptions and Training
Imperfect data: Training requires large share of good data
Limited potency of the network: More layers not always better
INPUT OUTPUT INPUT
For good data only

9. 9
Discriminating Good and Bad Data Records
Clustering the Reconstruction Errors
MeanSquaredError
Individual Data Records
Challenge: Where to draw the separating line?
Kernel Density Estimate

10. 10
Discriminating Good and Bad Data Records
Clustering the Reconstruction Errors
MeanSquaredError
Individual Data Records
Challenge: Where to draw the separating line?
Kernel Density Estimate

11. 11
Discriminating Good and Bad Data Records
Sequence of Autoencoders
1st iteration
Keep Rest of Data
Challenge: Magnitude of reconstruction error varies across
data error types
Remove Detected
Anomalies

12. 12
Discriminating Good and Bad Data Records
Sequence of Autoencoders

13. 13
Discriminating Good and Bad Data Records
Sequence of Autoencoders
Across iterations: Increase model complexity
Stopping: When threshold separates large chunk of data

14. 14
2
What’s Wrong with Data Quality?
3
Error Detection using ML
Demo & Features
1
4 Error Remediation using RPA

15. 15
Video Demo
Birth date

16. 16
Video Demo
First name

17. 17
Video Demo
Revenue

18. 18
variational
autoencoder
complete
autoencoder with
regularization
undercomplete
autoencoder with
custom loss
one-hot
encoding
character
fields
categorical
fields
numerical
feature
extraction date
fields
normalization
numerical
fields
Reusability of Pre-Processing and Model Setup
Setup per feature type
Pre-processing
Models

19. 19
2
What’s Wrong with Data Quality?
3
Error Detection using ML
Demo & Features
1
4 Error Remediation using RPA

20. 20
Identifying Where DQ Exception Occurred
Searching for Correct Data using the ID
Customer Number Name Street Number City Zip Code
65465456 Jane Doe Riverstreet 32 Zurich 8004
12564321 John Doe Central Town i Zurich 8001
56543651 Peter Smith Quiet Street 2 Zurich 8045
36364448 Lisa Miller Busy Street 3 Zurich 8022
AUTOENCODER
Example: CRM data at a bank

21. 21
Account Opening Form
Fictitious
Bank of
Zurich
Sender:
Customer Number: 12564321
John Doe (leave empty, filled by the bank)
Central Town 1
8001 Zurich Application Date: 1.9.2019
Opening Date: 1.1.2020
Finding Correct Data via “Intelligent” Robots
Computer Vision or NLP Help Find and Read Correct Data
Customer Number Name Street Number City Zip Code
65465456 Jane Doe Riverstreet 32 Zurich 8004
12564321 John Doe Central Town i Zurich 8001
56543651 Peter Smith Quiet Street 2 Zurich 8045
36364448 Lisa Miller Busy Street 3 Zurich 8022

22. 22
Remediation Proposal
RPA Robot Proposes DQ Exception Correction
Customer Number Name Street Number City Zip Code
65465456 Jane Doe Riverstreet 32 Zurich 8004
12564321 John Doe Central Town i Zurich 8001
56543651 Peter Smith Quiet Street 2 Zurich 8045
36364448 Lisa Miller Busy Street 3 Zurich 8022
Customer Number Name Street Number City Zip Code
65465456 Jane Doe Riverstreet 32 Zurich 8004
12564321 John Doe Central Town 1 Zurich 8001
56543651 Peter Smith Quiet Street 2 Zurich 8045
36364448 Lisa Miller Busy Street 3 Zurich 8022
SUGGESTS THE CHANGE
BASED ON

23. 23
Error Detection
via ML
Client Data
Repository
Extract DQ
exceptions using
autoencoders
+
RPA robot finds correct data using
NLP and computer vision, and
proposes remediation of DQ
exceptions
Cognitive
Services
SME reviews, and
approves or rejects
DQ resolutions are used to improve error detection
and correction
RPA robot
corrects
DQ exceptions
Corrected data
flows back into repository
Automation of Data Quality Remediation
Process Overview

24. 24
Key Findings from Projects and Experience
High reusability: Only one-time customization effort per data type
Extension: ML can replicate rule-based DQ checks and find new errors
High scalability: Operations can be performed in parallel
Multivariate relationships: Detection of interdependencies
Unsupervised learning! Training data quality matters
Competency: Subject Matter Experts can focus on value-adding tasks
AutoencodersRPA
Cost savings: Automate interactions with existing IT infrastructure