Practical Anomaly Detection Examples with BigML, by Guillem Vidal, Machine Learning Engineer at BigML. *MLSEV 2020: Virtual Conference.

1. 2nd edition

2. #MLSEV 2
Anomaly Detectors
Practical Examples with BigML
Guillem Vidal
Machine Learning Engineer, BigML

3. #MLSEV 3
Outline
2 Demo 1: Removing Outliers
3 Demo 2: Fraud Detection
4 Demo 3: Novel Categories Discovery
1 Anomaly Detection Recap

4. #MLSEV 4
Anomaly Detection Recap

5. #MLSEV 5
Anomaly Detection
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
An unsupervised algorithm that looks for unusual instances in a dataset. Anomaly
detectors provide an anomaly score to each instance, the higher is the score the
most unusual is the instance. Example:
• Amount $2,459 is higher than all other transactions
• Only transaction
• In zip 21350
• For the purchase class “tech"

6. #MLSEV 6
Graphical Example
Which object appears more unusual within this group?

7. #MLSEV
“Round”“Skinny” “Corners”
“Skinny”
but not “smooth”
No
“Corners”
Not
“Round”
Most unusual
7
Graphical Example

8. #MLSEV 8
Isolation Forest
“easy” to isolate
“hard” to isolate
Depth
Now repeat the process several
times and use average depth to
compute anomaly score:
0 (similar) 1 (dissimilar)
Isolation Forest: Grow random
decision trees until each instance is
in its own leaf. Random features
and splits

9. #MLSEV 9
Isolation Forest Splits
https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf
AnomalyUsual data point

10. #MLSEV 10
Removing Outliers

11. #MLSEV 11
Removing Outliers
https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e

12. #MLSEV 12
Outliers
• Data points that differ significantly from other observations
• Outliers can cause serious problems in statistical analyses
• Examples:
1
2
3
4
5
6
10 20 30 40 50 60 70 80 900
Price
(100k €)
Square Meters
Regression:
1
2
3
4
5
6
0
Price
(100k €)
10 20 30 40 50 60 70 80 90
Square Meters
Unsold
Sold
Classification:

13. #MLSEV 13
Outliers
• Data points that differ significantly from other observations
• Outliers can cause serious problems in statistical analyses
• Examples:
1
2
3
4
5
6
10 20 30 40 50 60 70 80 900
Price
(100k €)
Square Meters
Regression:
1
2
3
4
5
6
0
Price
(100k €)
10 20 30 40 50 60 70 80 90
Square Meters
Unsold
Sold
Classification:

14. #MLSEV 14
Removing Outliers
ORIGINAL
DATASET
TRAIN SET
TEST SET
ALL
MODEL
CLEAN
DATASET
REJECT MOST
ANOMALOUS
CLEAN
MODEL
COMPARE
EVALUATIONS
ANOMALY
DETECTOR
• Anomaly detectors can be used to remove outliers
• With this methodology outliers removal can be tested
ALL
EVALUATION
CLEAN
EVALUATION

15. #MLSEV 15
Outliers Demo
pregnancies
plasma
glucose
blood
pressure
triceps skin
thickness
insulin bmi
diabetes
pedigree
age diabetes
6 148 72 35 0 33.6 627 50 TRUE
1 85 66 29 0 26.6 351 31 FALSE
8 183 64 0 0 23.3 672 32 TRUE
1 89 66 23 94 28.1 167 21 FALSE
0 137 40 35 168 43.1 2.288 33 TRUE
5 116 74 0 0 25.6 201 30 FALSE
3 78 50 32 88 31.0 248 26 TRUE
10 115 0 0 0 35.3 134 29 FALSE
2 197 70 45 543 30.5 158 53 TRUE
8 125 96 0 0 0.0 232 54 TRUE
4 110 92 0 0 37.6 191 30 FALSE
10 168 74 0 0 38.0 537 34 TRUE
Diabetes dataset
• Predict whether patients are diabetic or not
BigML Gallery

16. #MLSEV 16
Summary
•An anomaly detector improved a classifier performance by removing top
10 anomalies as outliers
•Usually removing anomalies with score over 60% works

17. #MLSEV 17
Fraud Detection

18. #MLSEV 18
Fraud Detection
HISTORIC NON
FRAUD
TRANSACTIONS
ANOMALY
DETECTOR
NEW
TRANSACTION(S)
ANOMALY
SCORE
KEEP HIGH
SCORES
SUSPICIOUS
TRANSACTION(S)
FRAUD
ANALYST
• Use Machine Learning to detect fraudulent financial transactions
• Fraud transactions being unusual can be detected with an anomaly
detector

19. #MLSEV 19
Fraud Detection Demo
Credit card transactions dataset
• Anonymized credit card transactions with a fraud label
• Very unbalanced
Time V1 V2 V3 V4
0 -1.3598 -0.0727 2.5363 1.3781
0 1.1918 0.2661 0.1664 0.4481
1 -1.3583 -1.3401 1.7732 0.3797
1 -0.9662 -0.1852 1.7929 -0.8632
2 -1.1582 0.8777 1.5487 0.4030
2 -0.4259 0.9605 1.1411 -0.1682
4 1.2296 0.1410 0.0453 1.2026
7 -0.6442 1.4179 1.0743 -0.4921
7 -0.8942 0.2861 -0.1131 -0.2715
9 -0.3382 1.1195 1.0443 -0.2221
10 1.4490 -1.1763 0.9138 -1.3756
V27 V28 Amount Class
0.1335 -0.0210 149.62 0
-0.0089 0.0147 2.69 0
-0.0553 -0.0597 378.66 0
0.0627 0.0614 123.5 0
0.2194 0.2151 69.99 0
0.2538 0.0810 3.67 0
0.0345 0.0051 4.99 0
-1.2069 -1.0853 40.8 1
0.0117 0.1424 93.2 0
0.2462 0.0830 3.68 0
0.0428 0.0162 7.8 0
…
…
…
https://www.kaggle.com/mlg-ulb/creditcardfraud

20. #MLSEV 20
Summary
• Anomaly detectors can be an unsupervised alternative to classifiers
in extremely unbalanced datasets
• Fraud detection is an example. A similar approach can be used for other
use cases such as predictive maintenance or network intrusion
detection
• With this approach, the most challenging aspect is finding the features
that work

21. #MLSEV 21
Novel Categories Discovery

22. #MLSEV 22
Novel Categories
• A classification model performance could be reduced over time in
production with real data evolution over time
• Model degradation can be addressed by retraining with new data
• What if new data is not labeled?
• What if new data contains novel categories?
• Anomaly detectors can be used to spot model degradation and to
discover novel categories

23. #MLSEV 23
Novel Categories Discovery
ORIGINAL
DATASET
CLASSIFICATION
MODEL
ANOMALY
DETECTOR
NEW
INSTANCES
HIGH SCORED
INSTANCES, POTENTIAL
NOVEL CATEGORIES
REJECT HIGH
ANOMALY SCORES
SIMILAR
INSTANCES
PREDICTION
LABEL/RETRAIN
MODEL ALERT
WHEN CUMULATED
ANOMALY
SCORE
DATA ANALYST

24. #MLSEV 24
Novel Categories Demo
Steel plates faults dataset
• Each instance represents a faulty steel plate with fault type label
• Objective: predict fault type given a faulty steel plate
…
…
…
X_Min X_Max Y_Min Y_Max Pixels Areas X_Perim Y_Perim
42 50 270900 270944 267 17 44
645 651 2538079 2538108 108 10 30
829 835 1553913 1553931 71 8 19
853 860 369370 369415 176 13 45
1289 1306 498078 498335 2409 60 260
430 441 100250 100337 630 20 87
413 446 138468 138883 9052 230 432
190 200 210936 210956 132 11 20
330 343 429227 429253 264 15 26
74 90 779144 779308 1506 46 167
106 118 813452 813500 442 13 48
Orientation_Index Luminosity_Index SigmoidOfAreas Fault
0.8182 -0.2913 0.5822 Pastry
0.7931 -0.1756 0.2984 Bumps
0.6667 -0.1228 215 Bumps
0.8444 -0.1568 0.5212 Dirty
0.9338 -0.1992 1.0 Stains
0.8736 -0.2267 0.9874 Pastry
0.9205 0.2791 1.0 Stains
0.5 0.1841 0.3359 Bumps
0.5 -0.1197 0.5593 Pastry
0.9024 -0.0651 1.0 Pastry
0.75 -0.1093 0.8612 Pastry
28
fields
total
BigML Gallery

25. #MLSEV 25
Summary
• Novel plates faults categories could be spotted with this method
• Model degradation in general can be monitored with anomaly detectors