This session was presented at the CFA Institute on May 6th 2020 This deep-dive session discusses core methods and applications to provide an understanding of supervised and unsupervised machine learning. Participants will be introduced to advanced topics that include time series analysis, reinforcement learning, anomaly detection, and natural language processing. Case studies will also examine how to predict interest rates and credit risk with alternative data sets and how to analyze earning calls from EDGAR using Natural Language Processing Techniques.

1. MACHINE LEARNING AND AI:
CORE METHODS AND
APPLICATIONS
06 May 2020, 9:00 am - 11:00 am EDT
Sri Krishnamurthy, CFA
President
QuantUniversity
Richard Fernand, Moderator
Senior Director, Global Content, Professional Learning
CFA Institute

2. Use Q&A to submit questions for the presenters
Use CHAT to share comments and to see
what others are saying. You can also select
who you would like to send the message to
by clicking on the drop down next to To:.

3. Machine Learning and AI
Core methods and Applications
Part 2
2020 Copyright QuantUniversity LLC.
Presented By:
Sri Krishnamurthy, CFA, CAP
sri@quantuniversity.com
www.quantuniversity.com
05/06/2020
CFA Institute - Online

4. 4
Speaker bio
• Advisory and Consultancy for Financial
Analytics
• Prior Experience at MathWorks, Citigroup
and Endeca and 25+ financial services and
energy customers.
• Columnist for the Wilmott Magazine
• Author of forthcoming book
“The Model-Driven Enterprise”
• Teaches AI/ML and Fintech Related topics in
the MS and MBA programs at Northeastern
University, Boston
• Reviewer: Journal of Asset Management
Sri Krishnamurthy
Founder and CEO
QuantUniversity

5. 5
QuantUniversity
• Boston-based Data Science, Quant
Finance and Machine Learning
training and consulting advisory
• Trained more than 1000 students in
Quantitative methods, Data Science
and Big Data Technologies using
MATLAB, Python and R
• Building a platform for AI
and Machine Learning Exploration
and Experimentation

6. 1. Key trends in AI, Machine Learning & Fintech
2. An intuitive introduction to AI and ML
3. Case study
▫ Alternative investments: Interest rate predication for Peer-to-Peer
Market places using ML techniques
▫ Scenario analysis: Synthetic VIX data generation using Neural
Networks
Recap from Part 1

7. Machine Learning Workflow
Data Scraping/
Ingestion
Data
Exploration
Data Cleansing
and Processing
Feature
Engineering
Model
Evaluation
& Tuning
Model
Selection
Model
Deployment/
Inference
Supervised
Unsupervised
Modeling
Data Engineer, Dev Ops Engineer
Data Scientist/QuantsSoftware/Web Engineer
• AutoML
• Model Validation
• Interpretability
Robotic Process Automation (RPA) (Microservices, Pipelines )
• SW: Web/ Rest API
• HW: GPU, Cloud
• Monitoring
• Regression
• KNN
• Decision Trees
• Naive Bayes
• Neural Networks
• Ensembles
• Clustering
• PCA
• Autoencoder
• RMS
• MAPS
• MAE
• Confusion Matrix
• Precision/Recall
• ROC
• Hyper-parameter
tuning
• Parameter Grids
Risk Management/ Compliance(All stages)
Analysts&
DecisionMakers

8. 8
8

9. 9
Demo 1 – Loan Data Synthesizer
(Anonymization + Data Augmentation)

10. 10
Demo 2: Synthetic Sales data generation
What-if & Scenario analysis

11. 11
Demo 3 : Synthetic VIX generation
Extreme scenario generation

12. 12
Demos, slides and video available on QuAcademy
Go to https://academy.qusandbox.com
Use code ‘CFAMasterclass’ for access
12

14. • Machine learning
▫ Unsupervised learning – Clustering etc.
▫ Supervised machine learning - Classification
• Case studies
▫ Investments: Using Clustering for investment decision-support
▫ NLP: Building your own Sentiment Analysis Engine for EDGAR filings
• Frontier topics
▫ Anomaly detection
▫ Natural Language Processing
▫ Deep learning
▫ Risk in Machine Learning and AI
▫ Model governance, Interpretability and Model Management
Part 2: Machine Learning and AI :Core methods and
applications

15. Machine Learning Workflow
Data Scraping/
Ingestion
Data
Exploration
Data Cleansing
and Processing
Feature
Engineering
Model
Evaluation
& Tuning
Model
Selection
Model
Deployment/
Inference
Supervised
Unsupervised
Modeling
Data Engineer, Dev Ops Engineer
Data Scientist/QuantsSoftware/Web Engineer
• AutoML
• Model Validation
• Interpretability
Robotic Process Automation (RPA) (Microservices, Pipelines )
• SW: Web/ Rest API
• HW: GPU, Cloud
• Monitoring
• Regression
• KNN
• Decision Trees
• Naive Bayes
• Neural Networks
• Ensembles
• Clustering
• PCA
• Autoencoder
• RMS
• MAPS
• MAE
• Confusion Matrix
• Precision/Recall
• ROC
• Hyper-parameter
tuning
• Parameter Grids
Risk Management/ Compliance(All stages)
Analysts&
DecisionMakers

17. 17
Machine Learning
Unsupervised Supervised
Reinforcement Semi-Supervised
Machine Learning

18. 18
Goal
Descriptive
Statistics
Cross
sectional
Numerical Categorical
Numerical vs
Categorical
Categorical vs
Categorical
Numerical vs
Numerical
Time series
Predictive
Analytics
Cross-
sectional
Segmentation Prediction
Predict a
number
Predict a
category
Time-series
Machine Learning Algorithms
18

19. 19
Supervised Algorithms
▫ Given a set of variables 𝑥!, predict the value of another variable 𝑦 in
a given data set such that
▫ If y is numeric => Prediction
▫ If y is categorical => Classification
▫ Example: Given a review, is the author’s sentiment good or bad?
Machine Learning
19
x1,x2,x3… Model F(X) y

20. 20
• Parametric models
▫ Assume some functional form
▫ Fit coefficients
• Examples : Logistic Regression, Neural Networks
Supervised Learning models - Classification
20
Logistic Regression Model Neural network Model

21. 21
• Non-Parametric models
▫ No functional form assumed
• Examples : K-nearest Neighbors, Decision Trees
Supervised Learning models
21
K-nearest neighbor Model Decision tree Model

22. 22
Unsupervised Algorithms
▫ Given a dataset with variables 𝑥!, build a model that captures the
similarities in different observations and assigns them to different
buckets => Clustering
▫ Example: Given a list of emerging market stocks, can we segment them
into three buckets?
Machine Learning
22
Obs1,
Obs2,Obs3
etc.
Model
Obs1- Class 1
Obs2- Class 2
Obs3- Class 1

23. 23
• These methods partition the data into k clusters by assigning each data point to its
closest cluster centroid by minimizing the within-cluster sum of squares (WSS), which
is:
!
!"#
$
!
%∈'!
!
("#
)
(𝑥%( − 𝜇!()*
where 𝑆! is the set of observations in the kth cluster and 𝜇!( is the mean of jth
variable of the cluster center of the kth cluster.
• Then, they select the top n points that are the farthest away from their nearest
cluster centers as outliers.
K-means clustering
23

24. 24
Euclidean distance:
Distance functions

25. 25
Correlation distance:
Distance functions

26. 26
Hierarchical Clustering
• Agglomerative: This is a "bottom-
up" approach: each observation
starts in its own cluster, and pairs
of clusters are merged as one
moves up the hierarchy.
• Divisive: This is a "top-down"
approach: all observations start in
one cluster, and splits are
performed recursively as one
moves down the hierarchy.
Source:
https://en.wikipedia.org/wiki/Hierarchical_cluster
ing

27. 27
Affinity propogation
• Affinity propagation (AP) is a clustering algorithm based on
the concept of "message passing" between data points
• Affinity propagation finds "exemplars," members of the
input set that are representative of clusters
• Affinity propagation does not require the number of clusters
to be determined or estimated before running the algorithm
• Source: https://en.wikipedia.org/wiki/Affinity_propagation

28. 28
Unsupervised Algorithms
▫ PCA->
– Dimension reduction using orthogonal linear transformations
▫ Manifold Learning ->
– Non-linear dimension reduction
▫ Association Rule Mining ->
– People who bought X also bought Y
▫ https://en.wikipedia.org/wiki/Principal_component_analysis
Machine Learning
28

29. 29
http://www.asimovinstitute.org/neural-network-zoo/

30. 30
Machine
Learning
Supervised
Prediction
Parametric
Linear
Regression
Neural
Networks
Non-
parametric
KNN
Decision
Trees
Classification
Parametric
Logistic
Regression
Neural
Networks
Non
Parametric
Decision
Trees KNN
Unsupervised
algorithms
K-means
Associative
rule mining
Machine Learning Algorithms
30

31. 31
Machine Learning movers and shakers
Deep
Learning
Automatic
Machine
Learning
Ensemble
Learning
Natural
Language
Processing
Data Robot
H20.ai
Autosklearn
autokkeras
Tensorflow
Pytorch
NLTK
HuggingFace
Bagging
Boosting
DNN
CNN
LSTM
GAN

33. 33
The Process
33
Data
ingestion
Data
cleansing
Feature
engineering
Training
and testing
Model
building
Model
selection

35. 35
Evaluating
Machine learning
algorithms
Supervised -
Prediction
R-square RMS MAE MAPE
Supervised-
Classification
Confusion Matrix ROC Curves
Evaluation framework
35

36. 36
• Consider a two-class case with classes 𝐶! and 𝐶"
• Classification matrix:
Classification matrix
Predicted Class
Actual Class 𝐶! 𝐶"
𝐶!
𝑛!,!= number of 𝐶! cases
classified correctly
𝑛!,"= number of 𝐶! cases
classified incorrectly as 𝐶"
𝐶"
𝑛",!= number of 𝐶" cases
classified incorrectly as 𝐶!
𝑛","= number of 𝐶" cases
classified correctly

37. 37
• Estimated misclassification rate (overall error rate) is a main
accuracy measure
𝑒𝑟𝑟 =
𝑛!," + 𝑛",!
𝑛!,! + 𝑛!," + 𝑛",! + 𝑛","
=
𝑛!," + 𝑛",!
𝑛
• Overall accuracy:
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 1 − 𝑒𝑟𝑟 =
𝑛!,! + 𝑛","
𝑛
Accuracy Measures

38. 38
• The ROC curve plots the pairs {sensitivity, 1-
specificity} as the cutoff value increases from 0
and 1
• Sensitivity (also called the true positive rate, or
recall in some fields) measures the proportion of
positives that are correctly identified
▫ e.g., the percentage of COVID19 diagnosed people who
are correctly identified as having COVID19
• Specificity (also called the true negative rate)
measures the proportion of negatives that are
correctly identified as such.
▫ e.g., the percentage of healthy people who are correctly
identified as not having COVID19
• Better performance is reflected by curves that are
closer to the top left corner
ROC Curve

39. 39
1. Data
2. Goals
3. Machine learning algorithms
4. Process
5. Performance Evaluation
Recap

40. Machine Learning Workflow
Data Scraping/
Ingestion
Data
Exploration
Data Cleansing
and Processing
Feature
Engineering
Model
Evaluation
& Tuning
Model
Selection
Model
Deployment/
Inference
Supervised
Unsupervised
Modeling
Data Engineer, Dev Ops Engineer
Data Scientist/QuantsSoftware/Web Engineer
• AutoML
• Model Validation
• Interpretability
Robotic Process Automation (RPA) (Microservices, Pipelines )
• SW: Web/ Rest API
• HW: GPU, Cloud
• Monitoring
• Regression
• KNN
• Decision Trees
• Naive Bayes
• Neural Networks
• Ensembles
• Clustering
• PCA
• Autoencoder
• RMS
• MAPS
• MAE
• Confusion Matrix
• Precision/Recall
• ROC
• Hyper-parameter
tuning
• Parameter Grids
Risk Management/ Compliance(All stages)
Analysts&
DecisionMakers

42. Investments: Clustering of stocks

43. 43
1. Case Intro
2. Data Exploration of DOW stock data
3. Problem Definition and Machine learning
4. Deployment
Case study

44. 44
Introduction
44
Source: https://novelinvestor.com/sector-performance/

45. 45
Clustering stocks
• Which stocks are like each other?
• Are growth stocks behaving like growth stocks or value
stocks?
• Does the time series of prices & returns reveal which
stocks are close to each other?

46. 46
The Data
46
https://fred.stlouisfed.org/series/DJIA

47. 47
Clustering workflow
Data
Ingestion
Pre-
Processing
Clustering
Visualization
& analysis
Model
Deployment
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

48. 48
• Clustering
▫ How do we define distance between stocks?
– Correlation
– 1- Correlation2 (https://arxiv.org/abs/cond-mat/9802256)
▫ Hierarchical Clustering
Methodology

49. 49
• Clustering
▫ How do we define distance between stocks?
– Covariance Estimation - https://scikit-
learn.org/stable/modules/covariance.html#sparse-inverse-covariance
▫ Clustering: Affinity propagation - https://scikit-
learn.org/stable/modules/clustering.html#affinity-propagation
▫ Visualizing relationships with manifold learning –
https://scikit-learn.org/stable/modules/manifold.html#manifold
Methodology

50. 50
▫ Visualizing relationships
Methodology

51. 51
▫ Visualizing relationships
https://scikit-learn.org/stable/modules/manifold.html#manifold
Methodology

52. 52
52

53. 53
53

54. 54
54

55. 55
1. http://awesome-streamlit.org/
2. https://scikit-
learn.org/stable/auto_examples/applications/plot_stock_mar
ket.html
Acknowledgements

57. Natural Language Processing

58. 58
• Understanding sentiments in Earnings call transcripts
Goal

59. 59
• Interpreting emotions
• Labeling data
Challenges

60. 60
What is NLP ?
AI
Linguistics
Computer
Science

61. 61
• Q/A
• Dialog systems - Chatbots
• Topic summarization
• Sentiment analysis
• Classification
• Keyword extraction - Search
• Information extraction – Prices, Dates, People etc.
• Tone Analysis
• Machine Translation
• Document comparison – Similar/Dissimilar
Sample applications

62. 62
NLP in Finance

63. 63
• If computers can understand language, opens huge possibilities
▫ Read and summarize
▫ Translate
▫ Describe what’s happening
▫ Understand commands
▫ Answer questions
▫ Respond in plain language
Language allows understanding

64. 64
• Describe rules of grammar
• Describe meanings of words and their
relationships
• …including all the special cases
• ...and idioms
• ...and special cases for the idioms
• ...
• ...understand language!
Traditional language AI
https://en.wikipedia.org/wiki/Formal_language

65. 65
What is NLP ?
Jumping NLP Curves
https://ieeexplore.ieee.org/document/6786458/

66. 66
Q: What’s hard about writing programs
to understand text?

67. 67
• Ambiguity:
▫ “ground”
▫ “jaguar”
▫ “The car hit the pole while it was moving”
▫ “One morning I shot an elephant in my pajamas. How he got into my
pajamas, I’ll never know.”
▫ “The tank is full of soldiers.”
“The tank is full of nitrogen.”
Language is hard to deal with

68. 68

69. 69
• Many ways to say the same thing
▫ “the same thing can be said in many ways”
▫ “language is versatile”
▫ “The same words can be arranged in many different ways to express
the same idea”
▫ …
Language is hard to deal with

70. 70
• APIs
• Human Insight
• Expert Knowledge
• Build your own
Options?

71. 71
NLP pipeline
Data Ingestion
from Edgar
Pre-Processing
Invoking APIs to
label data
Compare APIs
Build a new
model for
sentiment
Analysis

74. • Frontier Topics
• 10 Things to remember

75. 75
Machine Learning movers and shakers
Deep
Learning
Automatic
Machine
Learning
Ensemble
Learning
Natural
Language
Processing
Data Robot
H20.ai
Autosklearn
autokkeras
Tensorflow
Pytorch
NLTK
HuggingFace
Bagging
Boosting
DNN
CNN
LSTM
GAN

76. 76
http://www.asimovinstitute.org/neural-network-zoo/

77. 77
What is AutoML?
Automated Machine Learning provides methods and processes to
make Machine Learning available for non-Machine Learning experts,
to improve efficiency of Machine Learning and to accelerate research
on Machine Learning.
• Incarnations: DataRobot, H20.ai, Azure ML, Amazon Sagemaker
AutoPilot, Google Automl
• Check https://www.automl.org/automl/ for more details
AutoML
http://www.asimovinstitute.org/neural-network-zoo/

78. 78
• Simple weighting (Crude but still being used!)
• Bayesian averaging
• Gradient Boosted Decision Trees, Random Forest etc.
Ensemble learning

79. 79
• Requires lots of data
• Labeling and annotation is a big problem
• ML as-a-service is the way to go unless very specific domain
expertise and data is available
Natural Language Processing

80. 80
• Fraud Detection
▫ Credit card fraud detection
– By owner or by operation
▫ Mobile phone fraud/anomaly detection
– Calling behavior, volume etc.
▫ Insurance claim fraud detection
– Medical malpractice
– Auto insurance
▫ Insider trading detection
▫ Pricing issues
▫ Network issues
Anomaly Detection

81. 81
Elements of Model Risk Management

82. 82
• Components that needs to be tracked
WHAT CONSTITUTES AN ML MODEL?
82
• Programming environment
• Execution environment
• Hardware specs
• Cloud
• GPU
• Interdependencies
• Lineage/Provenance of
individual components
• Model params
• Hyper parameters
• Pipeline specifications
• Model specific
• Tests
• Data versions
Data Model
EnvironmentProcess

83. 83
ELEMENTS OF A MACHINE LEARNING SYSTEM
Source: Sculley et al., 2015 "Hidden Technical Debt in Machine Learning Systems"

84. 84
The reproducibility challenge

87. 87
1. Articulate your business problem
10 Things to Remember

88. 88
2. The Data questions
1. Do you know what data you need ?
2. Do you know if the data is available?
3. Do you have the data ?
4. Do you have the right data?
5. Will you continue to have the data?
10 Things to Remember

89. 89
3. Develop a data acquisition and data prep strategy
1. Do you know how to get the data ?
2. Who gets the data?
3. How do you process it?
4. How do you access it?
5. How do you version and govern the data?
10 Things to Remember

90. 90
4. Explore and evaluate your data and get it in the right format
10 Things to Remember

91. 91
5. Define your goal:
1. Summarization
2. Fact finding
3. Understanding relationships
4. Prediction
10 Things to Remember

92. 92
6. Shortlist (not “Choose” ) the
techniques/methodologies/algorithms
10 Things to Remember

93. 93
7. Evaluate/establish business constraints and narrow down your
choices of techniques/methodologies/algorithms
1. Cloud/Cost/Expertise/Cost-Value
2. Build/buy/access
10 Things to Remember
Outcomes
Time
Quality
Cost

94. 94
8. Establish criteria to know if the methodology/models/algorithms
work
1. Is the process replicable?
2. What performance metrics do we choose?
3. Can you evaluate the performance and validate if the models meet
the criteria?
4. Does it provide business value?
10 Things to Remember

95. 95
9. Fine tune your algorithms and algorithm selection
1. Hyper parameter tuning
2. Bias-variance tradeoff
3. Handling imbalanced class problems
4. Ensemble techniques
5. AutoML
10 Things to Remember
https://support.sas.com/resources/papers/proceedings17/SAS0514-2017.pdf

96. 96
10. How will this process reach decision makers
1. Deployment choices (On-prem/Cloud)
2. Frequency of data/model updates
3. Governance/Role/Responsibilities
4. Speed, Scale, Availability, Disaster recovery, Rollback, Pull-Plug
10 Things to Remember

97. 97
How do you monitor the efficacy of your solution?
1. Retuning
2. Monitoring
3. Model decay
4. Data augmentation
5. Newer innovations
10+1 Things to Remember

98. Machine Learning Workflow
Data Scraping/
Ingestion
Data
Exploration
Data Cleansing
and Processing
Feature
Engineering
Model
Evaluation
& Tuning
Model
Selection
Model
Deployment/
Inference
Supervised
Unsupervised
Modeling
Data Engineer, Dev Ops Engineer
Data Scientist/QuantsSoftware/Web Engineer
• AutoML
• Model Validation
• Interpretability
Robotic Process Automation (RPA) (Microservices, Pipelines )
• SW: Web/ Rest API
• HW: GPU, Cloud
• Monitoring
• Regression
• KNN
• Decision Trees
• Naive Bayes
• Neural Networks
• Ensembles
• Clustering
• PCA
• Autoencoder
• RMS
• MAPS
• MAE
• Confusion Matrix
• Precision/Recall
• ROC
• Hyper-parameter
tuning
• Parameter Grids
Risk Management/ Compliance(All stages)
Analysts&
DecisionMakers

100. 1. Key trends in AI, Machine Learning & Fintech
2. An intuitive introduction to AI and ML
3. Case study
▫ Alternative investments: Interest rate predication for Peer-to-Peer
Market places using ML techniques
▫ Scenario analysis: Synthetic VIX data generation using Neural
Networks
Part 1

101. • Machine learning
▫ Unsupervised learning – Clustering etc.
▫ Supervised machine learning - Classification
• Case studies
▫ Investments: Using Clustering for investment decision-support
▫ NLP: Building your own Sentiment Analysis Engine for EDGAR filings
• Frontier topics
▫ Anomaly detection
▫ Natural Language Processing
▫ Deep learning
▫ Risk in Machine Learning and AI
▫ Model governance, Interpretability and Model Management
Part 2: Machine Learning and AI :Core methods and
applications

102. 102
Demos, slides and video available on QuAcademy
Go to https://academy.qusandbox.com
Use code ‘CFAMasterclass’ for access
102

103. 103
https://qupython.splashthat.com/

104. Register at
https://mlinfinance.splashthat.com
Classes start
May 12th
104
Use code ‘QUAffiliate’ to take $100 off

106. Thank you!
Sri Krishnamurthy, CFA, CAP
Founder and CEO
QuantUniversity LLC.
srikrishnamurthy
www.QuantUniversity.com
Contact
Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be
distributed or used in any other publication without the prior written consent of QuantUniversity LLC.
106