1.
Statistics: Unlocking the Power of Data Lock5
Financial DATA ANALYTICS
Dr. M.Vijayalakshmi, VESIT
4th jan 2018, SAKEC Mumbai

2.
Statistics: Unlocking the Power of Data Lock5
Financial Data
The financial industry has always been driven by data.
Today, Big Data is prevalent at various levels of this field, ranging from
the financial services sector to capital markets.
The availability of Big Data in this domain has opened up new avenues
for innovation and has offered immense opportunities for growth and
sustainability.
At the same time, it has presented several new challenges that must be
overcome to gain the maximum value out of it.

3.
Statistics: Unlocking the Power of Data Lock5
Financial Data Analytics in a Nut Shell

4.
Statistics: Unlocking the Power of Data Lock5
Motivation
There has been an explosion in the velocity, variety and volume of financial
data. Social media activity, mobile interactions, server logs, real-time market
feeds, customer service records, transaction details, information from existing
databases – there’s no end to the flood.
To make sense of these giant data sets, companies are increasingly turning to
data scientists for answers. These numbers gurus are:
 Capturing and analyzing new sources of data, building predictive models and running
live simulations of market events
 Using technologies such as Hadoop, NoSQL and Storm to tap into non-traditional data
sets (e.g., geolocation, sentiment data) and integrate them with more traditional
numbers (e.g., trade data)
 Finding and storing increasingly diverse data in its raw form for future analysis
They’ve been aided in this quest by the development of cloud-based data
storage and the surge of sophisticated (and sometimes free or open-source)
analytics tools.

5.
Statistics: Unlocking the Power of Data Lock5
Important Applications of Financial
Data Analytics
1. Predictive Analytics / Trading
2. Sentiment Analysis
3. Financial Fraud
4. Credit Scoring Ratings
5. Pricing
6. Customer Segmentation
7. Know Your Customer

6.
Statistics: Unlocking the Power of Data Lock5
Sentiment Analysis
Sentiment analysis (aka opinion mining) applies natural-language
processing, text analysis and computational linguistics to source material
to discover what folks really think.
Several big Businesses like MarketPsy Capital, Think Big Analytics and
MarketPsych Data are using it to:
Build algorithms around market sentiment data (e.g., Twitter feeds) that
can short the market when disasters (e.g., storms, terrorist attacks) occur
Track trends, monitor the launch of new products, respond to issues and
improve overall brand perception
Analyze unstructured voice recordings from call centers and recommend
ways to reduce customer churn, up-sell and cross-sell products and detect
fraud
Some data companies are even acting as intermediaries, collecting and
selling sentiment indicators to retail investors.

7.
Statistics: Unlocking the Power of Data Lock5
Automated Risk Credit Management
Internet finance companies are finding ways to approve loans and manage risk.
Aliloan (from AliBaba) is an automated online system that provides flexible
micro-loans to entrepreneurial online vendors.
To gauge whether a vendor is creditworthy, Alibaba collects data from its e-
commerce and payment platforms and analyzes transaction records, customer
ratings, shipping records and a host of other info.
These findings are confirmed by third-party verification and cross-checked
against external data sets (e.g., customs, tax data, electricity records, etc.).
Once the loan is granted, Alibaba continues to monitor the use of funds and
assess the business’s strategic development.
Entrepreneurs in emerging markets are also reaping the benefits. Like Aliloan,
companies such as Kreditech and Lenddo provide automated small loans based
on innovative credit scoring techniques. In these cases, much of the score is
calculated from applicants’ online social networking data.

8.
Statistics: Unlocking the Power of Data Lock5
Real Time Analytics
In days of yore, financial institutions were hampered by the lag-time between data
collection and data analysis. Real-time analytics short-circuits this problem and provides
the industry with new ways to:
Fight Financial Fraud: Banks and credit card companies routinely analyze account
balances, spending patterns, credit history, employment details, location and a load of
other data points to determine whether transactions are above aboard. If suspicious
activity is detected, they can immediately suspend the account and alert the owner.
Improve Credit Ratings: A continuous feed of online data means credit ratings can
be updated in real time. This provides lenders with a more accurate picture of a
customer’s assets, business operations and transaction history.
Provide More Accurate Pricing: Progressive Insurance already tailors its policies to
account for a customer’s changing financial situation. In the Internet of Things, data
from automobile sensors will also help insurance companies issues its policy holders
with warnings about accidents, traffic jams and weather conditions. That makes for
safer drivers and fewer payouts

9.
Statistics: Unlocking the Power of Data Lock5
Customer Segmentation
Like every other industry on the planet, banks and financial
institutions are hungry to know more about the people using their
products and services. And though they already store a ton of data
– from credit scores to day-to-day transactions – they’re not too
proud to look for it elsewhere.
 This kind of customer segmentation allows them to:
 Offer customized product offerings and services
 Improve existing profitable relationships and avoid customer churn
 Create better marketing campaigns and more attractive product offerings
 Tailor product development to specific customer segments

10.
Statistics: Unlocking the Power of Data Lock5
Predictive Analytics
By combining segmentation with predictive analytics, companies can also cut down on
risk. For example, to decide whether certain customers are likely to pay off their credit
cards, some major banks use technology developed by the company Sqrrl. This analysis
takes into account the demographic characteristics of customers’ neighborhoods and
makes calculated predictions.
Similar strides have been made in forecasting market behavior. Once upon a time (e.g.,
2009), high-frequency trading – the speedy exchange of securities – was hugely
lucrative. With competition came a drop in profits and the need for a new strategy.
HFT traders adapted by employing strategic sequential trading, using big data analytics
to identify specific market participants and anticipate their future actions. In a field of
breakneck speed, this gives HFT traders an unmistakable advantage.
By studying search volume data provided by Google Trends, they were able to identify
online precursors for stock market moves. Their results suggest that increases in search
volume for financially relevant search terms usually precede big losses in financial
markets.

11.
Statistics: Unlocking the Power of Data Lock5
Analytics of Financial Times Series
A vast majority of Financial data occurs in the form of a times series
 Stock prices (ticker data)
 Asset prices
 Customer Numbers
 Etc
So Financial Data Analytics places a lot of importance on Financial times
series analytics

12.
Statistics: Unlocking the Power of Data Lock5
Examples of financial time series
Daily log returns of Apple stock: 2007 to 2016 (10 years)
BSE index
Quarterly earnings of Coca-Cola Company: 1983-2009 Seasonal time
series useful in
 earning forecasts
 pricing weather related derivatives (e.g. energy)
 modeling intraday behavior of asset returns
Exchange rate between US Dollar vs Re
Size of insurance claims Values
High-frequency financial data: Tick-by-tick data of stock, etc

13.
13
Mining Time-Series Data
A time series is a sequence of data points, measured typically at
successive times, spaced at (often uniform) time intervals
Time series analysis: A subfield of statistics, comprises methods that
attempt to understand such time series, often either to understand the
underlying context of the data points or to make forecasts (or
predictions)
Methods for time series analyses
 Frequency-domain methods: Model-free analyses, well-suited to
exploratory investigations
 spectral analysis vs. wavelet analysis
 Time-domain methods: Auto-correlation and cross-correlation
analysis
 Motif-based time-series analysis
Applications
 Financial: stock price, inflation
 Industry: power consumption
 Scientific: experiment results
 Meteorological: precipitation

14.
Statistics: Unlocking the Power of Data Lock5 14
Time-Series Data Analysis: Prediction &
Regression Analysis
(Numerical) prediction is similar to classification
 construct a model
 use model to predict continuous or ordered value for a given input
Prediction is different from classification
 Classification refers to predict categorical class label
 Prediction models continuous-valued functions
Major method for prediction: regression
 model the relationship between one or more independent or
predictor variables and a dependent or response variable
Regression analysis
 Linear and multiple regression
 Non-linear regression
 Other regression methods: generalized linear model, Poisson
regression, log-linear models, regression trees

15.
Statistics: Unlocking the Power of Data Lock5 15
What is Regression?
Modeling the relationship between one response variable and one or
more predictor variables
Analyzing the confidence of the model
E.g, height v.s weight

16.
Statistics: Unlocking the Power of Data Lock5 16
Regression Yields Analytical Model
Discrete data points →Analytical model
 General relationship
 Easy calculation
 Further analysis
Application - Prediction

17.
Statistics: Unlocking the Power of Data Lock5 17
Application - Detrending
Obtain the trend for irregular data series
Subtract trend
Reveal oscillations
trend

18.
Statistics: Unlocking the Power of Data Lock5 18
Linear Regression - Single Predictor
Model is linear
y = w0 + w1 x
where w0 (y-intercept) and w1
(slope) are regression coefficients
Method of least squares:
y: response
variable
x: predictor
variable
w1
w0
| |
1
| |
2
1
( )( )
1
( )
D
i i
i
D
i
i
x x y y
x x
w 

 



 x
w
y
w
1
0



19.
Statistics: Unlocking the Power of Data Lock5 19
Training data is of the form (X1, y1), (X2, y2),…, (X|D|, y|D|)
E.g., for 2-D data or
y = w0 + w1 x1+ w2 x2
Solvable by
 Extension of least square method
(XTX ) W=Y →W = (XTX ) -1Y
 Commercial software (SAS, S-Plus) x1
x2
y
Linear Regression – Multiple Predictor

20.
Statistics: Unlocking the Power of Data Lock5 20
Nonlinear Regression with Linear Method
Polynomial regression model
 E.g., y = w0 + w1 x + w2 x2 + w3 x3
Let x2 = x2, x3= x3
y = w0 + w1 x + w2 x2 + w3 x3
Log-linear regression model
 E. g., y = exp(w0 + w1 x + w2 x2 + w3 x3 )
Let y’=log(y)
y’= w0 + w1 x + w2 x2 + w3 x3

21.
Statistics: Unlocking the Power of Data Lock5 21
Generalized Linear Regression
Response y
 Distribution function in the exponential family
 Variance of y depends on E( y), not a constant
E( y) = g-1( w0 + w1 x + w2 x2 + w3 x3 )
Examples
 Logistic regression (binomial regression): probability of some
event occurring
 Poisson regression: number of customers
 …
References: Nelder and Wedderburn, 1972; McCullagh and
Nelder, 1989

22.
22
Regression Tree (Breiman et al., 1984)
Partition the domain space
Leaf: (1) a continuous-valued
prediction; (2) average value

23.
Statistics: Unlocking the Power of Data Lock5 23
Model Tree
Leaf – a linear equation
More general than regression tree
Figure source: http://datamining.ihe.nl/research/model-trees.htm

24.
Statistics: Unlocking the Power of Data Lock5 24
Regression Trees and Model Trees
Regression tree: proposed in CART system (Breiman et al. 1984)
 CART: Classification And Regression Trees
 Each leaf stores a continuous-valued prediction
 It is the average value of the predicted attribute for the training tuples that
reach the leaf
Model tree: proposed by Quinlan (1992)
 Each leaf holds a regression model—a multivariate linear equation for the
predicted attribute
 A more general case than regression tree
Regression and model trees tend to be more accurate than linear
regression when the data cannot be represented well by a simple
linear model

25.
Statistics: Unlocking the Power of Data Lock5 25
A time series can be illustrated as a time-series graph
which describes a point moving with the passage of time

26.
Statistics: Unlocking the Power of Data Lock5 26
Categories of Time-Series Movements
Categories of Time-Series Movements
 Long-term or trend movements (trend curve): general direction in
which a time series is moving over a long interval of time
 Cyclic movements or cycle variations: long term oscillations about a
trend line or curve
e.g., business cycles, may or may not be periodic
 Seasonal movements or seasonal variations
i.e, almost identical patterns that a time series appears to follow
during corresponding months of successive years.
 Irregular or random movements
Time series analysis: decomposition of a time series into these four
basic movements
 Additive Modal: TS = T + C + S + I
 Multiplicative Modal: TS = T  C  S  I

27.
Statistics: Unlocking the Power of Data Lock5
Estimation of Trend Curve
The freehand method
 Fit the curve by looking at the graph
 Costly and barely reliable for large-scaled data mining
The least-square method
 Find the curve minimizing the sum of the squares of the deviation of points on
the curve from the corresponding data points
The moving-average method
27

28.
Statistics: Unlocking the Power of Data Lock5 28
Moving Average
Moving average of order n
 Smoothes the data
 Eliminates cyclic, seasonal and irregular movements
 Loses the data at the beginning or end of a series
 Sensitive to outliers (can be reduced by weighted moving
average)

29.
Statistics: Unlocking the Power of Data Lock5 29
Trend Discovery in Time-Series (1):
Estimation of Seasonal Variations
Seasonal index
 Set of numbers showing the relative values of a variable during the
months of the year
 E.g., if the sales during October, November, and December are 80%,
120%, and 140% of the average monthly sales for the whole year,
respectively, then 80, 120, and 140 are seasonal index numbers for
these months
Deseasonalized data
 Data adjusted for seasonal variations for better trend and cyclic
analysis
 Divide the original monthly data by the seasonal index numbers for
the corresponding months

30.
Statistics: Unlocking the Power of Data Lock5
February 2, 2023 Data Mining: Concepts and Techniques 30
Seasonal Index
0
20
40
60
80
100
120
140
160
1 2 3 4 5 6 7 8 9 10 11 12
Month
Seasonal Index
Raw data from
http://www.bbk.ac.uk/mano
p/man/docs/QII_2_2003%2
0Time%20series.pdf

31.
Statistics: Unlocking the Power of Data Lock5
Trend Discovery in Time-Series (2)
Estimation of cyclic variations
 If (approximate) periodicity of cycles occurs, cyclic index can be constructed in
much the same manner as seasonal indexes
Estimation of irregular variations
 By adjusting the data for trend, seasonal and cyclic variations
With the systematic analysis of the trend, cyclic, seasonal, and irregular
components, it is possible to make long- or short-term predictions with
reasonable quality
31

32.
Statistics: Unlocking the Power of Data Lock5 32
Similarity Search in Time-Series Analysis
Normal database query finds exact match
Similarity search finds data sequences that differ only
slightly from the given query sequence
Two categories of similarity queries
 Whole matching: find a sequence that is similar to the query
sequence
 Subsequence matching: find all pairs of similar sequences
Typical Applications
 Financial market
 Market basket data analysis
 Scientific databases
 Medical diagnosis

33.
Statistics: Unlocking the Power of Data Lock5 33
Data Transformation
Many techniques for signal analysis require the data to be
in the frequency domain
Usually data-independent transformations are used
 The transformation matrix is determined a priori
 discrete Fourier transform (DFT)
 discrete wavelet transform (DWT)
The distance between two signals in the time domain is
the same as their Euclidean distance in the frequency
domain

34.
Statistics: Unlocking the Power of Data Lock5 34
Discrete Fourier Transform
DFT does a good job of concentrating energy in the first
few coefficients
If we keep only first a few coefficients in DFT, we can
compute the lower bounds of the actual distance
Feature extraction: keep the first few coefficients (F-index)
as representative of the sequence

35.
Statistics: Unlocking the Power of Data Lock5 35
DFT (continued)
Parseval’s Theorem
The Euclidean distance between two signals in the time
domain is the same as their distance in the frequency
domain
Keep the first few (say, 3) coefficients underestimates the
distance and there will be no false dismissals!







1
0
2
1
0
2
|
|
|
|
n
f
f
n
t
t X
x
|
]
)[
(
]
)[
(
|
|
]
[
]
[
|
3
0
2
0
2

 






f
n
t
f
Q
F
f
S
F
t
Q
t
S 


36.
Statistics: Unlocking the Power of Data Lock5 36
Multidimensional Indexing in Time-Series
Multidimensional index construction
 Constructed for efficient accessing using the first few Fourier coefficients
Similarity search
 Use the index to retrieve the sequences that are at most a certain small distance
away from the query sequence
 Perform post-processing by computing the actual distance between sequences in
the time domain and discard any false matches

37.
Statistics: Unlocking the Power of Data Lock5
Subsequence Matching
Break each sequence into a set of pieces of window with length w
Extract the features of the subsequence inside the window
Map each sequence to a “trail” in the feature space
Divide the trail of each sequence into “subtrails” and represent each of
them with minimum bounding rectangle
Use a multi-piece assembly algorithm to search for longer sequence
matches
37

38.
Statistics: Unlocking the Power of Data Lock5 38
Analysis of Similar Time Series

39.
Statistics: Unlocking the Power of Data Lock5
Enhanced Similarity Search Methods
Allow for gaps within a sequence or differences in offsets or amplitudes
Normalize sequences with amplitude scaling and offset translation
Two subsequences are considered similar if one lies within an envelope of
 width around the other, ignoring outliers
Two sequences are said to be similar if they have enough non-
overlapping time-ordered pairs of similar subsequences
Parameters specified by a user or expert: sliding window size, width of an
envelope for similarity, maximum gap, and matching fraction
39

40.
Statistics: Unlocking the Power of Data Lock5 40
Steps for Performing a Similarity Search
Atomic matching
 Find all pairs of gap-free windows of a small length that are
similar
Window stitching
 Stitch similar windows to form pairs of large similar
subsequences allowing gaps between atomic matches
Subsequence Ordering
 Linearly order the subsequence matches to determine whether
enough similar pieces exist

41.
Statistics: Unlocking the Power of Data Lock5 41
Similar Time Series Analysis
VanEck International Fund Fidelity Selective Precious Metal and Mineral Fund
Two similar mutual funds in the different fund group

42.
Statistics: Unlocking the Power of Data Lock5 42
Sequence Distance
A function that measures the differentness of two
sequences (of possibly unequal length)
Example: Euclidean Distance between TS Q,C



n
i i
i c
q
C
Q
D 1
2
)
(
)
,
(

43.
Statistics: Unlocking the Power of Data Lock5 43
Motif: Basic Concepts
What is a motif? A previously unknown, frequently
occurring sequential pattern
Match: Given subsequences Q,C ⊆ T,
C is a match for Q iff for some R
Non-Trivial Match: C = T[p..*], Q = T[q..*] and C match Q.
If p = q or ∄ non-match N = T[s..*] such that s between p,q
then match is non-trivial.
(i.e. C,Q must be separated by a non-match)
1-Motif: the subsequence with most non-trivial matches
(least variance decides ties)
k-Motif: Ck such that D(Ck,Ci) > 2R ∀i ∈ [1,k)
R
C
Q
D 
)
,
(

44.
Statistics: Unlocking the Power of Data Lock5 44
SAX: Symbolic Aggregate approXimation
Dim. Reduction/Compression
“Symbolic Aggregate approXimation”
SAX : ℝ → ∑
SAX : ↦ ccbaabbbabcbcb
Essentially an alphabet over the Piecewise Aggregate
Approximation (PAA) rank
Faster, simpler, more compression, yet on par with DFT,
DWT and other dim. reductions

45.
Statistics: Unlocking the Power of Data Lock5 45
SAX Illustration

46.
Statistics: Unlocking the Power of Data Lock5 46
SAX Algorithm
Parameters: alphabet size, word (segment) length (or output
rate)
1.Select probability distribution for TS
2.z-Normalize TS
3.PAA: Within each time interval, calculate aggregated value
(mean) of the segment
4.Partition TS range by equal-area partitioning the PDF into
n partitions (eq. freq. binning)
5.Label each segment with arank ∈∑ for aggregate’s
corresponding partition rank

47.
Statistics: Unlocking the Power of Data Lock5 47
Finding Motifs in a Time Series
EMMA Algorithm: Finds 1-(k-)motif of fixed length n
SAX Compression (Dim. Reduction)
 Possible to store D(i,j) ∀(i,j) ∈ ∑∑
 Allows use of various distance measures (Minkowski, Dynamic Time
Warping)
Multiple Tiers
 Tier 1: Uses sliding window to hash length-w SAX subsequences
(aw addresses, total size O(m)).
Bucket B with most collisions & buckets with
MINDIST(B) < R form neighborhood of B.
 Tier 2: Neighborhood is pruned using more precise ADM
algorithm. Ni with max. matches is 1-motif. Early stop if |ADM
matches| > maxk>i(|neighborhoodk|)

48.
Statistics: Unlocking the Power of Data Lock5 48
Hashing
c e c a b b c b a c c e c a b b c b a c
c c c c b b c c d c
w
n
2 4 2 0 1 1 2 1 0 2
5
2 2 2 2 1 1 2 2 3 2
5
2 4 2 0 1 1 2 1 0 2
5
… …
… …
…
… …
…
…
…
…

49.
Statistics: Unlocking the Power of Data Lock5
Classification in Time Series
Application: Finance,
1-Nearest Neighbor
 Pros: accurate, robust, simple
 Cons: time and space complexity (lazy learning); results are not
interpretable
0 200 400 600 800 1000 1200

50.
Statistics: Unlocking the Power of Data Lock5
Financial Data Applications
Fraud Detection - Anomaly Analysis

51.
Statistics: Unlocking the Power of Data Lock5
What are Anomalies?
Anomaly is a pattern in the data that does not conform to
the expected behavior
Also referred to as outliers, exceptions, peculiarities,
surprise, etc.
Anomalies translate to significant (often critical) real life
entities
 Cyber intrusions
 Credit card fraud

52.
Statistics: Unlocking the Power of Data Lock5
Real World Anomalies
Credit Card Fraud
 An abnormally high purchase made on a
credit card
Cyber Intrusions
 A web server involved in ftp traffic

53.
Statistics: Unlocking the Power of Data Lock5
Simple Example
N1 and N2 are regions of
normal behavior
Points o1 and o2 are
anomalies
Points in region O3 are
anomalies
X
Y
N1
N2
o1
o2
O3

54.
Statistics: Unlocking the Power of Data Lock5
Related problems
Rare Class Mining
Chance discovery
Novelty Detection
Exception Mining
Noise Removal
Black Swan*

55.
Statistics: Unlocking the Power of Data Lock5
Key Challenges
Defining a representative normal region is
challenging
The boundary between normal and outlying
behavior is often not precise
The exact notion of an outlier is different for
different application domains
Availability of labeled data for training/validation
Malicious adversaries
Data might contain noise
Normal behavior keeps evolving

56.
Statistics: Unlocking the Power of Data Lock5
Data Labels
Supervised Anomaly Detection
 Labels available for both normal data and anomalies
 Similar to rare class mining
Semi-supervised Anomaly Detection
 Labels available only for normal data
Unsupervised Anomaly Detection
 No labels assumed
 Based on the assumption that anomalies are very rare compared to normal data

57.
Statistics: Unlocking the Power of Data Lock5
Applications of Anomaly Detection
Insurance / Credit card fraud detection
Anti-Money Laundering (AML)
Fraud
Identity Theft and Fake Account Registration
Risk Modeling
Account Takeover
Promotion Credit Abuse
Customer Behavior Analytics
Cyber Security

58.
Fraud Detection
Fraud detection refers to detection of criminal activities
occurring in commercial organizations
 Malicious users might be the actual customers of the organization
or might be posing as a customer (also known as identity theft).
Types of fraud
 Credit card fraud
 Insurance claim fraud
 Mobile / cell phone fraud
 Insider trading
Challenges
 Fast and accurate real-time detection
 Misclassification cost is very high

59.
Statistics: Unlocking the Power of Data Lock5
Classification Based Techniques
Main idea: build a classification model for normal (and anomalous (rare))
events based on labeled training data, and use it to classify each new
unseen event
Classification models must be able to handle skewed (imbalanced) class
distributions
Categories:
 Supervised classification techniques
 Require knowledge of both normal and anomaly class
 Build classifier to distinguish between normal and known anomalies
 Semi-supervised classification techniques
 Require knowledge of normal class only!
 Use modified classification model to learn the normal behavior and then detect any
deviations from normal behavior as anomalous

60.
Statistics: Unlocking the Power of Data Lock5
Classification Based Techniques
Advantages:
 Supervised classification techniques
 Models that can be easily understood
 High accuracy in detecting many kinds of known anomalies
 Semi-supervised classification techniques
 Models that can be easily understood
 Normal behavior can be accurately learned
Drawbacks:
 Supervised classification techniques
 Require both labels from both normal and anomaly class
 Cannot detect unknown and emerging anomalies
 Semi-supervised classification techniques
 Require labels from normal class
 Possible high false alarm rate - previously unseen (yet legitimate) data records
may be recognized as anomalies

61.
Statistics: Unlocking the Power of Data Lock5
Supervised Classification Techniques
Manipulating data records (oversampling /
undersampling / generating artificial examples)
Rule based techniques
Model based techniques
 Neural network based approaches
 Support Vector machines (SVM) based approaches
 Bayesian networks based approaches
Cost-sensitive classification techniques
Ensemble based algorithms (SMOTEBoost,
RareBoost

62.
Statistics: Unlocking the Power of Data Lock5
Semi-supervised Classification Techniques
Use modified classification model to learn the
normal behavior and then detect any deviations
from normal behavior as anomalous
Recent approaches:
 Neural network based approaches
 Support Vector machines (SVM) based approaches
 Markov model based approaches
 Rule-based approaches

63.
Statistics: Unlocking the Power of Data Lock5
Nearest Neighbor Based Techniques
Key assumption: normal points have close neighbors
while anomalies are located far from other points
General two-step approach
1. Compute neighborhood for each data record
2. Analyze the neighborhood to determine whether data
record is anomaly or not
Categories:
 Distance based methods
 Anomalies are data points most distant from other points
 Density based methods
 Anomalies are data points in low density regions

64.
Statistics: Unlocking the Power of Data Lock5
Clustering Based Techniques
Key assumption: normal data records belong to large and
dense clusters, while anomalies belong do not belong to any of
the clusters or form very small clusters
Categorization according to labels
 Semi-supervised – cluster normal data to create modes of normal
behavior. If a new instance does not belong to any of the clusters or it is
not close to any cluster, is anomaly
 Unsupervised – post-processing is needed after a clustering step to
determine the size of the clusters and the distance from the clusters is
required fro the point to be anomaly
Anomalies detected using clustering based methods can be:
 Data records that do not fit into any cluster (residuals from clustering)
 Small clusters
 Low density clusters or local anomalies (far from other points within the
same cluster)

65.
Statistics: Unlocking the Power of Data Lock5
Clustering Based Techniques
Advantages:
 No need to be supervised
 Easily adaptable to on-line / incremental mode suitable for
anomaly detection from temporal data
Drawbacks
 Computationally expensive
Using indexing structures (k-d tree, R* tree) may alleviate this
problem
 If normal points do not create any clusters the techniques
may fail
 In high dimensional spaces, data is sparse and distances
between any two data records may become quite similar.
Clustering algorithms may not give any meaningful clusters

66.
Statistics: Unlocking the Power of Data Lock5
Statistics Based Techniques
Data points are modeled using stochastic distribution 
points are determined to be outliers depending on their
relationship with this model
Advantage
 Utilize existing statistical modeling techniques to model various type
of distributions
Challenges
 With high dimensions, difficult to estimate distributions
 Parametric assumptions often do not hold for real data sets

67.
Statistics: Unlocking the Power of Data Lock5
Types of Statistical Techniques
Parametric Techniques
 Assume that the normal (and possibly anomalous) data is generated
from an underlying parametric distribution
 Learn the parameters from the normal sample
 Determine the likelihood of a test instance to be generated from this
distribution to detect anomalies
Non-parametric Techniques
 Do not assume any knowledge of parameters
 Use non-parametric techniques to learn a distribution – e.g. parzen
window estimation

68.
Statistics: Unlocking the Power of Data Lock5
Information Theory Based Techniques
Compute information content in data using information
theoretic measures, e.g., entropy, relative entropy, etc.
Key idea: Outliers significantly alter the information content
in a dataset
Approach: Detect data instances that significantly alter the
information content
 Require an information theoretic measure
Advantage
 Operate in an unsupervised mode
Challenges
 Require an information theoretic measure sensitive enough to detect
irregularity induced by very few outliers

69.
Statistics: Unlocking the Power of Data Lock5
Visualization Based Techniques
Use visualization tools to observe the data
Provide alternate views of data for manual
inspection
Anomalies are detected visually
Advantages
 Keeps a human in the loop
Disadvantages
 Works well for low dimensional data
 Can provide only aggregated or partial views for high
dimension data

70.
Statistics: Unlocking the Power of Data Lock5
Visual Data Mining*
Detecting Tele-
communication fraud
Display telephone call
patterns as a graph
Use colors to identify
fraudulent telephone
calls (anomalies)

71.
Statistics: Unlocking the Power of Data Lock5
Contextual Anomaly Detection
Detect context anomalies
General Approach
 Identify a context around a data instance (using a set of
contextual attributes)
 Determine if the data instance is anomalous w.r.t. the context
(using a set of behavioral attributes)
Assumption
 All normal instances within a context will be similar (in terms of
behavioral attributes), while the anomalies will be different

72.
Statistics: Unlocking the Power of Data Lock5
Contextual Attributes
Contextual attributes define a neighborhood
(context) for each instance
For example:
 Spatial Context
Latitude, Longitude
 Graph Context
Edges, Weights
 Sequential Context
Position, Time
 Profile Context
User demographics

73.
Statistics: Unlocking the Power of Data Lock5
Sequential Anomaly Detection
Detect anomalous sequences in a database of
sequences, or
Detect anomalous subsequence within a sequence
Data is presented as a set of symbolic sequences
 System call intrusion detection
 Proteomics
 Climate data

74.
Statistics: Unlocking the Power of Data Lock5
Motivation for On-line Anomaly Detection
Data in many rare events applications arrives continuously
at an enormous pace
There is a significant challenge to analyze such data
Examples of such rare events applications:
 Video analysis
 Network traffic monitoring
 Credit card fraudulent transactions

75.
Statistics: Unlocking the Power of Data Lock5
Sentiment Analysis for Finance
Sentiment analysis is an emerging area where structured and
unstructured data is analyzed to generate useful insights leading to
improved performances.
Information obtained from multiple sources including news wires, macro-
economic announcements, social media, micro blogs /twitter, online
(search) information such as Google trends and Wikipedia influence both
business intelligence and performance evaluation.
This sentiment data can help investors and finance professionals to
exploit the market and manage their risk exposure.
 Stock market prediction
 New product review
 Stock Trading
 Customer Brand Building

76.
Statistics: Unlocking the Power of Data Lock5
Sentiment Analysis in Finance

77.
Statistics: Unlocking the Power of Data Lock5

78.
Statistics: Unlocking the Power of Data Lock5
Thank You