Monojit Basu, TechYugadi

1. MANIFOLDS IN SEMI-SUPERVISED LEARNING
Monojit Basu
Director, TechYugadi IT Solutions & Consulting, Bangalore
EXTENDED

2. 2
Outline
● Semi-supervized Learning and Graph-based Algorithms
● Data Distribution on Manifold and Multi-manifold
● Classification Algorithms with Manifold Regularization
● Implementation Hints
● Closing Remarks

3. 3
Outline
● Semi-supervized Learning and Graph-based Algorithms
● Data Distribution on Manifold and Multi-manifold
● Classification Algorithms with Manifold Regularization
● Implementation Hints
● Closing Remarks

4. 4
Semi-supervized Learning: Overview
● Training Samples consist of data with and without class label
● Images with and without captions
● Text with and without tags, ..
● Model is built with both labeled and unlabeled data
Prob(y|x) Prob(x)
● Smoothness Property: If two data points are close, their labels
should be similar
Label Data
Based on labeled samples Based on both labeled
and unlabeled samples

5. 5
Graph-based Algorithms For SSL
● There are many many ways of exploiting smoothness property
● A simplistic baseline approach is self-training (not graph-based)
● Graph-based Algorithms are particularly effective
● Label Propagation
● Random-Walk
● Min-Cut
● Density-based Distances
● Local and Global Consistency
● Using Graph Kernels, ..

6. 6
Label Propagation
● Generates a weighted graph where edges between similar
neighbours have higher weights (Zhu and Ghahramani, 2002)
● Defines a transition matrix:
● Tij = probability of node i ‘jumping’ into node j, that is, taking up j’s label
● Repeatedly multiplies the current label matrix with the transition
matrix (which itself gets updated)
● Until labels on all nodes stabilize (convergence)
● In effect labels propagate from labeled to unlabeled nodes
1
1
1
00
0 unlabeled

7. 7
Outline
● Semi-supervized Learning and Graph-based Algorithms
● Data Distribution on Manifold and Multi-manifold
● Classification Algorithms with Manifold Regularization
● Implementation Hints
● Closing Remarks

8. 8
Manifold Structures
● Data (nodes) are distributed over low and high density regions
● Two nodes that are geometrically close may not be similar
● Or equivalently, the geometry / distance measure should be redefined
● Euclidean distances and weights based on them may not work
● Such data is said to lie on a manifold
● Although not necessary, manifold structures are often
observed with high-dimensional data
● More complex scenario: data may not lie on a single manifold
● This is called multi-manifold structure

9. 9
Single Manifold Structures
SWISS ROLL TWO MOONS

10. 10
Multi-manifold Structures
$
Dollar Symbol
Surface Sphere

11. 11
Outline
● Semi-supervized Learning and Graph-based Algorithms
● Data Distribution on Manifold and Multi-manifold
● Classification Algorithms with Manifold Regularization
● Implementation Hints
● Closing Remarks

12. 12
Manifold Regularization
● This is the technical term for semi-supervized classification of
data distributed on a (single) manifold (Belkin et al., 2006)
● Key is to establish connectivity between similar nodes by
staying along a high-density region
● Mathematically it involves
● Computing a matrix L derived from the ordinary weight matrix W
● Taking the top n eigenvalues of L
● Computing an indicator function using the dot product of a data point
with the eigenvalues
● It is based on a theory known as Kernel Hilbert Spaces

13. 13
Maniford Regularization (Schematic)
DATA
W
L=D-W
Eigen(L)
dotxData Point >0
+ve
-ve
CLASS LABELS

14. 14
Multi-manifold Regularization
● This is the technical term for semi-supervized classification of
data distributed on a multi-manifold (Goldberg et al., 2009)
● Single manifold algorithm still starts with Euclidean distances,
but reformulates steps based on the derived matrix L
● Multi-manifold algorithm straight away changes distance
metrics
● It is based on Hellinger distances H, and
● A Mahalnabis k-nearest neighbor graph computed from H
● Complete algorithm is much longer, involving spectral
clustering and self-training on each cluster

15. 15
Multi-manifold Regularization (Schematic)
DATA
Σs
Sample Cov. Mat.
H
kNN graph
Spectral Clustering
Self-trained Clusters

16. 16
Multi-view Semi-supervised Learning
● Multi-view learning involves two or more independent
projections for each data point
● Classic Example: web-page classification using
● Bag of words
● Links to other web-pages
● Instead of representing data as (X, y) where y is class label, it
may be represented as (X1, X2, y), where Xi are views
● Somewhat related to multimodal learning (like video and
audio)

17. 17
Multi-view Manifold Regularization
● Can manifold regularization be extended to multi-view data
● Yes, algorithms exist, based on strong mathematical
foundations, like Sindhwani and Rosenberg, 2008
● There is actually a generic pattern for multi-view semi-
supervized learning, called co-training
● Sindhwani et al., extends co-training with an algorithm called
co-regularization
● It reduces the problem to a convex optimization to minimize a
loss function
● The total loss function depends on individual class predictors
for each view, and a couple of regularization hyperparameters

18. 18
Outline
● Semi-supervized Learning and Graph-based Algorithms
● Data Distribution on Manifold and Multi-manifold
● Classification Algorithms with Manifold Regularization
● Implementation Hints
● Closing Remarks

19. 19
Python Implementation
● An implementation of some of these algorithms in Python 3.x is
published on github:
https://github.com/techyugadi/manifold_ssl
● These algorithms offer an interface similar to scikit-learn
● There are some programs to generate synthetic data and also
use the MNIST handwritten digits data
● Note: scikit-learn as of now supports only label propagation
algorithm for semi-supervized learning
● R package has more algorithms but not maifold regularization
● This is early-access release, more algorithms to be published !

20. 20
Outline
● Semi-supervized Learning and Graph-based Algorithms
● Data Distribution on Manifold and Multi-manifold
● Classification Algorithms with Manifold Regularization
● Implementation Hints
● Closing Remarks

21. 21
Summary
● Manifold regularization is an improvement over the standard
label propagation algorithm for semi-supervised learning
● It may lead to better results when data is distributed over a
manifold or multi-manifold
● This class of algorithms cover a wide range of scenarios,
including multi-view datasets
● These algorithms can be implemented in Python using
common numpy and linear algebra packages (see github)

22. 22
References
● Zhu and Ghahramani, 2002: Learning from Labeled and
Unlabeled Data with Label Propagation
● Belkin, Niyogi and Sindhwani, 2006: Manifold Regularization:
A Geometric Framework for Learning from Labeled and
Unlabeled Examples
● Sindhwani and Rosenberg, 2008: An RKHS for Multi-View
Learning and Manifold Co-Regularization
● Goldberg, Zhu, Singh, Xu and Nowak, 2009: Multi-Manifold
Semi-Supervised Learning

23. 23
THANK YOU
monojit@techyugadi.com