1. 1/39
An Algorithm for Incremental
Unsupervised Learning and
Topology Representation
Shen Furao
Hasegawa Lab
Department of Computational
Intelligence and Systems Science
3. 3/39
Introduction
Clustering: Construct decision boundaries
based on unlabeled data.
Topology learning: find a topology
structure that closely reflects the topology
of the data distribution
Online incremental learning: Adapt to new
information without corrupting previously
learned information
4. 4/39
Vector Quantization
Targets
To minimize the average distortion through a
suitable choice of codewords
Application
Data compression, speech recognition
Separate the data set to Voronoi regions, find the
centroid of the Voronoi regions
LBG method (Linde, Buzo & Gray, 1980)
Dependence on initial starting conditions
Tendency to result in local minima
5. 5/39
Adaptive incremental LBG
(Shen & Hasegawa, 2005)
To solve the problem caused by poorly chosen
initial conditions
independent of initial conditions
With fixed number of codewords, to find a suitable
codebook to minimize the distortion error MQE.
It can work better than or same as ELBG (Patane &
Russo, 2001)
With fixed distortion error, to minimize the number
of codewords and find a suitable codebook.
Meaning: To get the same reconstruction quality for
different vector set, the codebook will have different size
and thus can save plenty of storage.
6. 6/39
Test Image
Lena (512*512*8) is
separated to 4*4 blocks. Such
blocks are the input vectors.
There are totally 16384
vectors.
Peak Signal to Noise Ratio
(PSNR) is used to evaluate the
resulting images after the
quantization process.
2552
PSNR 10 log10
1
N
i 1
( f (i ) g (i )) 2
N Lena (512*512*8)
7. 7/39
Improvement I:
Incrementally inserting codewords
The optimal
solution of k-
clustering
problem can
be reachable
from the (k-
1)-clustering
problem.
8. 8/39
Improvement II:
Distance measure function
Within cluster
distance must be
significantly less
than between
cluster distance.
l
d ( x, c) ( ( xi ci ) 2 ) p
i 1
p log10 q 1
9. 9/39
Improvement III:
Delete and insert codeword
Delete codeword
with lowest local
distortion error
Insert codeword
near the codeword
with highest local
distortion error
10. 10/39
Experiment 1
PSNR
Number of
codewords LBG (Linde Mk (Lee et ELBG(Pata
AILBG
et al.,1980) al., 1997) ne, 2001)
256 31.60 31.92 31.94 32.01
512 32.49 33.09 33.14 33.22
1024 33.37 34.42 34.59 34.71
Meaning: With the same number of codewords, proposed
method can get highest PSNR, i.e., with the same compression
ratio, proposed method can get best reconstruction quality.
11. 11/39
Experiment 2
Number of codewords
PSNR ELBG (Patane,
AILBG
2001)
31.94 256 244
33.14 512 488
34.59 1024 988
Meaning:
• With a predefined reconstruction quality, proposed method can
find a good codebook with reasonable number of codewords.
13. 13/39
Results of experiment 3
PSNR Number of codewords
(dB) Gray21 Lena Boat
28.0 9 22 54
30.0 12 76 199
33.0 15 454 1018
Meaning:
1. For different images, with the same PSNR, number of codewords will be different.
2. Proposed method can be used to set up an image database with same
reconstruction quality (PSNR)
14. 14/39
Unsupervised learning
Clustering
K-means (King, 1967), ELBG (Patane, 2001), Global k-means (Likas, 2003),
AILBG (Shen, 2005)
Determine the number of clusters k in advance
data sets consisting only of isotropic clusters
Single-link (Sneath, 1973), complete-link (King, 1967), CURE (Guha, 1998)
Computation overload, much memory space
Unsuitable for large data sets or online data
Topology Learning: Reflects topology of high-dimension data distribution
SOM (Kohonen, 1982): predetermined structure and size
CHL+NG (Martinetz, 1994): a priori decision about the network size
GNG (Fritzke, 1995): permanent increase in the number of nodes
Online Learning
GNG-U (Frutzke, 1998): destroy learned knowledge
LLCS (Hamker, 2001): supervised learning
15. 15/39
Self-organizing incremental neural
network (Shen & Hasegawa, 2005)
1. To process the on-line non-stationary data.
2. To do the unsupervised learning without any priori
condition such as:
• suitable number of nodes
• a good initial codebook
• how many classes there are
3. Report a suitable number of classes
4. Represent the topological structure of the input probability
density.
5. Separate the classes with some low-density overlaps
6. Detect the main structure of clusters polluted by noises
16. 16/39
The Proposed algorithm
First Layer Second Layer
Input Growing First Growing Second
pattern Network Output Network Output
Insert Delete
Classify
Node Node
17. 17/39
Algorithms
Insert new nodes
Criterion: nodes with high errors serve as a criterion to
insert a new node
error-radius is used to judge if the insert is successful
Delete nodes
Criterion: remove nodes in low probability density
regions
Realize: delete nodes with no or only one direct topology
neighbor
Classify
Criterion: all nodes linked with edges will be one cluster
18. 18/39
First-layer Second-layer
Input signals==
Initialize multiple of
Input signal Within-class
Insertion
Find winner Judge if insertion
and second winner is successful
Delete overlap and
Y Between-class noise nodes
Insertion
N N Input signals==
Connect winner multiple of LT
and second winner
Y
Update weight of First-layer Y
winner and neighbor
N
Output results
19. 19/39
Experiment
Environment
I II III IV V VI VII
A 1 0 1 0 0 0 0
B 0 1 0 1 0 0 0
C 0 0 1 0 0 1 0
D 0 0 0 1 1 0 0
E1 0 0 0 0 1 0 0
E2 0 0 0 0 0 1 0
Original Data Set E3 0 0 0 0 0 0 1
20. 20/39
Experiment:
Stationary environment
Original Data Set GNG (Fritzke, 1995)
21. 21/39
Experiment:
Stationary environment
Proposed method: first layer Proposed method: final results
33. 33/39
Application: Handwritten
character recognition
Optical Recognition of Handwritten Digits
database (optdigits) (UCI repository, 1996)
10 classes (handwritten digits) from a total of 43
people
30 contributed to the training set, 3823 samples
Different 13 to the test set, 1797 samples
Dimension of the samples is 64
Method:
Train: A separate SOINN to describe each class of data
Test: Classify an unknown data point according to
whichever model gives the best match (nearest
neighbor)
37. 37/39
Journal papers (2003~2005)
1. Shen Furao & Osamu Hasegawa, “An adaptive incremental LBG
for vector quantization,” Neural Networks, accepted.
2. Shen Furao & Osamu Hasegawa, “An incremental network for on-
line unsupervised classification and topology learning,” Neural
Networks, accepted.
3. Shen Furao & Osamu Hasegawa, Fractal image coding with
simulated annealing search, Journal of Advanced Computational
Intelligence and Intelligent Informatics, Vol.9, No.1, pp.80-88,
2005.
4. Shen Furao & Osamu Hasegawa, A fast no search fractal image
coding method, Signal Processing: Image Communication, vol.19,
pp.393-404, (2004)
5. Shen Furao & Osamu Hasegawa, A growing neural network for
online unsupervised learning, Journal of Advanced Computational
Intelligence and Intelligent Informatics, Vol.8, No.2, pp.121-129,
(2004)
38. 38/39
Refereed International
Conference (2003~2005)
1. Shen Furao, Youki Kamiya & Osamu Hasegawa, “An incremental neural network for online
supervised learning and topology representation,” 12th International Conference on Neural
Information Processing (ICONIP 2005), Taipei, Taiwan, October 30 - November 2, 2005, accepted.
2. Shen Furao & Osamu Hasegawa, “An incremental k-means clustering algorithm with adaptive
distance measure,” 12th International Conference on Neural Information Processing (ICONIP
2005), Taipei, Taiwan, October 30 - November 2, 2005, accepted.
3. Shen Furao & Osamu Hasegawa, “An on-line learning mechanism for unsupervised classification
and topology representation,” IEEE Computer Society International Conference on Computer
Vision and Pattern Recognition (CVPR 2005), San Diego, CA, USA, June 21-26, 2005.
4. Shen Furao & Osamu Hasegawa, “An incremental neural network for non-stationary unsupervised
learning,” 11th International Conference on Neural Information Processing (ICONIP 2004), Calcutta,
India, November 22-25, 2004.
5. Shen Furao & Osamu Hasegawa, “An effective fractal image coding method without search,” IEEE
International Conference on Image Processing (ICIP 2004), Singapore, October 24-27, 2004.
6. Youki Kamiya, Shen Furao & Osamu Hasegawa, “Non-stop learning : a new scheme for continuous
learning and recognition,” Joint 2nd SCIS and 5th ISIS, Keio University, Yokohama, Japan,
September 21-24, 2004.
7. Osamu Hasegawa & Shen Furao, “A self-structurizing neural network for online incremental
learning,” CD-ROM SICE Annual Conference in Sapporo, FAII-5-2, August 4-6, 2004.
8. Shen Furao & Osamu Hasegawa, “A self-organized growing network for on-line unsupervised
learning,” 2004 International Joint Conference on Neural Networks (IJCNN 2004), Budapest,
Hungary, CD-ROM ISBN 0-7803-8360-5, Vol.1, pp.11-16, 2004.
9. Shen Furao & Osamu Hasegawa, “A fast and less loss fractal image coding method using
simulated annealing,” 7th Joint Conference on Information Science (JCIS 2003), Cary, North
Carolina, USA, September 26-30, 2003.
