1. NOTES ON THE LOW-RANK
MATRIX APPROXIMATION
OF KERNEL MATRICES
Hiroshi Tsukahara
Denso IT Laboratory, Inc.
Aug. 23 (Fri) 2013

2. KERNEL METHOD
 Supervised Learning Problem
 Solving in Reproducing Kernel Hilbert Spaces
( ){ }niYXyxD iin ,,2,1, =×∈= )(s.t.:Find ii xfyYXf =→
2
1 2
))(,(
1
min f
n
xfyl
n
ii
n
i
Ff
λ
+∑=
∈
( )∑=
=→
n
i
ii xfFX
1
s.t.:Assuming ϕαϕ
n
RinonOptimizati
(1)
ill-defined problem!!
cf. representer theorem

3.  Kernel method
 If the loss function is given by
the explicit form of is not necessary but their inner
products:
 Define the mapping implicitly by a kernel function:
),(:)( ⋅= xkxϕ
( )2
)(
2
1
))(,( xfyxfyl −=
ϕ
),(:)(),( xxkxx ′=′ϕϕ
ϕ
RHS is called as a kernel function

4.  Solution can formally be written as:
 However, the complexity for computing the
solution is very high:
T
njiijini yyyyxxkyIK ),,,(and),(Kwhere])[( 21
1
==+= −
λα
)( 3
nO

5. LOW-RANK APPROXIMATION
 Low-rank approximation of kernel matrices
 Their rank is usually very low comparing to n.
 Making use of this property, assume that the kernel
matrix can be written as
 Then, the complexity of calculating the solution can
be reduced considerably, due to the formula:
( )[ ]T
r
T
nn
T
RIRRRIIRR
11 1
)(
−−
++=+ λ
λ
λ
nrrnRRRK T
<<×≈ hmatrix witiswhere
O(r2
n)

6.  Rough sketch for the derivation of the formula
1
)( −
+ n
T
IRR λ
( )[ ].
1
,
1
,
1
,
1
,
1
1
1
2
32
T
r
T
n
T
T
rn
T
TT
rn
TTT
n
T
n
RIRRRI
R
RR
I
R
I
R
RRRR
I
R
I
RRRRRR
I
RR
I
−
−
+−=














+−=
















+





+





−−=








+





−





+−=






+=
λ
λ
λλλ
λλλλ
λλλλ
λλ



7.  There are several algorithms for deriving the
low-rank approximation:
 Nystrom approximation
 Incomplete Cholesky decompositon