Reproducing Kernel Hilbert Spaces

9.520 Class 03, 15 February 2006

Andrea Caponnetto

 About this class

Goal To introduce a particularly useful family of hypoth-

esis spaces called Reproducing Kernel Hilbert Spaces
(RKHS) and to derive the general solution of Tikhonov
regularization in RKHS.
 Here is a graphical example for

generalization: given a certain number of

samples...

f(x)

x
suppose this is the “true” solution...

f(x)

x
... but suppose ERM gives this solution!

f(x)

x
 Regularization

The basic idea of regularization (originally introduced in-

dependently of the learning problem) is to restore well-

posedness of ERM by constraining the hypothesis space

H. The direct way – minimize the empirical error subject

to f in a ball in an appropriate normed functional space
H – is called Ivanov regularization. The indirect way is
Tikhonov regularization (which is not ERM).
 Ivanov regularization over normed spaces

ERM finds the function in H which minimizes

n
1 �
V (f (xi), yi)
n i=1
which in general – for arbitrary hypothesis space H – is
ill-posed. Ivanov regularizes by finding the function that
minimizes
n
1 �
V (f (xi), yi)
n i=1
while satisfying

�f �2
H ≤ A,
with � · �, the norm in the normed function space H

 Function space

A function space is a space made of functions. Each

function in the space can be thought of as a point. Ex-
amples:

1. C[a, b], the set of all real-valued continuous functions

in the interval [a, b];

2. L1[a, b], the set of all real-valued functions whose ab-

solute value is integrable in the interval [a, b];

3. L2[a, b], the set of all real-valued functions square inte-

grable in the interval [a, b]
 Normed space

A normed space is a linear (vector) space N in which a

norm is defined. A nonnegative function � · � is a norm iff
∀f, g ∈ N and α ∈ IR

1. �f � ≥ 0 and �f � = 0 iff f = 0;

2. �f + g� ≤ �f � + �g�;

3. �αf � = |α| �f �.

Note, if all conditions are satisfied except �f � = 0 iff f = 0

then the space has a seminorm instead of a norm.
 Examples

1. A norm in C[a, b] can be established by defining

�f � = max |f (t)|.
a≤t≤b

2. A norm in L1[a, b] can be established by defining

� b
�f � = |f (t)|dt.
a

3. A norm in L2[a, b] can be established by defining

�� �1/2
b
�f � = f 2(t)dt .
a
 From Ivanov to Tikhonov regularization

Alternatively, by the Lagrange multipler’s technique, Tikhonov

regularization minimizes over the whole normed function
space H, for a fixed positive parameter λ, the regularized
functional
n
1 �
V (f (xi), yi) + λ�f �2
H. (1)
n i=1

In practice, the normed function space H that we will con-

sider, is a Reproducing Kernel Hilbert Space (RKHS).
 Lagrange multiplier’s technique

Lagrange multiplier’s technique allows the reduction of the

constrained minimization problem

Minimize I(x)

subject to Φ(x) ≤ A (for some A)

to the unconstrained minimization problem

Minimize J(x) = I(x) + λΦ(x) (for some λ ≥ 0)

 Hilbert space

A Hilbert space is a normed space whose norm is induced

by a dot product �f, g� by the relation
�
�f � = �f, f �.
A Hilbert space must also be complete and separable.

• Hilbert spaces generalize the finite Euclidean spaces IRd,

and are generally infinite dimensional.

• Separability implies that Hilbert spaces have countable

orthonormal bases.
 Examples

• Euclidean d-space. The set of d-tuples x = (x1, ..., xd)

endowed with the dot product
d
�
�x, y� = x i yi .
i=1
The corresponding norm is
�
d
�
��
�x� = � x2
i.
�

i=1
The vectors

e1 = (1, 0, . . . , 0), e2 = (0, 1, . . . , 0), . . . , ed = (0, 0, . . . , 1)

form an orthonormal basis, that is �ei, ej � = δij .
 Examples (cont.)

• The function space L2[a, b] consisting of square integrable

functions. The norm is induced by the dot product
� b
�f, g� = f (t)g(t)dt.
a
It can be proved that this space is complete and separable.

An important example of orthogonal basis in this space is

the following set of functions
2πnt 2πnt
1, cos , sin (n = 1, 2, ...).
b−a b−a

• C[a, b] and L1[a, b] are not Hilbert spaces.

 Evaluation functionals

A linear evaluation functional over the Hilbert space of

functions H is a linear functional Ft : H → IR that evaluates
each function in the space at the point t, or

Ft[f ] = f (t)

The functional is bounded if there exists a M s.t.

|Ft[f ]| = |f (t)| ≤ M �f �H ∀f ∈ H
where � · �H is the norm in the Hilbert space of functions.

• we don’t like the space L2[a, b] because the its evaluation

functionals are unbounded.
 Evaluation functionals in Hilbert space

The evaluation functional is not bounded in the familiar

Hilbert space L2([0, 1]), no such M exists and in fact ele-
ments of L2([0, 1]) are not even defined pointwise.

4
f(x)

0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

 RKHS

Definition A (real) RKHS is a Hilbert space of real-valued

functions on a domain X (closed bounded subset of IRd)
with the property that for each t ∈ X the evaluation func-
tional Ft is a bounded linear functional.
 Reproducing kernel (rk)

If H is a RKHS, then for each t ∈ X there exists, by the

Riesz representation theorem a function Kt of H (called
representer of evaluation) with the property – called by
Aronszajn – the reproducing property

Ft[f ] = �Kt, f �K = f (t).

Since Kt is a function in H, by the reproducing property,
for each x ∈ X
Kt(x) = �Kt, Kx�K
The reproducing kernel (rk) of H is

K(t, x) := Kt(x)
.
 Positive definite kernels

Let X be some set, for example a subset of IRd or IRd itself.

A kernel is a symmetric function K : X × X → IR.

Definition

A kernel K(t, s) is positive definite (pd) if

n
�
cicj K(ti, tj ) ≥ 0
i,j=1
for any n ∈ IN and choice of t1, ..., tn ∈ X and c1, ..., cn ∈ IR.
 RKHS and kernels

The following theorem relates pd kernels and RKHS.

Theorem

a) For every RKHS the rk is a positive definite kernel on.

b) Conversely for every positive definite kernel K on

X × X there is a unique RKHS on X with K as its rk

 Sketch of proof

a) We must prove that the rk K(t, x) = �Kt, Kx�K is sym-

metric and pd.

• Symmetry follows from the symmetry property of dot

products
�Kt, Kx�K = �Kx, Kt�K

• K is pd because
n n
cj Ktj ||2
� � �
cicj K(ti, tj ) = cicj �Kti , Ktj �K = || K ≥ 0.
i,j=1 i,j=1
 Sketch of proof (cont.)

b) Conversely, given K one can construct the RKHS H as

the completion of the space of functions spanned by the
set {Kx|x ∈ X} with a inner product defined as follows.

The dot product of two functions f and g in span{Kx|x ∈

X}
s
�
f (x) = αiKxi (x)
i=1
s�
�
g(x) = βiKx� (x)
i
i=1
is by definition
s�
s �
αiβj K(xi, x�j ).
�
�f, g�K =
i=1 j=1
 Examples of pd kernels

Very common examples of symmetric pd kernels are

• Linear kernel
K(x, x�) = x · x�
• Gaussian kernel
�x−x� �2
K(x, x�) = e σ2 , σ>0
• Polynomial kernel

K(x, x�) = (x · x� + 1)d, d ∈ IN

For specific applications, designing an effective kernel is a

challenging problem.
 Historical Remarks

RKHS were explicitly introduced in learning theory by Girosi

(1997). Poggio and Girosi (1989) introduced Tikhonov
regularization in learning theory and worked with RKHS
only implicitly, because they dealt mainly with hypothesis
spaces on unbounded domains, which we will not discuss
here. Of course, RKHS were used much earlier in approx-
imation theory (eg Wahba, 1990...) and computer vision
(eg Bertero, Torre, Poggio, 1988...).
 Back to Tikhonov Regularization

The algorithms (Regularization Networks) that we want to

study are defined by an optimization problem over RKHS,

n
1 �
fS = arg min V (f (xi), yi) + λ�f �2
K
f ∈H n i=1

where the regularization parameter λ is a positive number,

H is the RKHS as defined by the pd kernel K(·, ·), and
V (·, ·) is a loss function.
 Norms in RKHS, Complexity, and

Smoothness

We measure the complexity of a hypothesis space using

the the RKHS norm, �f �K .

The next result illustrates how bounding the RKHS norm

corresponds to enforcing some kind of “simplicity” or smooth-
ness of the functions.
 Regularity of functions in RKHS

Functions over X, in the RKHS H induced by a pd kernel

K, fulfill a Lipschitz-like condition, with Lipschitz constant
given by the norm �f �K .

In fact, by the Cauchy-Schwartz inequality, we get ∀x, x� ∈

X

|f (x) − f (x�)| = |�f, Kx − Kx� �K | ≤ �f �K d(x, x�),

with the distance d over X, defined by

d2(x, x�) = K(x, x) − 2K(x, x�) + K(x�, x�).

 A linear example

Our function space is 1-dimensional lines

f (x) = w x and K(x, xi ) ≡ x xi .
For this kernel
d2 (x, x� ) = K(x, x) − 2K(x, x� ) + K(x� , x� ) = |x − x� |2,
and using the RKHS norm
�f �2K = �f, f �K = �Kw , Kw �K = K(w, w) = w2
so our measure of complexity is the slope of the line.

We want to separate two classes using lines and see how the magnitude
of the slope corresponds to a measure of complexity.

We will look at three examples and see that each example requires
more complicated functions, functions with greater slopes, to separate
the positive examples from negative examples.
 A linear example (cont.)

here are three datasets: a linear function should be used to

separate the classes. Notice that as the class distinction
becomes finer, a larger slope is required to separate the
classes.
2 2

1.5 1.5

1 1

0.5 0.5

f(X)
f(x)

0 0

−0.5 −0.5

−1 −1

−1.5 −1.5

−2 −2
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2

x x

1.5

0.5
f(x)

−0.5

−1

−1.5

−2
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

 Again Tikhonov Regularization

The algorithms (Regularization Networks) that we want to

study are defined by an optimization problem over RKHS,

n
1 �
fS = arg min V (f (xi), yi) + λ�f �2
K
f ∈H n i=1

where the regularization parameter λ is a positive number,

H is the RKHS as defined by the pd kernel K(·, ·), and
V (·, ·) is a loss function.
 Common loss functions

The following two important learning techniques are im-

plemented by different choices for the loss function V (·, ·)

• Regularized least squares (RLS)

V = (y − f (x))2
• Support vector machines for classification (SVMC)

V = |1 − yf (x)|+
where
(k)+ ≡ max(k, 0).
 The Square Loss

For regression, a natural choice of loss function is the

square loss V (f (x), y) = (f (x) − y)2.

5
L2 loss

0
−3 −2 −1 0 1 2 3
y−f(x)
 The Hinge Loss

3.5

2.5
Hinge Loss

1.5

0.5

−3 −2 −1 0 1 2
3
y * f(x)
 Existence and uniqueness of minimum

If the positive loss function V (·, ·) is convex with respect

to its first entry, the functional
n
1 �
Φ[f ] = V (f (xi), yi) + λ�f �2
K
n i=1
is strictly convex and coercive, hence it has exactly one
local (global) minimum.

Both the squared loss and the hinge loss are convex.

On the contrary the 0-1 loss

V = Θ(−f (x)y),
where Θ(·) is the Heaviside step function, is not convex.

 The Representer Theorem

The minimizer over the RKHS H, fS , of the regularized

empirical functional

IS [f ] + λ�f �2
K,
can be represented by the expression

n
�
fS (x) = ciK(xi, x),
i=1
for some n-tuple (c1, . . . , cn) ∈ IRn.

Hence, minimizing over the (possibly infinite dimensional)

Hilbert space, boils down to minimizing over IRn.
 Sketch of proof

Define the linear subspace of H,

H0 = span({Kxi }i=1,...,n)

⊥ be the linear subspace of H,

Let H0
⊥ = {f ∈ H|f (x ) = 0, i = 1, . . . , n}.
H0 i

⊥
From the reproducing property of H, ∀f ∈ H0
� � �
�f, ciKxi �K = ci�f, Kxi �K = cif (xi) = 0.
i i i

⊥ is the orthogonal complement of H .

H0 0
 Sketch of proof (cont.)

Every f ∈ H can be uniquely decomposed in components

along and perpendicular to H0: f = f0 + f0⊥.

Since by orthogonality

�f0 + f0⊥�2 = �f0�2 + �f0⊥�2,

and by the reproducing property

IS [f0 + f0⊥] = IS [f0],

then

IS [f0] + λ�f0�2
K ≤ IS [f 0 + f ⊥ ] + λ�f + f ⊥ �2 .
0 0 0 K

Hence the minimum fSλ = f0 must belong to the linear

space H0.
 Applying the Representer Theorem

Using the representer theorem the minimization problem

over H
min IS [f ] + λ�f �2
K
f ∈H
can be easily reduced to a minimization problem over IRn

min g(c)

c∈IRn
for a suitable function g(·).

If the loss function is convex, then g is convex, and finding

the minimum reduces to computing the subgradient of g.

In particular, if the loss function is differentiable (eg. squared

loss), we just have to compute (and set to zero) the gra-
dient of g.
Tikhonov Regularization for RLS and SVMs

In the next two classes we will study Tikhonov regulariza-

tion with different loss functions for both regression and
classification. We will start with the square loss and then
consider SVM loss functions.