Chapter 3

Orthogonal Functions
In previous chapters we saw how a function f (x) could be written in terms of a linear combination of sine functions. In this chapter we will give a general and more abstract denition of a vector space so that functions can be viewed as a type of vector. Expressing a function as a linear combination of basis functions is then the same as writing a vector as a linear combination of basis vectors. We will start in Sec. 3.1 by reviewing what we mean by a vector and how such a concept can be dened in a more general way. In Sec. 3.2 we dene formally the concept of a basis. The similarities and dierences between usual vectors in three dimensions and more general vectors, which can include functions, are summarized in Sec. 3.3.

3.1

Vectors

A simple picture of a vector is a quantity v characterized by both a length and a direction, which we can represent by an arrow. If we place the tail of the arrow at the origin, then the tip sits at a point (v1 , v2 , v3 ), as shown in Fig. 3.1. 1 The values vi are the components of the vector v in this particular coordinate system.

Figure 3.1:

A vector in a threedimensional space (see text).

Here we use the subscripts 1, 2, 3 instead of x, y, z because we are quickly going to generalize this picture to more than three dimensions.

31

32

Lecture Notes on Mathematical Methods

We can add two vectors, say, u and v, and the result is also a vector w = u + v. We can view this as placing the tail v at the tip of u, and then drawing the new vector w from the tail of u to the tip of v. In terms of components this simply

w 1 = u1 + v 1 , w 2 = u2 + v 2 , w 3 = u3 + v 3 , from which it is clear that vector addition is commutative, i.e., w = u + v = v + u.

(3.1) (3.2) (3.3)

We can multiply a vector by a scalar (here a real number) which changes the length but not the direction of the vector (except that one would regard the direction as being reversed if the scalar is negative). We can also dene unit vectors ei as having unit length and pointing in the direction indicated by the subscript. The unit vectors e1 , e2 , and e3 are indicated in Fig. 3.1. The vector v can then be written v = v1 e1 + v2 e2 + v3 e3 . (3.4)

Keep in mind that the ei are vectors; they have unit length and they point in the direction of the corresponding coordinate axes. The quantity v is also a vector, but its components, v1 , v2 and v3 , are scalars. We dene the dot product or scalar product of two vectors v and u as u v = u1 v 1 + u2 v 2 + u3 v 3 . (3.5)

The length (or norm) squared of a vector is given by the scalar product of a vector with itself:
2 2 2 v 2 = v v = v1 + v2 + v3 . (3.6) Here we will often use the notation v = v 2 for the norm of a vector and v (in boldface or with an arrow) to indicate the vector itself.

We can also show that the dot product has the property u v = uv cos , where is the opening angle between u and v. This is a very useful relation for vectors in three-dimensions, but when we generalise the concept of a vector space to higher dimensions it is the denition (3.5) that is more fundamental. If we have a vector v and we want to nd one of its components, say, vj for some value of j (1, 2 or 3) then we can take the scalar product of it with the corresponding unit vector ej :

v ej

= ( v 1 e1 + v 2 e2 + v 3 e3 ) ej = v 1 e1 e j + v 2 e 2 e j + v 3 e3 e j = vj . (3.7)

Orthogonal Functions

33

The last line follows because the unit vectors ei are all perpendicular (or orthogonal) to each other, so ei ej = 0, unless i = j , in which case we have ej ej = 1. So only one of the three terms survives, namely, the one with the index equal to j , and its coecient is vj .

3.1.1

Denition of a vector space

We can give a more general denition of a vector as any element of a set called a vector space, for which the following properties hold: 1. If we have two vectors, say, v and u, we can add them (in either order) and the result is always a new vector: w = u + v = v + u. 2. Vector addition is associative: (u + v) + w = u + (v + w). 3. There exists a null vector, 0, such that for any other vector v one has 0 + v = v. 4. For every vector v there exists a vector, which we can call v, such that v + (v) = 0. 5. We can multiply any vector v by a scalar a such that the result av is also a vector. Multiplication by scalars is associative: a(bv) = (ab)v. 6. Multiplication by a sum of scalars satises (a + b)v = av + bv. 7. Multiplication of a scalar and a sum of vectors satises a(u + v) = au + av. 8. There is a scalar, 1, such that 1v = v for all v. For our purposes the scalars can be taken in general to be complex numbers, and in many problems they will in fact be real numbers. With this more abstract denition, many things become vectors that one may not have thought of in that way before. Is a polynomial of order n a vector? To answer that question we need to see if the set of all polynomials of order n forms a vector space. The elements of this set can be written Pn (x) = a0 + a1 x + a2 x2 + + an xn . (3.8)

Suppose we have two nth order polynomials Pn (x) and Qn (x), in general with dierent coecients a0 , a1 , . . . , an but having the same functional form as Eq. (3.8). Then it is easy to see that the sum Pn (x) + Qn (x) is also a polynomial of order n. Note that an could be zero; this is still consistent with our denition of a Polynomial of order n. We can easily check that all of the remaining properties of a vector space hold: multiplication of a polynomial by a scalar results in a polynomial; we can add zero, multiply by one, nd the inverse Pn (x), etc. So a polynomial can be a vector! In contrast to this, the set of real numbers between 0 and 1 is not a vector space, since the sum of two elements may be outside the set.

34

Lecture Notes on Mathematical Methods

3.1.2

Inner product space

The vectors dened on the basis of the axioms in Sec. 3.1.1 still lack an important feature, namely, something corresponding to the dot product. We dene an inner product space as a vector space with, in addition, a rule that maps any two vectors u and v onto a scalar, which we can write as u v but in this course we will use the more frequently seen notation u, v (this can also be written u|v , which is the Diract notation used in quantum mechanics). In a general denition of an inner product space we require the following properties to hold for any vectors u, v, w and any complex scalar a: 1. u + v, w = u, w + v, w , 2. v, au = a v, u , 3. u, v = v, u , 4. v, v 0, where equality holds if and only if v is the null vector. In the third point above the star indicates the complex conjugate. If we are dealing with a real inner product space then this requirement becomes v , u = u, v . (3.9)

If we allow, however, that the scalars can in general be complex numbers, then the second and third requirements together imply av , u = a v , u . (3.10)

That is, the inner product is linear under multiplication of the second argument by a scalar, but conjugate linear under multiplication of the rst argument by a scalar. In many references, particularly in pure mathematics, the opposite convention is used, i.e., one takes av, u = a v, u and v, au = a v, u , and furthermore a bar is usually used instead of a star to denote complex conjugation (a instead of a ). The convention we use is more common in physics and applied mathematics. We have seen that certain functions such as polynomials can be be viewed as elements of a vector space. Suppose we have a set of complex valued functions of a real variable x dened in an interval [a, b] that form a vector space. We can dene an inner product of any two functions f and g as
b

f, g =
a

f (x)g (x)w(x) dx ,

(3.11)

where w(x) is called the weight function. It must be specied as part of the denition of the inner product. If the vectors are real valued functions and we have w = 1, which in this course will often be the case, then the inner product is simply

Orthogonal Functions

35

f, g =
a

f (x)g (x) dx .

(3.12)

One can easily verify that the specic denitions of an inner product, (3.11) and (3.12), are consistent with the axioms of an inner product listed above.

3.2

Basis vectors, basis functions

In our initial discussion of three-dimensional vectors we said we could write any vector v as a superposition of the unit vectors e1 , e2 and e3 . The coecients of the unit vectors are the components of v:
3

v=
i=1

vi ei .

(3.13)

The idea here is that any vector v can be written in this form, and if that holds then the set of unit vectors is said to span the vector space. Equivalently the unit vectors are said to form a complete set of basis vectors (or for short, a complete basis). If we only took, say, e1 and e2 as a basis then it would not be complete, since any vector with a nonzero component in the e3 direction cannot be written as a linear combination of e1 and e2 . When the denition of a vector is generalized to include functions, we would like to include the notion of a basis. That is, we would like to nd a complete set of basis functions 1 , 2 , . . . such that any function f (x) in the space can be written as a linear combination f (x) =
n

c n n ( x ) ,

(3.14)

where the sum is carried out over all of the basis functions (often an innite number). Here the basis functions n play the role of the basis vectors ei and the coecients cn correspond to the components vi . As an example, suppose the set of functions in our vector space is the set of all polynomials of arbitrary order in the interval [0, 1], i.e., functions of the form

P (x) =
n=0

an xn ,

0x1.

(3.15)

We can therefore take as the basis functions n ( x ) = x n . (3.16)

Then by construction we can express any member of the vector space (any polynomial) as a possibly innite sum of the n (x). The n span the space of polynomials and form a complete set of basis functions. The unit vectors ei also possessed another important property, however, namely, they were all orthogonal to each other. Using the Kronecker delta symbol we can express this as

36

Lecture Notes on Mathematical Methods

ei ej = ij .

(3.17)

In general, two vectors are said to be orthogonal if their inner product vanishes. For our basis functions n = xn , however, this is not the case. That is, if we form the inner product of two basis functions n and m we nd
1

n , m =

n (x)m (x) dx =

xn xm dx =

xn+m+1 n+m+1

=
0

1 . n+m+1

(3.18)

These particular basis functions thus do not satisfy the same orthogonality relation as do the unit vectors above. There is a general procedure for constructing a new set of basis functions, say, n with n = 1, 2, . . . as a linear combination of the n such that the n are orthogonal, i.e., n , m = 0 unless n = m. This is called the Gram-Schmidt process, and is described in many texts such as Refs. [1, 2]. In our study of dierential equations, however, we will nd that in many cases we can nd, without need for the Gram-Schmidt process, a set of orthogonal functions. We have already encountered such a set, namely, the functions in the interval 0 x : n (x) = sin(nx) n = 1, 2, . . . . (3.19)

When we looked at heat equation in Sec. 2.3 we already used the fact that these functions satisfy

sin(nx) sin(mx) dx =
0

nm . 2

(3.20)

Beyond being orthogonal, the unit vectors ei also have the property that they are of unit length, i.e., ei ei = 1. We can dene the norm squared of a vector as the inner product of the vector with itself. Sometimes we write the norm squared with the notation v
2

= v, v ,

(3.21)

and the norm of the vector is then just the square root of this. The norm of a function that is an element of a vector space is thus dened in the same way as we did in Eq. (3.6) for vectors in a three-dimensional space. If we dene basis functions n as in Eq. (3.19), then their norm squared is

n , n =

sin2 (nx) dx =

. 2

(3.22)

The basis vectors ei we used in the three dimensional case have a norm equal to unity. We can arrange for our basis functions to have this same property by scaling (multiplying) them by an appropriate factor. That is, if we redene the functions n as

Orthogonal Functions

37

n ( x ) = then these will satisfy

2 sin(nx)

(3.23)

n , m = nm .

(3.24)

Functions with this property are said to be orthonormal. That is, they are mutually orthogonal and they all have norms equal to one. We are still left with the question as to whether our set of basis functions n is complete, i.e., whether an arbitrary function f (x) dened in the same interval (here 0 x ) can be expressed as a linear combination of them as in Eq. (3.14). If we restrict ourselves for the moment to functions f (x) in the interval [0, ], then one can show that the the n given by Eq. (3.23) constitute a complete set. For a dierent range, e.g., [ , ], we will need to include cosine terms as well, which we will discuss in Chapter 4. The proof of completeness requires some care in dening what it means for the innite sum on the right-hand side of (3.14) to converge to f (x). Mathematicians are concerned with proving rigorously this sort of theorem and more details can be found in many texts such as Refs. [1, 2]. Physicists are usually more interested in knowing to what extent a partial sum that includes, say, the rst n terms is a good approximation to f (x). So in this course we simply claim that the result is true and we will make similar assertions about other sets of basis functions we will encounter later on. We will look at examples of partially summed series in Sec. 4.1.

3.3

Summary on vectors

Finally in this section we recap the similarities and dierences between the usual vectors we talk about in three dimension and the more general vectors which can include functions. In both cases, any vector in the space can be expressed as a linear combination of basis vectors. For three dimensional vectors we can write
3

v=
i=1

v i ei

(3.25)

and for functions we have f (x) =

c n n ( x ) .

(3.26)

If, for example, the vector space is dened to consist of polynomials of order n, then there are n + 1 basis functions, which we could take as x0 , x1 , . . . , xn . The sum in Eq. (3.26) therefore contains n + 1 terms and we say the dimension of the vector space is n + 1. That is, the components v1 , v2 and v3 of the three-dimensional vector (3.25) can be viewed as a point in a three-dimensional space. The components (coecients) cn in Eq. (3.26) can in the same way be regarded as a point in an n + 1 dimensional space.

38

Lecture Notes on Mathematical Methods

If the vector space contains arbitrary functions rather than polynomials of a xed order, then the sum in Eq. (3.26) will contain an innite number of terms. So in general a function can be regarded as a vector of innite dimension. To specify the set of coecients that determine the function we need to give a point in an innite dimensional space. The dot product of two vectors u and v is given by

uv =

i=1

u i ei

3 j =1

v j ej =

3 i,j =1

u i v j ei ej =

ui vj ij =
i,j =1 i=1

ui v i .

(3.27)

If we have two (in general complex) functions f and g that can be expanded with coecients an and bn , respectively, in terms of a complete set of orthonormal basis functions n , then the inner product of f and g with respect to the weight function w is

f, g

f (x)g (x)w(x) dx =
n

a n n ( x )
m

bm m (x) w(x) dx

=
n,m

a n bm

n (x)m (x)w (x) dx =

a n bm n , m (3.28)

=
n,m

a n bm nm =
n

a n bn .

If the regions of integration or ranges of sums are not indicated explicitly then it is implied that they are carried out over all allowed values. If the functions are real (i.e., not complex) then the complex conjugation above can be dropped. In many examples the weight function w is equal to unity and in that case it is not explicitly written. The length (norm) squared of a vector in three dimensions is given by the dot product of the vector with itself:
3 2 2 vi . i=1

=vv =

(3.29)

The norm squared of a function is given by the inner product of the function with itself:

= =

f, f =

|f (x)| dx =

c n n ( x )
n m

cm m (x) w(x) dx (3.30)

c n c m n , m =
n,m n,m

c n cm nm =
n

|cn |2 .

Given a vector v we can nd its j th component by taking the dot product of v with the j th unit vector:

Orthogonal Functions

39

v ej =

i=1

v i ei ej =

i=1

vi ei ej =

vi ij = vj .
i=1

(3.31)

Given a function f we can nd the nth coecient cn in its basis function expansion by taking the inner product of n with f ,

n , f = so that cn is

n ,
m

c m m

=
m

c m n , m =
m

cm n , n nm = n , n cn ,

(3.32)

cn =

n , f . n , n

(3.33)

If the basis functions have unit norm, i.e., n , n = 1, then Eq. (3.33) becomes simply c n = n , f . (3.34)

Note that if the cn can be complex, then the order of the arguments is important, and reversing them will give the complex conjugate of the coecient: c n = f, n . n , n (3.35)

40

Lecture Notes on Mathematical Methods