Linear Algebra

Aakash Jog

Contents
I

General Information

1 Contact Information

2 Grades

II

Fields

1 Definition
1.1 Examples of Fields . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Examples of Non-fields (Rings) . . . . . . . . . . . . . . . . .

8
8
9

2 Examples

III

Matrices

10

1 Definition

10

This work is licensed under the Creative Commons Attribution-NonCommercialShareAlike 4.0 International License. To view a copy of this license, visit http:
//creativecommons.org/licenses/by-nc-sa/4.0/.

2 Addition of Matrices
10
2.0.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 Multiplication of a matrix by a scalar

10

4 Multiplication of matrices

11

5 Zero Divisor

11

6 Theorem (Good properties of matrix multiplication)

12

7 Square Matrices
7.1 Diagonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . .
7.1.1 Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2 Upper-triangular Matrices . . . . . . . . . . . . . . . . . . . .
7.3 Lower-triangular Matrices . . . . . . . . . . . . . . . . . . . .
7.4 Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.1 Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.5 Identity Matrix . . . . . . . . . . . . . . . . . . . . . . . . . .
7.6 Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.6.1 Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.7 Inverse of Matrix . . . . . . . . . . . . . . . . . . . . . . . . .
7.7.1 If AB = In and CA = In , then B = C . . . . . . . . .
7.7.2 Inverse of a Matrix . . . . . . . . . . . . . . . . . . . .
7.7.3 If AB = I, then BA = I. . . . . . . . . . . . . . . . . .
7.7.4 If A is invertible, then A cannot be a zero divisor. . . .
7.7.5 If A and B are invertible, then A + B may or may not
be invertible. . . . . . . . . . . . . . . . . . . . . . . .
7.7.6 If A and B are invertible, then AB must be invertible.

13
13
13
13
13
14
14
14
14
14
15
15
15
16
16
16
17

8 Transpose of a Matrix
17
t
8.1 Properties of A . . . . . . . . . . . . . . . . . . . . . . . . . . 17
9 Adjoint Matrix
17
9.0.1 Properties of Adjoint Matrices . . . . . . . . . . . . . . 18
10 Row Operations on Matrices
10.1 Elementary Row Operations . . . . . . . .
10.2 Theorems . . . . . . . . . . . . . . . . . .
10.2.1 EI A = the matrix obtained from A
row operation I . . . . . . . . . . .

18
. . . . . . . . . . . 18
. . . . . . . . . . . 18
by an elementary
. . . . . . . . . . . 18

10.2.2 EII A = the matrix obtained from A by an elementary

row operation II . . . . . . . . . . . . . . . . . . . . .
10.2.3 EIII A = the matrix obtained from A by an elementary
row operation III . . . . . . . . . . . . . . . . . . . .
10.2.4 All elementary matrices are invertible, moreover, the
inverses of EI , EII , EIII are also elementary matrices of
the same type. . . . . . . . . . . . . . . . . . . . . .
10.3 Row-equivalent of a Matrix . . . . . . . . . . . . . . . . . .

. 19
. 19

. 20
. 22

11 Row Echelon Form of a Matrix

22
11.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
11.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
12 Row Rank of a Matrix
13 Gauss Theorem
13.1 Elimination Algorithm .
13.1.1 Example . . . . .
13.2 Row Spaces of Matrices .
13.3 Column Equivalence . .

IV

23

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

Linear Systems

23
23
24
24
25

27

1 Definition

27

2 Equivalent Systems

27

3 Solution of a System of Equations

28

4 Homogeneous Systems
4.1 Definition . . . . . . . . . . . . . . . . . . . . . .
4.2 Solutions of Homogeneous Systems . . . . . . . .
4.2.1 Example . . . . . . . . . . . . . . . . . . .
4.2.2 General Solution . . . . . . . . . . . . . .
4.3 Properties . . . . . . . . . . . . . . . . . . . . . .
4.3.1 For a homogeneous system Ax = 0, if c
solutions, then c + d is also a solution. . .
4.3.2 For a homogeneous system Ax = 0, if c is
and F, then, c is a solution too. . . .
4.4 Fundamental Solutions . . . . . . . . . . . . . . .

. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
and d are
. . . . . .
a solution
. . . . . .
. . . . . .

.
.
.
.
.

29
29
29
30
31
31

. 31
. 31
. 32

4.4.1

4.5

Theorem: Any solution d of the system Ax = O can be

obtained from the basic solutions v1 , . . . , vt as a linear
combination of the basic solutions, d = 1 v1 + . . . t vt . 32
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5 Non-Homogeneous Systems
5.1 Definition . . . . . . . . . . . . . . . . .
5.2 Solutions of Non-Homogeneous Systems .
5.2.1 Case I: re = r . . . . . . . . . . .
5.2.2 Case II: re > r . . . . . . . . . . .
5.3 General Solution . . . . . . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

Vector Spaces

33
33
33
33
33
34

35

1 Definition
1.1 Examples . . . . . . . . . . . . . .
1.1.1 Geometric Vectors in Plane
1.1.2 Arithmetic Vector Space . .
1.1.3 . . . . . . . . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

35
35
35
35
36

2 Properties
36
2.0.1 Proof of 1 . . . . . . . . . . . . . . . . . . . . . . . . . 36
3 Subspaces
36
3.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Operations on Subspaces . . . . . . . . . . . . . . . . . . . . . 38
4 Spans

38

5 Linear Dependence
5.1 Properties of Linearly Dependent and
5.2 Changing a Basis . . . . . . . . . . .
5.3 Representation of Vectors in a Basis .
5.3.1 Properties of Representations

.
.
.
.

41
42
46
49
49

.
.
.
.
.

50
50
50
53
53
53

Independent Sets
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .

6 Determinants
6.1 Definition . . . . . . . . . . . . . . . . . . . . .
6.2 Properties . . . . . . . . . . . . . . . . . . . . .
6.3 Practical Methods for Computing Determinants
6.4 Expansion along a row/ column . . . . . . . . .
6.5 Determinant Rank . . . . . . . . . . . . . . . .
4

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

7 Linear Maps
7.1 Definition . . . . . . . . . .
7.2 Properties . . . . . . . . . .
7.3 Matrix of a Linear Map . .
7.4 Change of Bases . . . . . . .
7.5 Operations on Linear Maps
7.6 Kernel and Image . . . . . .
7.6.1 Dimensions of Kernel

VI

. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
and Image .

Linear Operators

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

54
54
54
54
56
57
58
60

62

1 Definition

62

2 Similar Matrices
62
2.1 Properties of Similar Matrices . . . . . . . . . . . . . . . . . . 63
3 Diagonalization

63

4 Eigenvalues and Eigenvectors

63

5 Characteristic Polynomial
65
5.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

VII

Inner Product Spaces

1 Definition

71
71

2 Computation of Inner Products

72
2.1 Change of Basis . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3 Norms
75
3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4 Orthogonality
75
4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5 Orthogonal and Orthonormal Bases

76

6 Unitary Matrices

78
5

7 Projections
7.1 Definition . . .
7.2 Properties . . .
7.3 Gram - Schmidt
7.4 Inequalities . .

. . . . .
. . . . .
Process
. . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

80
80
82
82
84

8 Angle

87

9 Triangle Inequality

87

10 Orthogonal Decomposition

87

11 Distance

89

12 Adjoint Map
90
12.1 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
12.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
13 Special Linear Operators

93

Part I

General Information
1

Contact Information

Prof. Boris Kunyavskii

[email protected]

Grades

Final Exam: 80%

Midterm Exam: 10%
Homework: 10%
Passing Criteria: 60%

Part II

Fields
1

Definition

Definition 1 (Field). The set F is a field if there are operations +,  satisfying

the following properties:
(A1) a, b F; a + b = b + a
(A2) a, b F; (a + b) + c = a + (b + c)
(A3) There is an element 0 F s.t. a + 0 = 0 + a = a
(A4) a F, b F s.t. a + b = 0
(M1) a, b F, a b = b a
(M2) a, b F, (a b) c = a (b c)
(M3) There is an element 1 F s.t. a 1 = 1 a = a(1 6= 0)
(M4) a F, (a 6= 0), b F s.t. a b = 1
(AM) (a + b) c = (a c) + (b c)
If F is a field, one can define subtraction and division as follows.
a b=a
+ (b)
a
1
=a

b
b

1.1

Examples of Fields

1. R
2. C
3. Fp

1.2

Examples of Non-fields (Rings)

1. Z, as M4 is not satisfied.
If we define F2 = 0, 1; 0 + 0 = 0; 0 + 1 = 1 + 0 = 1; then, necessarily,
1 + 1 = 0, otherwise, 1 will have no additive inverse.

Examples

Example 1. Let p be a prime number.

Fp is defined as follows.
m Z, m = a  p + m
The operations + and  are defined as
a + b = (a + b)
a  b = (a  b)
1. Fp is a field.
2. If F is a set of q elements, we can define on F a structure of a field iff
q = pt , where p is prime, t 1.
Example 2. For a field of 4 elements {0, 1 , }, the addition and multiplication tables are as follows.
+ 0
0 0
1 1

1
1
0

0
0

1
1

Part III

Matrices
1

Definition

Definition 2 (Matrix). Let F be a field, m, n 1.

Then, A(m n) is a table consisting of m rows and n columns, filled by
elements of F.

a
11
a21

A=
...

a12
a22
..
.

...
...

am1 am2 . . .

a1n

a2n

..
.
amn

Addition of Matrices

Definition 3 (Addition of matrices). Let A, B be m n matrices over F.

Then, C = A + B is defined as follows.
cij = aij + bij
2.0.1

Properties

1. A + B = B + A, A, B s.t. the sum is defined

2. (A + B) + C = A + (B + C), A, B, C s.t. the sums are defined
3. There is a matrix O, s.t. A + O = O + A = A
4. For any A, B s.t. B = A

Multiplication of a matrix by a scalar

Definition 4 (Multiplication of a matrix by a scalar). Let A be a m n

matrix over F. Let F be a scalar. Then, C = A is defined as follows.
cij = aij
10

Multiplication of matrices

Definition 5 (Multiplication of matrices). Let A be a m n matrix over F.

Let B be a n p matrix over F.
Then, C = AB is defined as follows.
cik =

n
X

aij bjk

j=1

Example 3. For matrices A, B, of same size, is AB = BA?

!

0 1
Solution. A =
,B =
0 0
!
0 0
0
AB =
, BA =
0 0
0
AB 6= BA

1 0
0 0
1
0

Remark 1. A 6= O, B 6= O, but AB = O.

Zero Divisor

Definition 6 (Zero divisor). We say that a square matrix A 6= O is a zero

divisor if either there is a square matrix B s.t. AB = O, or there is a square
matrix C, s.t. CA = O.
Remark 2. OB = CO = O.
Remark 3. AC = BC ; A = B. In general, we cannot cancel matrices on
either side of an equation.
!

0 1
1 0
A=
,B =
,C = O
0 0
0 0
AB = CB = O & B 6= O
But, we cannot cancel B, as A 6= C.

11

Theorem (Good properties of matrix multiplication)

Theorem 1.
(AB)C = A(BC)
A(B + C) = AB + AC
(A + B)C = AC + BC
(A) = (AB)

(1.1)
(1.2)
(1.3)
(1.4)

Proof. Denote AB = D, BC = G, (AB)C = F, A(BC) = H

We need to prove F = H
Let the dimensions of the matrices be as follows.
Amn , Bnp , Cpq
Fmq , Hmq

dik =

aij bjk

gjl =

bjk bkl

fil =

dik ckl =

hil =

X
j

aij gjl =

X X

aij bjk)ckl =

aij (

XX
k

bjkckl ) =

XX
k

fil = hil
F =H

12

aij bjk ckl

aij bjk ckl

Square Matrices

Let A be a square matrix of size n n, n 1

7.1

Diagonal Matrices

Definition 7 (Diagonal matrix). We say that A is a diagonal matrix if

aij = 0, whenever i 6= j.
Theorem 2. Let A and B be diagonal n n matrices.
arr = r , brr = r
Then, AB = BA = C, C is a diagonal matrix with crr = arr brr .
7.1.1
aij =
bij =
cik =

Proof

0, i

6= j

i , i = j

0, i 6= j
i , i
Pn

j=1

=j
aij bjk = aii bik = i bik =

0, i

6= k
i i , i = k

Similarly for BA.

7.2

Upper-triangular Matrices

We say that A is an upper-triangular matrix if aij = 0, whenever i > j.

7.3

Lower-triangular Matrices

We say that A is a lower-triangular matrix if aij = 0, whenever i < j.

Remark
Diagonal matrices are upper-triangular and lower-triangular. Conversely, if a
matrix is both upper-triangular and lower-triangular, it is a diagonal matrix.

13

7.4

Theorem

If A and B are both upper-triangular, then AB and BA are upper-triangular

too.

7.4.1

Proof

Denote C = AB.
cik =

n
X

aij bjk

j=1

Suppose i > k, then, either i > j or j > k. So, in each case, atleast one of aij
or bjk is 0.

7.5

Identity Matrix

Let n 1. We call In the n n identity matrix.

1 0 0 0

0 1 0 0

In =

0 0 . . . 0

0 0

7.6

Theorem

Let In be the identity n n matrix. Then, for any n n matrix B, we have

In B = BIn = B
7.6.1

Proof

0, i

6= j
=j
Denote C = In B. We have
In = (eij ); eij =

cik =

n
X

1, i

eij bjk = eii bik = 1 bik = bik

j=1

C = B In B = B
Similarly for BIn = B.

14

7.7

Inverse of Matrix

Let A be an n n matrix. We say that A is invertible if there exist B, C, s.t.

AB = In and CA = In

Remark
A = O is not invertible because OB = CO = O 6= In

Remark
There are non-zero
matrices which are not invertible.
!
0 1
Let A =
0 0
If possible, let !
there be C s.t. CA = I2 .
1 0
Let B =
0 0
We have CA = I.
(CA)B = IB
C(AB) = B
CO = B
O=B
But, B 6= 0. Therefore, C does not exist.

7.7.1

If AB = In and CA = In , then B = C
C = CI
= C(AB)
= (CA)B
= IB
=B

7.7.2

Inverse of a Matrix

If A is invertible, i.e. if there exists B, s.t. AB = BA = I, then, B is called

the inverse of A, and is denoted by A1 .

15

7.7.3

If AB = I, then BA = I.

7.7.4

If A is invertible, then A cannot be a zero divisor.

If possible, let A be a zero divisor.

Therefore, either AB = O, for some B 6= O; or CA = O, for some C 6= O

Case I: AB = O
AB = O
A1 (AB) = A1 O
(A1 A)B = O
IB = O
B=O
This contradicts the assumption B 6= O
Case II: CA = O
CA = O
(CA)A = OA1
C(A1 A) = O
CI = O
C=O
1

This contradicts the assumption C 6= O

7.7.5

If A and B are invertible, then A + B may or may not be

invertible.

If A = B, then A + B = 2A is invertible.
If A = B, then A + B = O is not invertible.

16

7.7.6

If A and B are invertible, then AB must be invertible.

(AB)(B 1 A1 ) = A(BB 1 )A1
= AIA1
= AA1
=I
Similarly, (B 1 A1 )(AB) = I
(AB)1 = B 1 A1

Transpose of a Matrix

Let A be a m n matrix, A = (aij )1im;1jn

B = At is defined as follows.
bji = aij

8.1

Properties of At

1. (A + B)t = At + B t
2. (A)t = At
3. (AB)t = B t At
4. If A is invertible, then, At must be invertible, and (At )1 = (A1 )t

Adjoint Matrix
t

A =A

For example,
!

1 1+i 21
A=
i 5i
3

1
i

i
5i
B=

2+i 3

17

9.0.1

Properties of Adjoint Matrices

1. (A + B) = A + B
2. (A) = A
3. (AB) = B A
4. If A is invertible, then A is invertible, and (A )1 = (A1 )

10
10.1

Row Operations on Matrices

Elementary Row Operations

Let A be a m n matrix with rows a1 , . . . am . We define 3 types of elementary

row operations.
I ai aj (Switch of the ith and j th rows.)
II ai ai ( 6= 0) (Multiplication of a row by a non-zero scalar.)
III ai ai + aj (j 6= i) (Addition of a row multiplied by a scalar, and
another row.)
EI , EII , EIII are matrices obtained from the identity matrix by applying elementary row operations I, II, III, respectively. These matrices are called
elementary matrices.

10.2

Theorems


Let ei = 0 . . . 0 1 0 . . .
Let A be any m n matrix.
Then, ei A = the ith row of A.
10.2.1

0 be a 1 m matrix.

EI A = the matrix obtained from A by an elementary row

operation I

Proof
Let A be any m n matrix.

18

eA
1
..
.
EI A =

e A
j
.
.
.

ei A

.
..

em A

10.2.2

EII A = the matrix obtained from A by an elementary row

operation II

Proof
Let A be any m n matrix.

e1 A
.
.
.

EI A =
ei A
.
..

em A

10.2.3

EIII A = the matrix obtained from A by an elementary row

operation III

Proof
Let A be any m n matrix.

19

EI A =

a
i1

10.2.4

ith

e1 A
..
.

+ aj1 + ain + ajn

..

ej A

..

em A

1st row of A
..
.
row of A + (j th ) row of
..
.
j th row of A
..
.
mth row of A

All elementary matrices are invertible, moreover, the inverses of EI , EII , EIII are also elementary matrices of the
same type.

EI1 = EI
EI2 = Im

20

EI2 = EI EI

eE
1 I
..
.
=

e E
j I
.
.
.

ei EI

.
..

em EI

1st row of A

..

.
=

j th

th
i

row of A

..

row of A

..

mth row of A

e
1
..
.
=

e
j
.
.
.

ei

.
..

= Im

em

21

Similarly for EII , to get the inverse, is replaced by

0
EII =
.
..

0 0 ...
1 0 ...
0 ...
.. .. . .
.
. .
0 0 0 ...

1 0

0 1

EII1

0
0

1
0 0
=

. . .
.. .. ..

0 0 0

...
...

..
.

. . . 0

. . ..
. .

... 1

Similarly for EIII , to get the inverse, is replaced by

EIII

=
0
.
..

0 0 ...
1 ...
0 1 ...
.. .. . .
.
. .
0 0 0 ...

1
EIII

10.3

0
=
.
..

0 0 ...
1 . . .
0 1 ...
..
..
..
.
.
.
0 0 0 ...

..
.

..
.

Row-equivalent of a Matrix

A matrix A0 is a row-equivalent of A, if A0 is obtained for A, by a finite

sequence of elementary row operations.

11
11.1

Row Echelon Form of a Matrix

Definition

Let A be an m n matrix.
Denote the ith row of A by ai .
22

The leading entry of a non-zero row ai is its first non-zero entry.

Denote the column where the leading entry occurs by li .
aij = 0 if j < l(i)
aij 6= 0 if j = l(i)

We say that A is in row echelon form(REF) if the following conditions hold.

1. The non-zero rows are at the top of A. (r = the number of non-zero
rows)
2. The leading entries go right as we go down, i.e. l(1) < l2 < < l(r)
3. All leading entries equal 1, i.e. if j = l(i), then, aij = 1
4. Any column which contains a leading entry must have all other entries
equal to 0, i.e. if j = l(i), then, akj = 0; k 6= i

11.2

Notation

The REF of A will be denoted by AR .

12

Row Rank of a Matrix

The number of non-zero rows in AR is called the row rank of A. It is denoted

by r.
rn

13

Gauss Theorem

Any m n matrix A can be brought to REF by a sequence of elementary

row operations.

13.1

Elimination Algorithm

Step 1 Find the first non-zero column Cp of A.

Step 2 Denote by aip the first non-zero entry of Cp .
Step 3 Switch the 1st and ith rows.
23

Step 4 Multiply the 1st row by

1
.
aip

Step 5 Using row operations of type III, make all other entries of the pth column
zeros.
Step 6 Ignoring the top row and Cp , repeat steps Step 1 to Step 5.
13.1.1

Example

0 0 0 1
0 1 4 7
0 1 4 7

R1 R2
R1 R1
R3 R3 +R1
0 1 4 7

0 1

0 0 0 1 0 0

0 1 7 6
0 1 7 6
0 1 7
6

0 1 4 7

0
1
4
7
0
1
4
7
R2

R2 R3
R2 3

1
R1 R1 +4R2
0 0 0 1

0 0 1

0 0 3 1

0 0 3 1
0 0 0 1

0 0 0
1

25
25
0 1 0
0 1 0
0 1 0
0

3
3

25
1
1
1 R3 R3
1 R1 R1 + 3 R3

R2 R2 + 3 R3
0 0 1 0 0 1 0 0 1

3
3
3

0 0

0 0

0 0

0 1 0 0

0 0 1 0

0 0 0 1

13.2

Row Spaces of Matrices

Definition 8 (Row space of a matrix). Let A be a m n matrix over F.

R(A) is defined as
R(A) = span v1 , . . . , vm
where v1 , . . . , vm are rows of A.
R(A) a subspace of the vector space of all rows of length n, is called the row
space of A.
Definition 9 (Row rank of a matrix). dim R(A) is called the row-rank of A,
and is denoted by rr(A).
24

Theorem 3. Let P be a l m matrix. Then

1. R(P A) R(A)
2. If P is an invertible m m matrix, then R(P A) = R(A)
Corollary 3.1.
R

A0 A = R(A0 ) = R(A)
Theorem 4. If A is in REF, and if r is the number of non-zero rows in A,
then
rr(A) = r
Corollary 4.1. The following are equivalent
R

1. A A0
2. There is an invertible matrix P, s.t. A0 = P A
3. R(A) = R(A0 )
4. A and A0 have the same REF

13.3

Column Equivalence

Definition 10 (Elementary column operations, column equivalence, column

echelon form, column space and column rank). If A is a m n matrix, we can
C
define elementary column operations, column equivalence (A ) and column
echelon form (CEF), the column space of A (C(A)), and the column rank of
A (cr(A)).
Theorem 5.
cr(A) = rr(A) = r
Proof. Let r = rr(A) = dim R(A).
Choose r rows of A which form a basis of R(A), WLG, say v1 , . . . , vr .
Let

Xrn

v
1
..
=.

vr

25

span(X) = R(A)
Hence, any row of A can be expressed as a linear combination of v1 , . . . , vr
vi =

r
X

yij vj

j=1

Let
Ymr = (yij )
Therefore,
A=YX
Considering each column of A as a linear combination of columns of Y ,
C(A) C(Y )
cr(A) cr(Y ) r = rr(A)
cr(A) rr(A)
Similarly,
rr(A) cr(A) cr(A) = rr(A)

Corollary 5.1. The following are equivalent

C

1. A A0
2. There is an invertible matrix Q, s.t. A0 = QA
3. C(A) = C(A0 )
4. A and A0 have the same CEF

26

Part IV

Linear Systems
1

Definition
a11 x1 + a12 x2 + + a1n xn = b1
a21 x1 + a22 x2 + + a2n xn = b2
..
.
am1 x1 + am2 x2 + + amn xn = bm

Here, all xi are taken to be unknowns, and all aij , bi are given.
A solution to such a system is a collection d1 , . . . , dn , s.t. after replacing xi
by di , we get equalities.
We assume that all aij , bi belond to F, and we are looking
di F.
for
solutions
b
x
1
1
..
..
Given such a system, we define Amn = (aij ), bm1 = . , xn1 = .

bm

xn

Then, we can write the system as

Ax = b

d
1
..
A solution to this system is dn = . , s.t. Ad = b

dn

d
1
..
Let D be the set of all d = .

dn
D may be empty, infinite, or a singleton set.

Equivalent Systems

Two systems Ax = b and A0 x = b0 are called equivalent, if every solution of

the first system is also a solution of the second system, and vice versa.

27

Solution of a System of Equations

We want to bring a given system

Ax = b
to the form
AR x = bR
using elementary row operations.
We denote the augmented or extended matrix of the system as follows.
Am(n+1) = (Amn |bm1 )
Then apply Gaussian elimination method to A, in order to get the matrix
(AR |bR )
As AR is obtained from A using elementary row operations,
AR = En . . . E2 E1 A
where every Ei is an elementary matrix.
Let P = En . . . E2 E1 . P is invertible, as it is a product of elementary matrices.
AR = P A
AR d = P Ad
= Pb
= bR
Conversely, let d be a solution to
AR d = bR
P Ad = bR
1
P (P Ad) = P 1 bR
Ad = b
If we have a system Ax = b, we may and will assume that A is in REF, i.e.
A = AR , b = bR .
Let l(1), . . . l(r) denote the numbers of the columns containing leading entries.

28

b
1
..
.
Let b =

b
r

br+1

..
.

bm

Therefore,
1 xl(1) + . . . = b1
1 xl(2) + . . . = b2
..
.
1 xl(r) = br
0 = br+1
..
.
0 = bm

4
4.1

Homogeneous Systems
Definition

A system of the form

Ax = O
is called a homogeneous system.
Remark
Any homogeneous system is consistent and has a trivial solution x = O

4.2

Solutions of Homogeneous Systems

If r = number of non-zero rows, let t = n r = number of free variables. If

t > 0 , denote the numbers of the columns that do not contain leading entries
by z(1), . . . , z(t)

29

4.2.1

Example

0
A=
0

1
0
0
0

2
0
0
0

0 3

0 1

1 7

0 0

0
1
0
0

Therefore,
m=4
n=6
r=3
t=3
l(1) = 2
l(2) = 4
l(3) = 5
z(1) = 1
z(2) = 3
z(3) = 6
Therefore,
x2 + 2x3 3x6 = 0
x4 x6 = 0
x5 + 7x6 = 0
Therefore,
x2 = 2x3 + 3x6
x4 = x6
x5 = 7x6

x
x
2
1
x4 = C33 x3


x5
x6

where C33 = 0
0
The free variables

2 3
0
1

0 7
x1 , x3 , x6 can be considered as parameters, x1 = 1 , x2 =
30

2 , x3 = 3 .
Therefore,
x2 = 23 + 36
x4 = 6
x5 = 76
4.2.2

General Solution

4.2.2.1 Case I: t = 0
If t = 0, there are no free variables, and the system has a unique trivial
solution.
4.2.2.2

Case II: t > 0

x
x
l(1)
z(1)
xl(2)
xz(2)

. = Crt .
..
..

xl(r)

xz(t)

C is filled by coefficients of the equations obtained after shifting the terms

containing all zi to the RHS.

4.3
4.3.1

Properties
For a homogeneous system Ax = 0, if c and d are solutions,
then c + d is also a solution.
Ac = O
Ad = O
A(c + d) = Ac + Ad
=O+O
=O

4.3.2

For a homogeneous system Ax = 0, if c is a solution and

F, then, c is a solution too.
Ac = O
A(c) = (Ac)
= O
=O
31

4.4

Fundamental Solutions

We define t fundamental solutions or basic solutions, v1 , . . . vt .

We define t columns, each of length n as follows.
For the ith column vi , we set
xz(1) = 0
xz(i) = 1
..
.
xz(t) = 0

and for xl(1) , . . . , xl(r) ,

x
x
l(1)
z(1)
..
..
. = C . = ith column of C

xl(r)
xz(t)
4.4.1

Theorem: Any solution d of the system Ax = O can be

obtained from the basic solutions v1 , . . . , vt as a linear combination of the basic solutions, d = 1 v1 + . . . t vt

One can choose another collection v10 , . . . , vt0 s.t. any solution of Ax = O
can be obtained as a linear combination of v10 , . . . , vt0 . In such a case, we get
another form of the general solution.

4.5
r min m, n

If r = n, i.e. t = 0, the system has a unique solution.

If r < n, i.e. t > 0, the system has more than one solutions. Its general
solution can be expressed as in terms of t parameters, where each free variable
serves as a parameter, whose value can be any element of F.
If m < n, then r < n. Therefore, the system has more than one solution.

32

5
5.1

Non-Homogeneous Systems
Definition

Consider a system Ax = b; b 6= O. The extended matrix is defined as

...

am1 . . .

a
11
..
e
A = (A|b) = .

5.2

a1n
..
.

b1
..
.

amn bm

Solutions of Non-Homogeneous Systems

e i.e. A
g.
Let re be the number of non-zero rows in the REF of A,
R

5.2.1

Case I: re = r
b0r+1 = = b0m = 0

5.2.1.1 Case a: r = n, i.e. t = 0

Therefore,
x1 = b01
...
xr = b0r

Hence, the system has a unique solution.

5.2.1.2 Case b: r < n, i.e. t > 0
Therefore,
xl (1) = b01 + c11 xz(1) + + c1t xz(t)
..
.
xl (r) = b01 + cr1 xz(1) + + crt xz(t)
5.2.2

Case II: re > r

In this case, the (r + 1)th row represents an equation of the form 0 = 1.

Therefore, the system is inconsistent.

33

5.3

General Solution

The general solution of Ax = b can be expressed by adding the general

solution of Ax = b and any particular solution of Ax = b.
If c is a solution of Ax = O, and d is a solution of Ax = b, then c + d
is a solution of Ax = b.
Conversely, if d and d0 are solutions of Ax = b, then, c = d0 d is a solution
of Ax = O.

34

Part V

Vector Spaces
1

Definition

Let F be a field. A vector space V , over F, is a set on which there are two
operations, denoted by + and , where
+ is the addition of elements of V
is the multiplication of an element of V by an element of F
s.t. the sum of elements of V lies in V , and the product of an element of V
by an element of F lies in V , and the following properties hold.
(A1) x + y = y + x; x, y V
(A2) (x + y) + x = x + (y + z); x, y, z V
(A3) O V , s.t. O + x = x + O = x; x V
(A4) x V, y V , s.t. x + y = O. (y is denoted as x.)
(M1) (x + y) = x + y; F, x, y V
(M2) ( + )x = x + y; , F, x V
(M3) ()x = (x) = (x); , F, x V
(M4) 1 x = x; x V
Elements of V are called vectors, and elements of F are called scalars.

1.1

Examples

1.1.1

Geometric Vectors in Plane

1.1.2

Arithmetic Vector Space

Let F be a field, and n 1 Z.

Let V = Fn be a set of ordered n-tuples.
We define
(1 , . . . , n ) + (1 , . . . , n ) = (1 + 2 , . . . , n + n )
(1 , . . . , n ) = (1 , . . . , n )
35

1.1.3
Let F be a field, and m, n 1 Z.
Let V = Fmn be the set of all (m n) matrices over F, i.e. a set of ordered
mn-tuples. For X, Y V , we use the usual definitions of X + Y and X
from algebra of matrices.

Properties
1. O = O; F
2. (x) = (x)
.
3. x y = x + (y)
4. 0x = O; x V
5. (1)x = x; x V
6. ( ) = x x; , F, x V

2.0.1

Proof of 1
O = (O + O
= O + O

For Oy s.t. O + y = O.
Therefore,
O + y = (O + O) + y
O = O + (O + y)
= O + O
= O

Subspaces

Let V be a vector space over F. Let U V . U is called a subspace of V if

the following properties hold.
Axiom 1 O U
36

Axiom 2 If x, y U , then, (x + y) U
Axiom 3 If x U, F, then, x U

3.1

Examples

Example 4. Let V be the set of all geometric vectors in plane.

If U1 is the set of all vectors along the x-axis, U2 is the singleton set of a
specific vector along the x-axis, and U3 is the set of all vectors along the
x-axis and a specific vector not along the x-axis. Which of U1 , U2 , U3 are
subspaces of V ?
Solution. U1 is a subspace of V as it satisfies all three axioms.
U2 is not a subspace of V as it does not satisfy any of the three axioms.
U3 is not a subspace of V as it does not satisfy Axiom 3
Example 5.
F=R
V = C = { + i; , R}
where + is addition in C and is multiplication by real scalars.
U1 = { + 0i}
U2 = {0 + i}
Which of U1 , U2 , U3 are subspaces of V ?
Solution. Both U1 and U2 are subspaces of V , as they satisfy all three axioms.
Example 6. Let V = F, where + is addition in F, and is multiplication in F.
U1 = { + 0i}
U2 = {0 + i}
Which of U1 , U2 are subspaces of V ?
Solution. Neither U1 nor U2 are subspaces of V .
Example 7. Let V = {f : [0, 1] R}, where + and is defined as follows.
(f + g)(x) = f (x) + g(x)
(f )(x) = f (x)

37

O is the function with graph x = 0.

U = {all continuous functions[0, 1] R}
Is U is subspace?
Solution. O R. Therefore, Axiom 1 is satisfied. Similarly, Axiom 2 and
Axiom 3 are also satisfied.

3.2

Operations on Subspaces

Let V /F be a vector space, and U1 , U2 be subspaces of V .

U1 U2 = {x V : x U1 and x U2 }
U1 U2 = {x V : x U1 or x U2 }
U1 + U2 = {x V : x = x1 + x2 , x1 U1 , x2 U2 }

Example 8. Let V be a set of geometric vectors in 3D space.

Let U1 be the xy-plane, and U2 be the yz-plane. If U1 U2 a subspace of V ?
Solution.
O U1 , O U2 O U1 U2
x, y U1 U2 x, y U1 , x, y U2
x + y U1 , x + y U2
= x + y U1 U2
Similarly, if x U1 U2 , inF, then, x U1 U2 . Therefore, U1 U2 is a
subspace of V .

Spans

Definition 11 (Span). Let V /F be a vector space. Let S V be non-empty.

span(S) = {x V : x = 1 v1 + +m vm , 1 , . . . , m F, v1 , . . . , vm S}
span(S) is the collection of all linear combinations of finite number of vectors
of S with coefficients from F
Theorem 1. span(S) is a subspace of V
38

Proof.
O = 0v O span(S)
x, y span(S) x = 1 v1 + + m vm , 1 w1 + + m wm
x + y = 1 v1 + + m vm + 1 w1 + + m wm span(S)
x span(S), F 1 v1 + + m vm
x = (1 v1 + + m vm )
x = 1 v1 + + m vm span(S)

Definition 12 (Spanning sets and dimensionality). Let V /F be a vector

space. A set S V is said to be a spanning set, if span(S) = V .
If V has atleast one finite spanning set, V is said to be finite-dimensional.
Otherwise, V is said to be infinite-dimensional.
Remark 4. V may have many finite spanning sets, of different sizes
Definition 13 (Basis of a vector space). Let V /F be a vector space. We
say that B = {v1 , . . . , vn } V is a basis of V if every vector v V can be
expressed in a unique way
v = 1 v1 + + n vn

; 1 , . . . , n F

that is, as a linear combination of elements of B.

Definition 14 (Isomorphic spaces). Let V /F and W/F be vector spaces. We
say that V is isomorphic to W if there is a map : V W , s.t.
1. is one-to-one and onto
2. (v1 + v2 ) = (v1 ) + (v2 ); v1 , v2 V
3. (v) = (v); v V, F
Theorem 2. If a vector space V /F has a basis B = {v1 , . . . , vn } consisting
of n elements, then it is isomorphic to the space

W = Fn =

1
..
.

39

Proof. Let B 0 = {e1 , . . . , en }, where

0
1


..
..
e1 = . , . . . , e n = .


1
0

B is a basis of Q, as any w =

1
..
.

W can be expressed in a unique way

w = 1 e1 + + n en
Let : V W ,
(v1 ) = e1
..
.
(vn ) = en

For any v = 1 v1 + + n vn V ,

1
..
(v) = .
n

Therefore,
(1 v1 + + n vn ) = 1 e1 + n en
= 1 (v1 ) + + n (vn )
If v 6= v 0 ,
v = 1 v1 + + n vn
v 0 = 10 v1 + + n0 vn
Hence is one-to-one.

1
..
For any w = . W .

n
40

Let v = 1 v1 + + n vn .
Therefore,

1
..
(v) = . = w
n

Therefore, is onto.

Linear Dependence

Definition 15 (Linearly dependent subsets). Let V /F be a vector space. Let

S V be a finite subset. S is said to be linearly dependent if there exist
scalars 1 , . . . , n F, not all equal to zero, s.t.
1 v1 + + n vn = O
Otherwise, S is said to be linearly independent if all 1 = = n = 0.
Example 9. Is S = {v1 , . . . , vl , v, v} linearly dependent?
Solution.
(0)v1 + + (0)vl + ()v + (1)v = O
Therefore, as not all coefficients are zero, S is linearly dependent.
Example 10. Is S = {v1 , . . . , vl , O} linearly dependent?
Solution.
(0)v1 + + (0)vl + (1)O = O
Therefore, as not all coefficients are zero, S is linearly dependent.
Theorem 3. Any basis B = {v1 , . . . , vn } of a vector space V is linearly
independent.
Proof. Let
1 v1 + + n vn = O
Also,
(0)v1 + + (0)vn = O

(3.1)
41

Therefore, there are two representations of v = O as linear combinations of

elements of B. By the definition of basis, they must coincide.
Therefore,
1 = 0
..
.
n = 0
Hence, B is linearly independent.

5.1

Properties of Linearly Dependent and Independent

Sets

Theorem 4. If S S 0 and S is linearly dependent, then S 0 is also linearly

dependent.
Theorem 5. If S S 0 and S 0 is linearly independent, then S is also linearly
independent.
Theorem 6. Let S = {v1 , . . . , vn }. S is linearly dependent iff one of the vi s
is a linear combination of the others.
Proof of statement. Suppose
vn = 1 v1 + + n1 vn1
1 v1 + + n1 vn1 + (1)vn = O
Therefore, S is linearly dependent.
Proof of converse. Suppose
1 v1 + + n1 vn1 + n vn = O
not all of i s are 0. WLG, let n 6= 0
vn =

1
n1
v1
vm1
m
m

Theorem 7. Let S = {v1 , . . . , vm }. Let w V . Suppose w is a linear

combination of vi s
w = 1 v1 + + n vn
Then, such an expression is unique iff S is linearly dependent.
42

Proof of statement. Let

w = 1 v1 + + n vn
be unique.
If possible, let
1 v1 + + n vn = O
not all i s are zero.
Then,
(1 + 1 )v1 + + (n n )vn = w
This is another expression for w, and contradicts the assumption.
Proof of converse. If possible, let S be linearly independent. Assume
w = 10 v1 + + n0 vn
Therefore,
(1 10 )v1 + + (n n0 )vn = O
Therefore, S is linearly dependent, which contradicts the assumption.
Theorem 8 (Main Lemma on Linear Independence). Suppose V is spanned
by n vectors.
Let S = {v1 , . . . , vm } V . Suppose m > n.
Then, S is linearly dependent.
Proof. Let E = {w1 , . . . , wn } be a spanning set for V , V = span(E).
Therefore, all elements of S can be represented as linear combinations of
elements of E.
v1 = 11 w1 + + 1n wn
..
.
vm = m1 w1 + + mnwn
Let
1 v1 + + m vm = O
1 (11 w1 + + 1n wn ) + + m (m1 w1 + + mn wn ) = O
(1 11 + + m m1 )w1 + + (1 1n + + m mn ) = O
43

Therefore
1 11 + + m m1 = 0
..
.
1 1n + + m mn = 0
These equations form a homogeneous linear system with respect to 1 , . . . , m .
As m > n, the system has a non-zero solution. Therefore not all i s are zero.
Hence S is linearly dependent.
Definition 16 (Alternative definition of a basis). B = {v1 , . . . , vn } is said
to be a basis of V if B is a spanning set and B is linearly independent.
Theorem 9. If B and B 0 are bases of V , then they contain the same number
of elements.
Proof. If possible, let B contain n elements {v1 , . . . , vn }, and B 0 contain m
elements {w1 , . . . , wm }, m > n.
Therefore, B is a spanning set and B 0 contains more elements than n, hence
by Main Lemma on Linear Independence, B 0 is linearly dependent. Also, B 0
is a basis, so it is linearly independent.
This is a contradiction.
Definition 17 (Dimension of a vector space). Let V /F be a finite-dimensional
vector space. The number of elements in any basis B of V is called the
dimension of V .
n = dim V
Remark 5. If V and W are vector spaces over F, s.t.
dim V = dim W
then, V is isomorphic to W
Theorem 10. If S = {v1 , . . . , vm } is a spanning set of V , and if S is not a
basis of V , a basis B of V can be obtained by removing some elements from
S.
Proof. If S is linearly independent, then it is a basis.
Otherwise, if S is linearly dependent, it has an element, WLG, say vm , which
is a linear combination of the others.
vm = 1 v1 + + m1 vm1
44

Let
S 0 = S {vm }
S 0 is a spanning set.
Therefore, v V
v = 1 v1 + + m1 vm1 + m vm
= 1 v1 + + m1 + m (1 v1 + + m1 vm1 )
= 1 v1 + + m1 vm1
If S 0 is linearly independent, then it is a basis, else the same process above
can be repeated till we get a basis.
Therefore, a basis is a smallest spanning set.
Theorem 11. If B0 = {v1 , . . . , vn } is a linearly independent set, and if B0
is a basis of V , a basis of V can be obtained by adding elements to B0 .
Theorem 12. Let V be a vector space, s.t. dim V = n.
If B satisfies 2 out of the 3 following conditions, then it is a basis.
1. B has n elements.
2. B is a spanning set.
3. B is linearly dependent.
Theorem 13 (Dimension Theorem).
dim(U + W ) = dim U + dim W dim(U W )
Theorem 14.
U + W = span(U W )
If
U = span(B)
W = span(B 0 )
then,
U + W = span(B B 0 )

45

Proof. Let v U + W .
Then,
v = u + w ; u U, w W
uU W
w U W
v span(U W )
Let
v span(U W ) v = 1 v1 + + k vk

; vi U W

Let
v1 , . . . , vl U
vl+1 , . . . , vk W
Therefore,
v = (1 v1 + + l vl ) + (l+1 vl+1 + + kvk )
v U +W

5.2

Changing a Basis

Let B = {v1 , . . . , vn } be a basis of V , s.t. dim V = n. Let B 0 = {v10 , . . . , vn0 }.

As B is a spanning set, all of v10 , . . . , vn0 can be expressed as a linear combination
of v1 , . . . , vn .
v10 = 11 v1 + + n1 vn
..
.
0
vn = 1n v1 + + nn vn
Definition 18 (Transition matrix). The matrix

...
11
..
C= .

n1 . . .

1n
..
.
nn

is called the transition matrix from B to B 0 .

46

If B and B 0 are considered as row vectors of length n filled by vectors,

v10 = 11 v1 + + n1 vn
..
.
0
vn = 1n v1 + + nn vn
can be written as
0
= B1n Cnn
B1n

Theorem 15. B 0 is a basis of V iff C is invertible.

Proof of statement. Let B 0 = BC be a basis.
B 0 is a basis, and hence is a spanning set. Therefore, any vector from B can
be expressed as a linear combination of elements of B 0 .
Therefore,
B = B0Q
= BCQ
Also,
B = BI
Therefore,
I = CQ
Similarly,
B 0 = BC
= B 0 QC
Also,
B0 = B0I
Therefore,
I = QC
Therefore,
CQ = QC = I
Hence C is invertible.
47

Proof of converse. Let B 0 = BC and C be invertible. Therefore, B 0 is a basis

iff B 0 is a spanning set.
Let z V . As B is a spanning set,
z = 1 v1 + + n vn
Therefore,
z = Bg
where

1
..
g= .

n
z = Bg
= B(Ig)
= B(CC 1 )g
= (BC)(C 1 g)
Let C 1 g = f
z = B0f
Therefore, z can be expressed as a linear combination of vectors from B 0 .
Remark 6. Let B be a basis of V . If
BP = BQ
where P and Q are n n matrices, then
P =Q
Example 11. Let B = {e1 , e2 } and B 0 = {e01 , e02 }, where
e01 = e1 + e2
e02 = e1 + e2
Solution.
e01 = e1 + e2
e02 = e1 + e2
!

1 1
C=
1 1

48

1
e1 = e01
2
1
e2 = e01 +
2

2
C 1 =
1

5.3

1 0
e
2 2
1 0
e2
2
1

Representation of Vectors in a Basis

Let V be a vector space of dimension n. Let B = {v1 , . . . , vn } be a basis of

V.
Let z V .
z can be written as a unique linear combination of elements of B.
z = 1 v1 + + n vn
The representation of z w.r.t B can be represented as

1
..
[z]B = .

n
5.3.1

Properties of Representations

1. [z1 + z2 ]B = [z1 ]B + [z2 ]B

2. [z]B = [z]B
3. [z1 ]B = [z2 ]B z1 = z2

1
1
..
..
n
4. . F , z V, s.t. [z]B = .


n
n

49

Determinants

6.1

Definition

Definition 19 (Determinants). Given an n n matrix A, n 1, det(A) is

defined as follows.
n=1

det(a) = a
!

n=2

a
a
det 11 12 = a11 a22 a12 a21
a21 a22

..
.
n=n
The determinant of a n n matrix is the summation of n! summands. Each
summand is the product of n elements, each from a different row and column.
Summand Permutation
!
1 2 3
a11 a22 a33
1 2 3!
1 2 3
a12 a23 a31
2 3 1!
1 2 3
a13 a21 a32
3 1 2!
1 2 3
a13 a22 a31
3 2 1!
1 2 3
a12 a21 a33
2 1 3!
1 2 3
a11 a23 a32
1 3 2

6.2

Number of Elementary Permutations1

Parity

even

2 ((1, 2, 3) (2, 1, 3) (2, 3, 1))

even

2 ((1, 2, 3) (1, 3, 2) (3, 1, 2))

even

1 ((1, 2, 3) (3, 2, 1)

odd

1 ((1, 2, 3) (2, 1, 3)

odd

1 ((1, 2, 3) (1, 3, 2)

odd

Properties

Theorem 16. If A, A0 are matrices s.t. all rows except the ith row are
identical, and A00 is obtained by addition of ith row of A and ith row of A0 ,
then
det(A00 ) = det(A) + det(A0 )
1

Any permutation can be represented as a result of a series of elementary permutations,

i.e. permutations of 2 elements only. The parity of a particular permutation depends of
the parity of the number of elementary functions required for it.

50

Theorem 17. If A0 is obtained from A by switching two rows, then

det(A0 ) = det(A)
Theorem 18. If A0 is obtained from A by multiplication of a row by a scalar
, then
det(A0 ) = det(A)
Theorem 19. If A0 is obtained from A by adding to the ith row the j th row
multiplied by a scalar , then
det(A0 ) = det(A)
Corollary 19.1 (Corollary of Property 2). If A has two identical rows, then
det(A) = 0.
Theorem 20. The determinant of upper triangular and lower triangular
matrices is the product of the elements on the principal diagonal.
Theorem 21.
det(At ) = det(A)
Corollary 21.1. In all above theorems, the properties which are applicable
to rows, are also applicable to columns.
Theorem 22. If A, B, C are some matrices, and O is the zero matrix,
!

Amm
B
= det(A) det(C)
O
Cnn
Theorem 23.
det(AB) = det(A) det(B)
Corollary 23.1. If A is invertible, then
det(A) 6= 0
Proof. A is invertible.
Therefore, P , s.t.
PA = I
det(P A) = det(I)
det(P ) det(A) = 1
det(A) 6= 0

51

Theorem 24. If
det(A) 6= 0
then A is invertible.
Proof. If possible let A be non invertible.
Let the REF of A be AR .
As A is non invertible, AR has a zero row. Therefore,
det(AR ) = 0
But
det(A) = 0
This is not possible as elementary row operations cannot change a non-zero
determinant to zero.
Therefore, A is invertible.
Theorem 25.
det(A) 6= 0
iff the rows of A are linearly independent iff the columns of A are linearly
independent.
Proof. If possible, let the rows of A be linearly dependent.
Therefore, either all of them are zeros, or one row is the linear combination
of the others.
Case 1 (All rows are zeros).
det(A) = 0
Case 2 (One row is a linear combination of the others). Let
vn = 1 v1 + + n1 vn1

v
1
..
.

A=

n1
vn

52

vn vn 1 v1 + + n1 vn1

v
1
..
.

A0 =

v
n1
O

det(A0 ) = 0
det(A) = 0
This contradicts det(A) 6= 0. Therefore, the rows of A must be linearly
independent.

If v1 , . . . , vn are linearly independent,

dim R(A) = n
r=n
Therefore, there are no zero rows in REF of A. Hence A is invertible.
det(A) 6= 0

6.3

Practical Methods for Computing Determinants

6.4

Expansion along a row/ column

Let A be a m n matrix, and let Aij be the matrix obtained by removing

the ith row and j th column from A.

det(A) =

n
X

(1)i+j aij det(Aij )

j=1

6.5

Determinant Rank

Definition 20 (Determinant rank). Let A be any m n matrix. Consider all

square sub-matrices of A and compute their determinants. If there is an r r
53

sub-matrix of A s.t. its determinant is non-zero, but the determinants of all

(r + 1) (r + 1) sub-matrices of A are zero, then, r is called the determinant
rank of A.
Theorem 26. The determinant rank of A is equal to the rank of A.

Linear Maps

7.1

Definition

Definition 21 (Linear map). Let V and W be vector spaces over the same
field F.
:V W
is said to be a linear map if
1. (v1 + v2 ) = (v1 ) + (v2 ); v1 , v2 V
2. (v) = (v); v V, F

7.2

Properties

1. (O) = O
2. (v) = (v)

7.3

Matrix of a Linear Map

Definition 22 (Matrix of a linear map). Let : V W be a linear map.

Let
n = dim V
m = dim W
Let
B = {v1 , . . . , vn }
B 0 = {w1 , . . . , wm }

54

be bases of V and W respectively.

Let
(v1 ) = 11 w1 + + m1 wm
..
.
(vn ) = 1n w1 + + mn wm
The matrix

11
..
A= .

...

m1 . . .

1n
..
.
mn

is called the matrix of with respect to the bases B and B 0 .

It is denoted as
A = []B,B 0
Theorem 27. Let
:V W
be a linear map.
Let B and B 0 be bases of V and W respectively, and let
A = []B,B 0
be the matrix of with respect to B and B 0 . Then, x V ,
[(z)]B 0 = A[z]B
Proof. Let
B = {v1 , . . . , vn }
B 0 = {w1 , . . . , wm }
Case 3 (z B). WLG, let z = vi . Then,

[z]B =

.
.
.

1

.
..

55

i.e. all rows except the ith row are 0.

Let this vector be ei .
Therefore,
A[z]B = Aei
is the ith column of A.

[(z)]B 0 = [(vi )]B 0

is the ith row in the formulae of (v1 ), . . . , (vn ).
Therefore, it is the ith column of A.
Case 4 (z V is an arbitrary vector). Let
z = 1 v1 + + n vn
Therefore,
[(z)]B 0 = [(1 v1 + + n vn )]B 0
= [1 (v2 ) + + n (vn )]B 0
= 1 [(v1 )]B 0 + + n [(vn )]B 0
= 1 (1st column of A) + + n cn (nth column of A)
= A[z]B

7.4

Change of Bases

Theorem 28. Let V , W be vector spaces over F, dim(V ) = n, dim(W ) = m.

f0
Let : V W be a linear map. Let B, Be be bases of V and let B 0 and B
be bases of W . Let A = []B,B 0 and Ae = []B,
e Be0 be the matrices of w.r.t.
e B
f0 . Let P denote the transition matrix from B to B,
e
the pairs B, B 0 and B,
f0 . Then,
and let Q denote the transition matrix from B 0 to B

Aemn = Q1
mm Amn Pnn
Proof. z V ,
[(z)]B 0 = A[z]B
[(z)]Be0 = A[z]Be

(28.1)
(28.2)
56

We have
(28.3)
(28.4)

[z]B = P [z]Be
[(z)]B 0 = Q[(z)]Be0
Therefore,
(28.1) in (28.4) =

(28.5)

A[z]B = Q[(z)]Be0
(28.3) in (28.5) =
AP [z]Be = Q[(z)]Be0

(28.6)

Multiplying on the left by Q1 ,

Q1 AP [z]Be = [(z)]Be0

[(z)]Be0 = Q1 AP [z]Be

Comparing with (28.2),

Ae = Q1 AP

7.5

Operations on Linear Maps

Definition 23 (Operations on linear maps). Let

:V W
0 : V W
be linear maps.
+ 0 : V W
is defined as
( + 0 )(v) = (v) + 0 (v)
and
: V W
is defined as
()(v) = (v)
57

Definition 24 (Composed map). Let

:V W
0 : W U
be linear maps.
(0 ) : V U
is defined as
(0 )(v) = 0 ((v))
Theorem 29 (Matrix of composed map). Let : V W , 0 : W U be
linear maps. Let ( 0 ) : V U be the composed map. Let dim V = n,
dim W = m, dim U = l. Let B, B 0 , B 00 be bases of V , W , U respectively. Let
A = []B,B 0 , A0 = [0 ]B 0 ,B 00 be the matrices of , 0 . Let A00 = [0 ]B,B 00 be
the matrix of the composed map. Then,
A00 = A0 A
Proof. Let z V .
[(0 )(z)]B 00 = [0 ((z))]B 00
= A0 [(z)]B 0
= A0 A[z]B
By definition,
[(0 )(z)]B 00 = A00 [z]B
Therefore,
A00 = A0 A

7.6

Kernel and Image

Definition 25 (Kernel and image). Let : V W be a linear map.

.
ker = {v V : (v) = O}
.
im = {(v) : v V }
58

Theorem 30. ker is a subspace of V and im is a subspace of W .

Proof.
(O) = O
O ker
If v1 , v2 ker , then
(v1 + v2 ) = (v1 ) + (v2 )
=O+O
=O
v1 + v2 ker V
If v ker , F, then
(v) = (v)
= O
= O v ker
Therefore, ker is a subspace of W .
(O) = O
O im
If w1 , w2 im , then
w1 = (v1 )
w2 = (v2 )
w1 + w2 = (v1 ) + (v2 )
= (v1 + v2 )
w1 + w2 im
If w W , F, then
w = (v)
= (v)
w im
Therefore, im is a subspace of W .
59

7.6.1

Dimensions of Kernel and Image

Theorem 31. Let : V W be a linear map. Then

dim(ker()) + dim(im ())
Proof. Let ker = U , U V .
Let B0 = {v1 , . . . , vk } be a basis of U .
Completing B0 to a basis B of V ,
B = {v1 , . . . , vk , vk+1 , . . . , vn }
Let
wk+1 = (vk+1 )
..
.
wn = (vn )
Therefore, we need to prove that B 0 is a basis of W 0 = im (), by proving
that B 0 is a spanning set and that B 0 is linearly independent.
Take w im (), so that there is v V s.t. (v) = w.
Representing v as a linear combination of elements of B,
v = 1 v1 + + k vk + k+1 vk+1 + + n vn
w = (v)
= (1 v1 + + k vk + k+1 vk+1 + + n vn )
= 1 (v1 ) + + k (vk ) + k+1 (vk+1 ) + + n (vn )
= k+1 (vk+1 ) + + n (vn )
= k+1 wk+1 + + n wn
span(B 0 )
Therefore, B 0 is a spanning set for W 0 .
Let
k+1 wk+1 + + n wn = O
Therefore, B 0 is linearly independent iff
k+1 = = n = 0
As is a linear map,
(k+1 vk+1 + + n vn ) = O
k+1 vk+1 + + n vn ker
60

Therefore, it can be expressed as a linear combination of vectors of B0 , which

is a basis of ker .
Let
k+1 vk+1 + + n vn = k+1 vk+1 + + n vn
k+1 vk+1 + + n vn k+1 vk+1 n vn = O
As {v1 , . . . , vn } is a basis of V , all coefficients must be 0
Therefore,
k+1 vk+1 = = n vn = 0
Hence, as B 0 is a spanning set of im and also linearly independent, B 0 is a
basis of im .
Therefore,
dim(im ) = size of B 0
=nk
= n dim(ker )
dim(im ) + dim(ker ) = dim V

Corollary 31.1.
dim(im ) = r
where r is the rank of A
Corollary 31.2. Let Amn be a matrix of rank r. Let C(A) be the column
space of A, and let dim C(A) be the column rank of A. Then
dim C(A) = r
Proof. Define
: Fn Fm
s.t. A = []B,B 0 , where B is the standard basis of F.

B=

0
1

..
..
. , . . . , .

= {e1 , . . . , en }
61

v Fn , we have
[(v)]B 0 = A[v]B
If v = ei ,
[(ei )] = Aei
which is the ith column of A. So, the space spanned by {(e1 ), . . . , (en )} is
equal to C(A). But it is also in im .
Therefore,
im = C(A)
and
dim(im ) = dim(C(A))
r = dim(C(A))

Remark 7. Let : V W be a linear map. Let w im (), so that there

is v V s.t. (v) = w. Then any v 0 s.t. (v 0 ) = w can be written down as
v 0 = v + v0 where v0 ker .

Part VI

Linear Operators
1

Definition

Definition 26 (Linear operator). A linear operator or transformation

T :V V
is a linear map from a vector space V to itself.

Similar Matrices

Let B and Be be bases of V . Let A and Ae be the representing matrices

A = [T ]B
Ae = [T ]Be
62

Both these are n n matrices, where n = dim V . Let P denote the transition
e Then,
matrix from B to B.
Ae = P 1 AP
Definition 27 (Similarity of matrices). Let A, Ae be n n matrices. A is
e denoted as A A,
e if there exists an invertible n n
said to be similar to A,
matrix P , s.t. Ae = P 1 AP .

2.1

Properties of Similar Matrices

1. A A
e then A
eA
2. If A A,
ee
e
3. If A Ae and Ae A,
then A Ae
e then det(A) = det(A)
e
4. If A A,

5. If A I, then A = I

Diagonalization

Given a square matrix Ann , decide whether or not A is similar to some

diagonal matrix D. If it is, find D, and P s.t. P 1 AP = D.
Alternatively,
Given an operator T : V V , decide whether or not there exists a basis B of
V , s.t. [T ]B is a diagonal matrix D. If it exists, find D, and B, s.t. [T ]B = D.
Definition 28 (Diagonalizability). If A is similar to a diagonal matrix, A
is said to be diagonalizable. P , s.t. P 1 AP = D is called a diagonalizing
matrix for A. D is called a diagonal form of A.

Eigenvalues and Eigenvectors

Definition 29 (Eigenvalue and eigenvector). Let A be a n n matrix over

F. F is said to be an eigenvalue of A, if v F, v 6= 0, such that
Av = v
v is called an eigenvector corresponding to .

63

Definition 30 (Alternate definition of eigenvalue and eigenvector). Let

T : V V be a linear operator, where V is a vector space over F. F is
said to be an eigenvalue of A, if v V, v 6= 0, such that
T (v) = v
v is called an eigenvector corresponding to .
Definition 31 (Spectrum). The collection of all eigenvalues of a matrix, or
a linear operator is called the spectrum.
Theorem 1. Let A be a n n matrix. F is an eigenvalue of A iff
det(In A) = 0
Proof. is an eigenvalue of A
v Fn , v =
6 0, s.t. Av = v
v Fn , v =
6 0, s.t. (I A)v = O

x
1
..
v = .

xn

(I

x
1
..
A) .

= 0 has a non-zero solution

xn

there are free variables

det(I A) = 0

Theorem 2 (General criterion for diagonalization). Let A be a n n matrix.

A is diagonalizable if and only if there exists a basis B = {v1 , . . . , vn } of Fn
consisting of eigenvectors of A. In such a case, the diagonal entries of D
are eigenvalues of A, and B can be chosen as consisting of the columns of P ,
where P 1 AP = D.
Corollary 2.1. If A has no eigenvalues, then it is not diagonalizable.
Theorem 3. Let 1 , . . . , s be pairwise distinct eigenvalues of an nn matrix
A, i.e. i 6= j, i 6= j . Let v1 , . . . , vs be eigenvalues of A corresponding to
1 , . . . , s . Then the set S = {v1 , . . . , vs } is linearly independent.
64

Proof. If possible, let S be linearly dependent. Let S 0 denote a linearly

dependent subset of S of smallest possible size, say l. WLG, let S 0 =
{v1 , . . . , vl }.
Hence, 1 , . . . , l F, s.t.
1 v1 + + l vl = O

(3.1)

Multiplying (3.1) on both sides by A,

1 Av1 + + l Avl = O
1 1 v1 + + l l vl = O

(3.2)
(3.3)

Multiplying (3.1) on both sides by l

1 1 v1 + + l Avl = O

(3.4)

Subtracting (3.4) from (3.3)

1 (1 l )v1 + + l1 (l1 l )vl1 = O

(3.5)

Solving,
1 = l = 0
This is a contradiction.
Corollary 3.1. Let Ann have n distinct eigenvalues. Then, A is diagonalizable.
Proof. Let v1 , . . . , vn be eigenvectors of A, corresponding to 1 , . . . , n . As
they are distinct, by the above theorem, they are linearly independent. The
number of elements in the set {v1 , . . . , vn } is n. Therefore, the set is a basis.
Hence, according to General criterion for diagonalization, A is diagonalizable.

Characteristic Polynomial

Definition 32 (Characteristic Polynomial). Let A be any n n matrix.

pA (x) = det(xIn A)
is called the characteristic polynomial.
65

5.1

Properties

1. The roots of pA (x) are the eigenvalues of A.

2. deg pA (x) = n
3. The coefficient of xn is 1.
4. The constant term is 0 = (1)n det(A).
5. The coefficient of xn1 is n1 = (a11 + + ann ).
Theorem 4. If A A0 , then pA (x) = pA0 (x).
Proof.
A0 = P 1 AP
pA0 (x) = det(xI A0 )
= det(xI P 1 AP )
= det(P 1 (xI)P P 1 AP )
= det(P 1 (xI A)P )


 1 ) det(xI A)
=
det(P
det(P
)
= det(xI A)
= pA (x)

Definition 33 (Alternative definition of characteristic polynomial). Let

T : V V be a linear operator. The characteristic polynomial of T is defined
as the characteristic polynomial of any representing matrix of T .
Theorem 5. Let f (x), g(x) be polynomials. Then q(x), r(x), s.t.
f (x) = g(x)q(x) + r(x)
and deg r(x) < deg g(x).
Definition 34 (Remainder). If
f (x) = g(x)q(x) + r(x)
r(x) is called the remainder after division of f (x) by g(x). If r(x) = O, f (x)
is said to be divisible by g(x).
66

Corollary 5.1. Let f (x) be a polynomial and let be a root of f . Then f (x)
is divisible by (x ).
Definition 35 (Algebraic multiplicity of eigenvalue). Let A be a nn matrix,
and let pA (x) be the characteristic polynomial of A, and let be an eigenvalue
of A. The algebraic multiplicity of is defined as the largest possible integer
value of k such that pA (x) is divisible by (x )k .
Definition 36 (Eigenspace). Let A be a n n matrix, and let be an
eigenvalue of A. The eigenspace of A corresponding to is defined as
V = {v Fn ; Av = v}
Theorem 6. An eigenspace of a matrix is a subspace of the field over which
the matrix is defined.
Definition 37 (Geometric multiplicity of eigenvalue). m = dim V is called
the geometric multiplicity of .
Theorem 7. Let be an eigenvalue of Ann . Let k be the algebraic multiplicity of and let m be the geometric multiplicity of . Then
mk
Proof.
m = dim V
Therefore, let B0 = {v1 , . . . , vm } be a basis of V .
Completing B0 to B = {v1 , . . . , vm , vm+1 , . . . , vn }, a basis of Fn .
Let Pnn be a matrix with columns v1 , . . . , vn .


P = v1 . . .

vm vm+1 . . .

vn

P is invertible as v1 , . . . , vn are linearly independent.

Consider A0 = P 1 AP .


P 1 AP = P 1 A v1 . . .

vm vm+1 . . .

vn

Avm Avm+1 . . .

vm ? . . .

= P 1 Av1 . . .
= P 1 v1 . . .


= P 1 (v1 ) . . .


= e1 . . .
!

Im ?
=
0 Ae

67

Avn

P 1 (vm ) ? . . .

em ) ? . . .

pA0 (x) = det(xIn + A0 )

Im ?
xIm
0
= det

0 xInm
0 Ae
!

(x )Im
?
= det
0
xInm Ae

e
= det((x )Im ) det(xInm A)
= (x )m pAe(x)
e
As A A,

pA (x) = pA0 = (x )m pAe(x)

By the definition of Algebraic multiplicity of eigenvalue, k m.
Theorem 8. If a matrix Ann is diagonalizable, then its characteristic polynomial pA (x) can be represented as a product of linear factors.
pA (x) = (x 1 )k1 . . . (x s )ks
where ki is the algebraic multiplicity of i and 1 , . . . , s are pairwise distinct.
Proof. As A is diagonalizable, let A D,

0
...

0
1

0
0

0
0

0
0

0
0

0
0
0
..
.

0
0

0
0

0
s

0
0

0
0

0
0
...

0
..
.

0
x 1

0
0

0
0

0
0

0
0

0
0

0
0

0
0
...
0

0
x s

0
0

0
0

0
0

0
0

0
0

0
0

0
0
..
.

Then,
pA (x) = pD (x)

det

x 1
0
0

= (x 1 )k1 . . . (x s )ks
68

0
x s

Theorem 9 (Explicit criterion for diagonalization). Let A be an n n matrix,

s.t. pA (x) splits completely. Then A is diagonalizable if and only if i of A,
the algebraic multiplicity coincides with the geometric multiplicity.
Proof of statement. If pA splits completely, then k1 + + kn = n.
If A is diagonalizable, then by the General criterion for diagonalization, there
is B = {v1 , . . . , vn }, a basis of Fn , s.t. each vi is an eigenvector of A.
Dividing v1 , . . . , vn into s groups corresponding to 1 , . . . , s , to each i , there
correspond at most mi = dim Vi eigenvectors, as they are a part of a basis
and hence linearly independent.
Therefore,
n m1 + + ms
As pA splits completely,
n = k1 + + ks
Also, ki mi
k1 + + ks = m1 + + ms
moreover, i, s.t. 1 i s,
ki = m i

Proof of converse.
i, s.t. 1 1 s
ki = mi
As k1 + + ks = n,
m1 + + ms = n

69

Let the bases of the eigenspaces V1 , . . . , Vs be B1 , . . . , Bs .

|B1 | = m1
..
.
|Bs | = ms
Let B = B1 Bs . |B| = n.
It is enough to prove that B is linearly independent.
Let
B = {v1 , v2 , . . . , w1 , w2 , . . . , u1 , u2 , . . . }
Suppose
1 v1 + 2 v2 + + 1 w1 + 2 w2 + + 1 u1 + 2 u2 + . . . = O
If possible, let at least one coefficient be non-zero. WLG, let 1 6= 0.
Hence, as v1 , v2 , . . . form B1 which is a basis of v1 ,
v = 1 v1 + 2 v2 + =
6 O
Let
w = 1 w 1 + 2 w 2 + . . .
...
u = 1 u1 + 2 u2 + . . .
Therefore,
v + w + + u = O
where v 6= O and v V1 , w V2 , . . . , u Vs .
But as 1 , . . . , s are pairwise distinct, v, w, . . . , u are linearly independent.
This is a contradiction. Therefore, B is a basis. Hence, as B consists of
eigenvectors of A, by the General criterion for diagonalization, A is diagonalizable.
Theorem 10 (Criterion for triangularization). An operator T : V V is
triangularizable, i.e. there is a basis B of V such that [T ]B is upper triangular,
if and only if pT (x) splits completely.

70

Theorem 11 (Jordan Theorem). Let T : V V be a linear operator such

that pT (x) splits completely. Then there exists a basis B of V such that [T ]B
is of the form

J
1

[T ]B =
0

0
...
0

Jl

where each Ji is of the form

0
0
1
0

.
.
0 0
.. ..

..

.
0 0 0

0 0 0
0

0 0 0
0

0 0
0 0

0 0

1 0

where is some eigenvalue of T .

Part VII

Inner Product Spaces

1

Definition

Definition 38 (Inner product). Let F be R or C. Let V be a vector space

over F. An inner product on V is a function in two vector arguments with
scalar values which associates to two given vectors v, w V their product
< v, w > F so that the following properties are satisfied.
1. h1 v1 + 2 v2 , wi = 1 hv1 , wi + 2 hv2 , wi, v1 , v2 , w V , 1 , 2 F
2. hv, wi = hw, vi, v, w V
3. hv, vi is a real non-negative number, v V
Example 12. The dot product of two vectors is defined as follows. Is it an
inner product?
V = Fn
71

1 1
.. ..
. , . = 1 1 + + n n
n

Solution. All three axioms are satisfied by this product. Hence, it is an inner
product.
Theorem 1 (Sesquilinearity).
hv, 1 w1 + 2 w2 i = 1 hv, w1 i + 2 hv, w2 i
v, w1 , w2 V, 1 , 2 F
Definition 39 (Length). The length of a vector

1
..
v= .

n
is defined to be
kvk =

12 + + n2

Example 13. Let V be the vector space consisting of all continuous functions
f : [a, b] R.
Zb

hf, gi =

f (x)g(x) dx

Solution. All three axioms are satisfied by this product. Hence, it is an inner
product.

Computation of Inner Products

Definition 40 (Gram matrix). Let V be an inner product space. Let

B = {v1 , . . . , vn }
be a basis of V .

GB =

hv1 , v1 i . . .
..
.

hv1 , vn i
..
.

hvn , v1 i . . .

hvn , vn i

is called the Gram matrix of the inner product with respect to B.

72

Example 14. Find the Gram matrix of V = Fn with standard dot product
with respect to

B=

1
0

..
..
. , . . . , .

Solution.

he , e i . . .
1 1

GB = ...

hen , e1 i . . .

hen , en i

1 ...

he1 , en i
..
.

0
..
.

..
.

0 ...

Example 15. Find the Gram matrix of V = Fn with standard dot product
with respect to
B=

!
6

3
,
4
7

Solution.
hv1 , v1 i hv1 , v2 i
GB =
hv2 , v1 i hv2 , v2 i
25 46
=
46 85

Theorem 2.
hv, wi = [v]tB GB [w]B
Proof. Let
B = {v1 , . . . , vn }
be a basis of V .
The Gram matrix is


GB = hvi , vj i = gij

73

To compute hv, wi, find

1
..
[v]B = .

n

1
..
[w]B = .

n

hv, wi = h1 v1 + + n vn , 1 v1 + + n vn i
= 1 1 hv1 , v1 i + + 1 n hv1 , vn i
+ 2 1 hv2 , v1 i + + 2 n hv2 , vn i
+ ...
+ n 1 hvn , v1 i + + n n hvn , vn i
= 1 g11 1 + + 1 g1n n
+ 2 g21 1 i + + 2 g2n n
+ ...
+ n gn1 1 + + n gnn n
= [v]tB GB [w]B

2.1

Change of Basis

Theorem 3. Let B, Be be bases of V . Let P be the transition matrix from B

e Then
to B.
GBe = P t GB P
where P is the matrix obtained by replacing all elements of P by their complex
conjugates.
Proof.
[v]B = P [v]Be

74

hv, wi = [v]tB GB [w]B

= (P [v]Be )t GB (P [w]Be )
= [v]tBe (P t GB P )[w]Be
Also,
hv, wi = [v]tBe GBe [w]Be
Therefore,
GBe = P t GB P

Norms

3.1

Definition

Definition 41 (Norm). Let V be a vector space over F with inner product.

v V ,
q
.
kvk = hv, vi
kvk is called the norm of v.

3.2

Properties

1. Positivity
kvk 0, v V
kvk = 0 v = O
2. Homogeneity
kvk = ||kvk, v V, F
3. Triangle Inequality
ku + vk kuk + kvk, u, v V

4
4.1

Orthogonality
Definition

Definition 42 (Orthogonality). A vector u V is said to be orthogonal to

v V if
hu, vi = 0
75

It is denoted as u v.

4.2

Properties

1. If u v, then v u.
2. If u v, , F, then u v.
3. O v, v V .

Orthogonal and Orthonormal Bases

Let V be a vector space over F with an inner product. Let S V .

Definition 43 (Orthogonal set). S is said to be orthogonal if any two distinct
vectors from S are orthogonal.
Definition 44 (Orthonormal set). S is said to be orthonormal if it is orthogonal and the norm of every vector is 1.
Definition 45 (Orthogonal basis). S is said to be an orthogonal basis of V
if it is orthogonal and a basis of V .
Definition 46 (Orthonormal basis). S is said to be an orthonormal basis of
V if it is orthonormal and a basis of V .
Theorem 4. Let S be an orthogonal set such that O
/ S. Then S is linearly
independent.
Proof. Let
1 , . . . , m F
v1 , . . . vm S
Let
1 v1 + + m vm = O
S is linearly independent if and only if
1 = = m = 0
1 v1 + + m vm = O
76

Multiplying both sides by v1 ,

h1 v1 + + m vm , v1 i = hO, v1 i
1 hv1 , v1 i + + m hvm , v1 i = 0
As v1 , . . . , vm are orthogonal,
hv2 , v1 i = = hvm , v1 i
1 hv1 , v1 i = 0
As v1 6= O
hv1 , v1 i =
6 0
1 = 0
Similarly,
2 = = m = 0

Corollary 4.1. Any orthonormal set is linearly independent.

Corollary 4.2. Any orthonormal set consisting of n = dim V vectors is an
orthonormal basis of V .
Example 16. Is the set
S=

1
,
1

!
1

orthonormal?
Solution. The norm of the elements of S is not 1. Hence S is not orthonormal.
Theorem
5. Let
B = {v1 , . . . , vn } be an orthonormal basis of V . Let v V .

1
..
Let [v]B = . . Then,

n
1 = hv, v1 i
..
.
n = hv, vn i
77

Proof.
v = 1 v1 + + n vn
hv, v1 i = h1 v1 + + n vn , v1 i
= 1 hv1 , v1 i + + n hvn , v1 i
= 1
Similarly, in general, 1 i n,
hv, vi i = i

Theorem 6 (Pythagoras Theorem).

Let
B = {v1 , . . . , vn } be an orthonormal

1
..
basis of V . Let v V . Let [v]B = . . Then,

n
kvk2 = |1 |2 + + |n |2
Proof.
kvk2 = hv, vi
= h1 v1 + + n vn , 1 v1 + + n vn i
= 1 1 + + n n
= |1 |2 + . . . |n |2

Unitary Matrices

Definition 47. Let F = R or F = C. Let A be an n n matrix. A is said

to be a unitary matrix if
t

A = A = At = A1
If F = R, unitary matrices are called orthogonal matrices.
1. I is a unitary matrix.
2. If A1 and A2 are unitary matrices, then (A1 A2 ) = A2 A1 .
78

3. If A is unitary, A1 is also unitary.

Theorem 7. Let A be an n n matrix. Let v1 , . . . , vn be the columns of
A. Let A be an n n matrix. Let r1 , . . . , rn be the columns of A. Then the
following are equivalent.
1. A is unitary.
2. {v1 , . . . , vn } is an orthonormal basis of Fn , with respect to standard dot
product.
3. {r1 , . . . , rn } is an orthonormal basis of Fn , with respect to standard dot
product.
Proof. As A is unitary, At is also unitary.
(At ) = (At )t
= (A )t
= (A1 )t
= (At )1
A is unitary
A = A1
AA = I
t

AA = I
t

(AA )ik = Iik

=

n
X

aij aik

j=1

= ri rk {r1 , . . . , rn } is an orthonormal basis

Theorem 8. Let V be an inner product space. Let B be an orthonormal

basis of V . Let B 0 be another basis of V . Let P be the transition matrix from
B to B 0 . Then B 0 is orthonormal if and only if P is unitary.
Proof of statement.
GB 0 = P t GB P

79

If B 0 is orthonormal,
I = P t IP
= P tP
Therefore, P is unitary.
Proof of converse. If P is unitary,
GB 0 = P t GB P
As B is orthonormal,
GB = I
GB 0 = P t P
As P is unitary,
P tP = I
GB 0 = I
Therefore, B 0 is orthonormal.

Projections

7.1

Definition

Definition 48. Let S V be a set of vectors.

.
S = {v V |hu, vi = 0u S}
Theorem 9. S is a subspace of V .
Proof.
hu, Oi = 0 O S
If v1 , v2 S ,
hu, v1 + v2 i = hu, v1 i + hu, v2 i
=0+0
=0
80

If v S ,
hu, vi = hu, vi
=0

Theorem 10.
S = span(S)
Proof. Let v S , u span(S).
Let 1 , . . . , m F, u1 , . . . , um S.
Therefore,
u = 1 u1 + + m vm
hu, vi = h1 u1 + + m vm , vi
= 1 hu1 , vi + + m hum , vi
= 1 0 + + m 0
=0
Therefore, v S .
Therefore, S span(S) .
S span(S). Therefore, let v span(S) . Then,
hu, vi = 0
for all u span(S).
Hence for all u S,
hu, vi = 0
Therefore, span(S) S .
Definition 49 (Projection). Let V be an inner product space. Let W be
a subspace of V . Let v V . Let B = {w1 , . . . , wm } be a basis of W . The
projection of v onto W is defined as follows.

B (v) =

hv, w1 i
hv, wm i
w1 + +
wm
hw1 , w1 i
hwm , wm i

81

7.2

Properties

1. B (v) W
2. B (v) = v v W
3. v B (v) W

7.3

Gram - Schmidt Process

Input
Any basis B = {v1 , . . . , vn } of V .
Intermediate Output Orthogonal basis Be = {vf1 , . . . , vfn of V
Final Output
Orthonormal basis B 0 = {v1 1 , . . . , vn 0 } of V
Step 1 vf1 = v1 , denote w1 = span{vf1 } = span{v1 }, B1 = {vf1 }
Step 2 vf2 = v2 B1 (v2 ) = v2

hv2 , vf1 i
vf1
hvf1 , vf1

As vf2 vf1 , B2 = {vf1 , vf2 } is an orthogonal set.

span{vf1 , vf2 } = span{v1 , v2 }.
Step 3 vf3 = v3 B2 (v3 ) = v3

Denote W2 =

hv3 , vf2 i
hv2 , vf1 i
vf1
hvf1 , vf1 i
hvf2 , vf2 i

As vf3 W2 , B3 = {vf1 , vf2 , vf3 } is an orthogonal set. Denote W2 =

span{vf1 , vf2 , vf3 } = span{v1 , v2 , v3 }.
..
.
g = {v
f1 , . . . , v
fn } which is an orthogonal basis of V .
Step n The nth step gives B
n
g.
B 0 is obtained by normalization of B
n

v1 0 =

1
kvf1 k

..
.
vn 0 =

1
kvfn k

82

Example 17.
B = {v1 , v2 , v3 }
=

1 0 1

1 , 2 , 1

Solution.
vf1 = v1

vf2 = v2

hv2 , vf1 i
hvf1 , vf1 i

0
1

2

= 2 1
2
0
0

= 1

0
vf3 = v3

hv3 , vf2 i
hv3 , vf1 i
vf1
vf2
hvf1 , vf1 i
hvf2 , vf2 i

1
1
1

= 1 1 0 1

2
0
1
0

f =
B
3

1
1
0


1 , 1 , 0

83

f,
Therefore, normalizing B
3

1/ 2

1/ 2

v1 0 =

v2 0

1/ 2

= 1/ 2

v3 =

B0 =

1/ 2
1/ 2

1/ 2 , 1/ 2 , 0

v2

y
v1
v2 0

v3v1

x
z

7.4

v3 0

Inequalities

Theorem 11 (Bessels Inequality). Let {v1 , . . . , vm } be an orthonormal set.

Let v V be any vector. Then
kvk2 |hv, v1 i|2 + + |hv, vm i|2
and the equality holds if and only if v span{v1 , . . . , vm }.
84

Proof. {v1 , . . . , vm } can be completed to an orthonormal basis

B = {v1 , . . . , vm , vm+1 , . . . , vn }
Using Pythagoras Theorem,
kvk2 = |hv, v1 i|2 + + |hv, vm i|2 + |hv, vm+1 i|2 + + |hv, vn i|2
kvk2 |hv, v1 i|2 + + |hv, vm i|2
The equality holds if and only if
|hv, vm+1 i|2 + + |hv, vn i|2 = 0
if and only if
|hv, vm+1 i|2 = 0
..
.
|hv, vn i|2 = 0
If v span{v1 , . . . , vm },
v = 1 v1 + + m vm
Therefore,
hv, vm+1 i = h1 v1 + + m vm , vm+1 i
= 1 hv1 , vm+1 i + + m hvm , vm+1 i
as the basis is orthonormal, hvi , vm+1 i
hv, vm+1 i = 0
Similarly,
|hv, vm+2 i|2 = 0
..
.
|hv, vn i|2 = 0
Conversely, if
|hv, vm+1 i|2 = 0
..
.
|hv, vn i|2 = 0
85

let
v = 1 v1 + + m vm + m+1 vm+1 + n vn
0 = hv, vm+1 i
0 = h1 v1 + + m vm + m+1 vm+1 + n vn , vm+1 i
All hvi , vm+1 i except hvm+1 , vm+1 i are 0.
Therefore,
|hv, vm+1 i|2 + + |hv, vn i|2 = 0
Theorem 12 (Cauchy - Schwarz Inequality). Let u, v V be any vectors.
Then
|hu, vi| kuk kvk
and the equality holds if and only if {u, v} is linearly dependent.
Proof. If u = O, the equality holds.
Let u 6= O.
Let
1
u0 =
kuk
0
ku k = 1
Applying Bessels Inequality to the orthonormal set {u0 },
2

kvk2 |hv, u0 i|
2

|hv, u0 i| =
=

*
+2



1
 v,
u 

kuk 

2

 1



hv,
ui


 kuk

!2

1
|hv, ui|
kuk
1
2
=
2 |hv, ui|
kuk
1
2
kvk2
2 |hv, ui|
kuk
=

By Bessels Inequality, the equality holds if and only if

v span{u0 } = span{u}
Therefore, v and u are linearly independent.
86

Angle

Definition 50 (Angle). Let V be a vector space over R with inner product

h, i. Let u, v V , u 6= O, v 6= O. The angle between u and v is defined as
.
cos =

hu, vi
kuk kvk

Triangle Inequality

Theorem 13 (Triangle Inequality Theorem). Let u, v V . Then

ku + vk kuk + kvk
Proof.
ku + vk2 = hu + v, u + vi
= hu, ui + hu, vi + hv, ui + hv, vi
= kuk2 + hu, vi + hu, vi + kvk2
= kuk2 + 2< hu, v + kvk2

As <(z) |z|,
ku + vk2 kuk2 + 2 hu, vi + kvk2


Hence, by Cauchy - Schwarz Inequality,

ku + vk2 kuk2 + 2kukkvk + kvk2
ku + vk2 kuk + kvk
ku + vk kuk + kvk

10

2

Orthogonal Decomposition

Theorem 14. Let W be a subspace of V . Then

V = W W

87

Proof. Let B be an orthogonal basis of V . Consider a projection B (v).

Therefore,

v = B (v) + v B (v)
B (v) W
v B (v) W
Therefore,
V = W + W

If possible, let u W W . Therefore, u W and u W . By the

definition of orthogonality,
hu W, u W i = 0
u=0
Therefore,
V = W W

Corollary 14.1. Let B be an orthogonal basis of W . Then B (v) does not

depend on the choice of B.
Proof. As B is an orthogonal basis of W ,

v = B (v) + v B (v)

Let B 0 be another orthogonal basis of W . Therefore,

v = B 0 (v) + v B 0 (v)
Therefore,
B (v) W
B 0 (v) W
and
v B (v) W
v B 0 (v) W P

88

As
V = W W
such a representation is unique. Therefore,
B (v) = B 0 (v)

Theorem 15. Let u, v V , s.t. u v. Then

ku vk2 = kuk2 + kvk2
Proof.
ku vk2 = kuk2 + kvk2
= hu, ui + hu, vi + hv, ui + hv, vi
= hu, ui + hv, vi
= kuk2 + kvk2

11

Distance

Definition 51 (Distance). Let u, v V . The distance d(u, v) from u to v is

defined as
.
d(u, v) = ku vk
Theorem 16. Let u, v V . Then
d(u, v) 0
and the equality holds if and only if u = v.
Theorem 17. Let u, v V . Then
d(u, v) = d(v, u)
Theorem 18. Let u, v V . Then
d(u, v) + d(v, w) d(u, w)
89

Theorem 19. The projection W (v) is the vector in W closest to v, i.e.

d v, W (v) = min d(v, w)

wW

Proof. Let v V . For any vector w W ,

2

d(v, u)

= kv wk2

2

= v = W (v) + W (v) w

= v W (v) 2 + W (v) w 2

v W (v) 2

2

d(v, u)

12

2

d v, W (v)

Adjoint Map

Definition 52 (Linear functional). A linear functional : V F is a linear

map, with F considered as a 1 dimensional vector space over itself.
Theorem 20 (Rieszs Representation Theorem). Let V be an inner product
space, s.t. n = dim V . Let : V F be any linear functional. Then there
exists a unique vector u V , dependent on , s.t. v V ,
(v) = hv, ui
Proof. If possible, let u1 , u2 V , s.t. v V ,
(v) = hv, u1 i = hv, u2 i
Therefore,
hv, u1 u2 i = 0
Let v = u1 u2 . Therefore,
hv, u1 u2 i = hu1 u2 , u1 u2 i
hu1 u2 , u1 u2 i = 0
u1 u2 = 0
u1 = u2

90

Therefore, u, if it exists, is unique.

Let
B = {v1 , . . . , v2 }
Be = {1}
be orthonormal bases of V and F respectively.
Let
A = []B,Be


= 1 . . .

be the representation matrix.

Therefore,
[(v)]Be = A[v]B
Let
v = 1 v1 + + n vn

1
..
[v]B = .

n
Therefore,


[(v)]Be = 1 . . .

 1
..
n .
n

= 1 1 + + n n
= 1 1 + + n n
= 1 1 + + n n

= 1 . . .

 1
..
n .
n

* 1
+
1
.. ..
. , .

91

Let u V , s.t.

1
..
[u]B = .

n

[(v)]Be = hv, ui
and
[(v)]Be = (v) 1
(v) = hv, ui

12.1

Construction

1. Let T : V W be a linear map.

2. Fix w W .
3. Let w : V F be a linear functional, s.t. w (v) = hT (v), wi.
w (1 v1 + 2 v2 ) = 1 w (v1 ) + 2 w (v2 ).
4. By Rieszs Representation Theorem, !u V , s.t. w (v) = hv, ui.
5. Define T (w) = u.
Therefore, it can be expressed as
hT (v), wi = hv, T (w)i

12.2

Properties

be an orthonorTheorem 21. Let B be an orthonormal basis of V and let B

mal basis of W . Let A = [T ]B,B be the representing matrix of T : V W with
Let A = [T ] be the representing matrix of T : W V
respect to B, B.
B,B
Then
with respect to B, B.
t
A = A = A

Theorem 22. If T1 , T2 : V W , then

(T1 + T2 ) = T1 + T2
92

Theorem 23. If T : V W , F, then

(T ) = T
Theorem 24.
(T )
Theorem 25. If T : V W , S : W U , then
(S T ) = T S

13

Special Linear Operators

Definition 53. Let T : V V be a linear operator, and let T : V V is

the adjoint operator.
T is said to be
1. normal if T T = T T
2. self-adjoint if T = T (If F = R, T is called symmeteric.)
3. unitary if T = T 1 (If F = R, T is called symmeteric.)
Remark 8. The same terminology is used for square matrices.
Remark 9. If B is orthonormal basis of V , A = [T ]B , then A is the normal,
self-adjoint or unitary according to T .
Theorem 26. Let v V . T is normal if and only if
kT (v)k = kT (v)k
Corollary 26.1. Let T : V V be normal, let be its eigenvalue, and let
v be an eigenvector of T corresponding to . Then is an eigenvalue of T ,
and v is an eigenvector of T corresponding to .
Theorem 27. If T is normal, 1 , 2 are its eigenvalues, v1 , v2 are eigenvectors corresponding to 1 , 2 respectively. If 1 6= 2 , then v1 v2 .
Theorem 28. Let T be a self-adjoint operator. Then any eigenvalue of T
is real.
Theorem 29. Let T : V V be a unitary operator. Then
1. T preserves inner products.
2. T preserves norms.
3. T preserves distances.
4. T preserves angles (in real case).
93