Lect-21

Applied Linear Algebra

3.2 Cosines And Projections Onto Lines

Course outcome: The course outcome of this article is to know about the Cosine
angle between two lines and also the projection matrix.

Inner Product and Cosines

The cosine of the angle is directly related to inner product. For this consider the
triangle in two dimensional case. Suppose the vector a and b makes an angles α and
β with x−axis as in fig. The length k a k is the hypoteneous in the triangle OaQ. So
a2 a1
the sine and cosine of α are sin α = , cos α = .
kak kak

The cosine of the angle θ = β − α using inner products. For angle β, the sine is
b2 b1
and the cosine is .
kbk kbk

The cosine formula

cos θ = cos(β − α)
= cos β cos α + sin β sin α
a1 b 1 + a2 b 2
=
k a kk b k
aT b
=
k a kk b k
ha, bi
=
k a kk b k

Law of Cosines k b − a k2 =k b k2 + k a k2 −2 k b kk a k cos θ. When θ is a right

angle, we have k b − a k2 =k b k2 + k a k2 , which is Pythagoras Theorem.
 k b − a k2 = (b − a)T (b − a)
= (bT − aT )(b − a)
= bT b − abT − aT b + aT a
= b T b − aT b − aT b + aT a
=k b k2 −2aT b+ k b k2
=k b k2 −2 k a kk b k cos θ+ k b k2 .

When θ is right angle, cos θ = 0. Thus k b − a k2 =k a k2 + k b k2 .

Projection Onto a Line

Suppose we want to find the distance from a point b to the line in the direction of the
vector a. We are looking also that line for the point p closest to b. The line connecting
b to p is perpendicular to a. The situation is the same when we are given a plane or
any subspace S instead of a time. Again the problem is to find the point P on the
subspace that is closed to b onto the subspace. Every point on the line is a multiple of
a. So,
aT b
p=x
ba, where x
b= .
aT a
Hence the projection of the vector b onto the line in the direction of a is
aT b aT b
p=x
ba = T a = a.
a a k a k2

Example 1 Project b = (1, 2, 3) onto the line through a = (1, 1, 1) to get xb and p.
T
a b 1+1×2+1×3 6
Solution: x b = T = √ = = 2. The projection is p = xba =
a a 2 2
( 1 +1 +1 ) 2 2 3
aT b 6
2(1, 1, 1) = (2, 2, 2). The angle between a and b is cos θ = =√ √ .
k a kk b k 3 14
Schwartz Inequality | ab |≤k a kk b k .

2
 Exercise-3.2

1. (a) Given any two positive numbers x and y, choose the vector b equal to
√ √ 
√ √ 
( x , y), and choose a = y , x ). Apply the Schwarz inequality
1 √
to compare the arithmetic mean (x + y) with the geometric mean xy.
2
√ 
√
 
y x
solution: If a = √  , b = √  . Applying Schwarz inequality, we
x y
get

aT b ≤k a kk b k
√ 

√ √  x
 √ √ √ √
⇒ y x √ ≤k ( y, x) kk ( x, y) k
y
√ √
q
√ 2 √ 2q √ 2 √
⇒ xy + xy ≤ ( y) + ( x) ( x) + ( y)2
√ √ √
⇒ 2 xy ≤ y + x x + y
√ x+y
⇒ xy ≤ (G.M. ≤ A.M.)
2

(b) Using the triangle inequality k x + y k≤k x k + k y k, reduce to Schwarz

inequality.
Solution: From the triangle inequality, we know

k x + y k≤k x k + k y k
⇒k x + y k2 ≤ (k x k + k y k)2
⇒ (x + y)T (x + y) ≤k x k2 +2 k x kk y k + k y k2
⇒ (xT + y T )(x + y) ≤k x k2 +2 k x kk y k + k y k2
⇒ xT x + xT y + y T x + y T y ≤k x k2 +2 k x kk y k + k y k2
⇒k x k2 +2xT y+ k y k2 ≤k x k2 +2 k x kk y k + k y k2
⇒ xT y ≤k x kk y k (Schwarz inequality).

3. By using the correct b in the Schwarz Inequality, Prove that

(a1 + a2 + · · · + an )2 ≤ n(a21 + a22 + · · · a2n ).

When does equality hold?

Solution: Let b = (1, 1, · · · , 1) and a = (a1 , a2 , · · · , an ). Using Schwarz inequal-

3
 ity, we have

aT b ≤k a kk b k
 
1
 
  1 q √
⇒ a1 a2 · · · an  .  ≤ a21 + a22 + · · · + a2n 12 + 12 + · · · + 12
 
 .. 
 
1
√ q 2
⇒ a1 + a2 + · · · an ≤ n a1 + a22 + · · · + a2n

⇒ (a1 + a2 + · · · an )2 ≤ n(a21 + a22 + · · · + a2n )

The enquality becomes equality, if ai = aj for i = j = 1, 2, . . . , n.

Assignments
Exercise-3.2, Q. 5,9.

4
 Lect-22

Projection Matrix of Rank 1

The projection onto a line is carried out by a projection matrix P which is a symmetric
aaT
matrix and P 2 = P, P = T .
a a
Example 2 The matrix that projects onto the line through a = (1, 1, 1) is
1 1 1
  3 3 3
 
1   
aaT 1  1
 
1 1

P = T =  1 1 1 1 = .
a a 3   3 3 3
1
 
 
1 1 1
 
3 3 3
Here, rank(P ) = 1.

Remark 1
aaT
1. Projection matrix is same if a is doubled a = (2, 2, 2, ),.i.e,
aT a
1 1 1

  3 3 3
 
2   
aaT 1    
1 1 1

P = T = 2 2 2 2 =  .
a a 12

3 3 3
  
2

 
1 1 1
 
3 3 3
aaT
2. To project b onto a, multiply it by the projection matrix P : p = P b, P = T
a a

5
 Exercise-3.2

aaT
8. Prove that the trace of P = , which is the sum of its diagonal entries, always
aT a
1.
Solution: Let a = (a1 a2 · · · an )T . Then

aaT
P =
aT a 

a1
 
a   
 2
 .  a1 a2 · · · an
 .. 
 
an
=  
a1
 
   a2 
a1 a2 · · · an  . 
 
 .. 
 
an
 
a21 a1 a2 · · · a1 an
 
2
a a
 1 2 a 2 · · · a 2 a n


 . .. .. .. 
 .. . . . 
 
2
an a1 an a2 · · · an
=
a21 + a22 + · · · + a2n

a21 + a22 + · · · + a2n

Hence tr(P ) = = 1.
a21 + a22 + · · · + a2n

17. Draw the projection of b onto a and also compute it from p = x

ba :

   
cos θ 1
(a) b =   and a =  
sin θ 0

6
 Solution:

P =x
ba
aT b
= a
aT a  
  cos θ
1 0   
sin θ 1
=    
  1 0
1 0  
0
 
cos θ 1
=
1 0
 
cos θ
= 
0

   
1 1
(b) b =   and a =  
1 −1

7
 Solution:

P =x
ba
aT b
= a
aT a  
  1
1 −1    
1 1
=   
  1 −1
1 −1  
−1
 
0 1
=  
2 −1
 
0
= 
0

Assignments
Exercise-3.2, Q. 11,19.

8
 Lect-23

3.3 Projections and Least Squares

A system of equations Ax = b has a solution iff b ∈ C(A).

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
.. (1)
.
am1 x1 + am2 x2 + · · · + amn xn = bm
Equation(1) can be written as
       
a11 a12 a1n b1
       
a  a  a  b 
 21   22   2n   2
 .  x1 +  .  x2 + · · ·  .  xn =  .  .
 ..   ..   ..   .. 
       
am1 am2 amn bm
Let the co-efficient vector of xi be ai . Now, ∈ C(A),i.e, there exists c1 , c2 , · · · , cn such
 b
x1
 
n x 
X  2
that b = ci ai . Then for xi = ci , x =  .  is a solution of (1).
 .. 
i=1  
xn
If x is a solution of (1), then ci = xi , i.e., b is a linear combination of columns of A,
i.e., b ∈ C(A).
Consider a system of linear equations

Ax = b (2)

with b 6∈ C(A). Then (2) has no solution. Then one must choose x
b a best fit solution
of it.

9
As b 6∈ C(A), let us obtain a column vector p in C(A), which is the most closest vector
of b in C(A) such that the error e = b − p is of minimum length .i.e,k e k2 =k b − p k2
is minimum. Hence it is called least square approximation.

k e k2 is minimum
⇐⇒ e ⊥ C(A)
⇐⇒ e ∈ N (AT )
⇐⇒ AT e = 0
⇐⇒ AT (b − p) = 0
⇐⇒ AT (b − Ab
x) = 0
⇐⇒ AT b = AT Ab
x

i.e., Normal equations AT Ab

x = AT b.
b = (AT A)−1 AT b.
Best estimate: x
x = A(AT A)−1 AT b.
Projection p = Ab
Projection matrix P = A(AT A)−1 AT .

Remark 2 1. Suppose b is actually in the column space of A it is a combination

b = Ax of the columns. Then the projection of b is still b: b in column space

p = A(AT A)−1 AT Ax = Ax = b :

The closest point p is just b itself, which is obvious.

2. At the other extreme, suppose b is perpendicular to every column, so AT b = 0. In

this case b projects to the zero vector: b in left nullspace, i.e., b ∈ N (AT )

p = A(AT A)−1 AT b = A(AT A)−1 0 = 0 :

3. When A is square and invertible, the column space is the whole space. Every
vector projects to itself, p equals b, and x
b = x : If A is invertible

p = A(AT A)−1 AT b = AA−1 (AT )−1 AT b = b :

This is the only case when we can take apart (AT A)−1 , and write it as A−1 (AT )−1 .
When A is rectangular that is not possible.

4. Suppose A has only one column, containing a. Then the matrix AT A is the
number aT a and x
b is aT b = aT a.

10
 5. AT A has the same nullspace as A.

If Ax = 0, then AT Ax = 0. Vectors x in the nullspace of A are also in the

nullspace of AT A. To go in the other direction, start by supposing that AT Ax = 0,
and take the inner product with x to show that Ax = 0 :

xT AT Ax = 0 ⇒k Ax k2 = 0 ⇒ Ax = 0.

Hence, the two nullspaces are identical.

6. If A has independent columns , then N (AT A) = N (A) = {0}.

7. If A has independent columns, then AT A is square, symmetric, and invertible.

Projection Matrices

P = Projection matrix, which projects a vector onto C(A)

= A(AT A)−1 AT

i.e, p = P b.
Note that

P 2 = A(AT A)−1 AT A(AT A)−1 AT

= A(AT A)−1 (AT A)(AT A)−1 AT
= A(AT A)−1 AT
=P

and

P T = [A(AT A)−1 AT ]T
= (AT )T [(AT A)−1 ]T AT
= A(AT A)−1 AT
= P.

Suppose that A is invertible. Then,

P = A(AT A)−1 AT
= AA−1 (AT )−1 AT
= I.

Hence, p = P b = Ib = b. Therefore, error = p − b = 0.

11
    
1 0 1
   
1. Solve Ax = b by least squares and find p = Ab x, if A = 
0 1 and 1. Verify
  

1 1 0
that the error b − p is perpendicular to the columns of A.
Solution:

b = (AT A)−1 AT b
x
 
   1
1 2 −1 1 0 1  
=    1
3 −1 2 0 1 1
 
0
1
3
= 
 
1
 
3
and

P = Ab
x
= (AT A)−1 AT b
  
1 0 1
 3
  
=0 1   1 
1 1 3
1
3
 
 
1
 
=  .
3
 
 
2
 
3
1  2 

   3  3 
   
1    
   1  2 
  
Here, Error= b − p =  1 −
  3  =  .
  3 
0
   
   
2 −2
   
3 3
b − p is perpendicular to thecolumn
 of the matrix.
  1
2 2 −2   2 2
(b − p)T C1 = 0 = − = 0 and
3 3 3   3 3
1

12
  
  0
2 2 −2   2 2
(b − p)T C2 = 1 = − = 0.
  3 3
3 3 3
1
   
1 −1   4
  C  
4. The following system has no solution: Ax =  1 0 
   = 5 = b.
D
 
1 1 9
Sketch and solve a straight-line fit that leads to the minimization of the quadratic
(C − D − 4)2 + (C − 5)2 + (C + D − 9)2 . What is the projection of b onto the
column space of A?
Solution:

b = (AT A)−1 AT b
x
   
1  4
0 1 1 1  
= 3 1   5

−1 0 1
 
0
2 9
 1 1 1  
 3 3 3  4
=  5
  
−1 1
 
0 9
 2 2
6
= 5 .
2
Now,

P = Ab
x
 
1 −1  
  6
= 1 0  5

1 1 2
7
2
=  6 .
 
17
 
2

Assignments
Exercise-3.3, Q. 2,9,12,24.

13
 Lect-24

Least Squares Fitting a Straight Line

b = C + Dt

or

C + Dt1 = b1
C + Dt2 = b2
..
.
C + Dtm = bm

   
1 t1   b1
   
1 t  C b 
2  2
or  . =  .  or Ax = b, x b = (C,
b D).
   b
 ..  .. 

 D
   
1 tm bm
m
X
E 2 =k b − Ax k2 = (bi − C − Dti )2 , where k E k2 is minimum for some C and D.
i=1

∂E 2
=0 (3)
∂C

∂E 2
=0 (4)
∂D

Solving (3) and (4) for C and D we get the required straight line

b = C + Dt.

14
or
 
1 t1
 
1 t 
2
AT Ab
x = AT b, A =  .
 
 ..


 
1 tm
 
C
b
⇐⇒ AT A   = AT b
Db
   
  1 t1   b1 
  
1 1 ... 1  1 t2   C 1 1 ... 1   b2 
 b 
⇐⇒   .  =    . 
. . 

t1 t2 . . . tm  .
 D

b t1 t2 . . . tm  . 
1 tm bm
 Xm   Xm 
m t bi 
 
 i  C 
i=1   b   i=1 
⇐⇒ X = X

m m  m 

ti
X
2
ti D
b 
ti bi

i=1 i=1 i=1

15
 Exercise-3.3

6. Find the projection of b onto the column space of A :

   
1 1 1
   
A=
1 ,
−1 b=
2 .


−2 4 7

Split b into p + q, with p in the column space and q perpendicular to that space.
Which of the four subspaces contains q?
Solution:

p = Ab
x
18 8
 
   
1 1  44 44    1
 1 1 −2  

 
=
 1 −1  8

6
  2
 1 −1 4
 

−2 4  44 44  7

 26 14 
 44 44   
   1
 
 10

2  1
 1 −2  
=   2
 44 44  −1 4
 1
 
7

 
−4 8
 
44 44
 40 12 4 
 44 44 44  
 
  1
 12 8 −12  
  
=  2
 44 44 44   
 7
 

4 −12 40
 
44 44 44
 
92
1  
= −56 .
44  
260

  
92 1
1    
Let find the vector q such that b = p + q ⇒ q = p − b = −56 − 2 =
44    
260 7

16
      
48 48 1 1 0
1  
−144 . Thus, q A = T 1     
−144  1 −1 = 0. Hence, q belongs
44   44     
−48 −48 −2 4 0
to the left null space.

Assignments
Exercise-3.3, Q. 12,24.

17
 Lect-25

4.2 Properties of the Determinant

   
 
a b a b 
1. det  =  = ad − bc.
c d  c d
 

 
 
a b c a b c 
 












    e f  d f  d e 
2. det   − b
d e f  = d e f  = a 
     + c 
h i  g i   g h
    
 
g h i g h i 
 
  1 0 0
   
1 0   
3. The determinant of the identity matrix is 1, eg-

 = 1 and 0 1 0 = 1
  
0 1   
0 0 1
   
   
a b   b a
4. The determinant changes sign when two rows are interchanged,i.e.,  = −
   .
 c d d c 


   
 0 0   
a + a b + b  a b 
5. The determinant depends linearly on the first row, i.e.,  = +
 c d   c d
  
     
 0 0    
a b  ta tb a b 
  and   = t .
 c d  c d  c d
     

 
 
a a 
6. If any two rows of A are equal then det(A) = 0, eg, 
  = 0.
c c


7. Substracting a multiple of one row from another row leaves the same determinant.

8. If A is singular then det(A) = 0 and if A is nonsingular then det(A) 6= 0.

9. det(AB) = det(A)det(B)

10. det(A) = det(AT )

18
 Exercise-4.2

4. By applying row operations to produce an upper triangular U, compute

 
1 2 −2 0
 
2 3 −4 1 
(i) det  .
 
−1 −2 0 2
 
0 2 5 3
Solution:

   
1 2 −2 0 1 2 −2 0
   
   
2 3 −4 1  0 −1 0 1
 = R2 → R2 − 2R1 , R3 → R3 + R1
   
 
−1 −2 0 2 0 0 −2 1
   
   
0 2 5 3   0 2 5 3
 
1 2 −2 0
 
 
0 −1 0 1
= R4 → R4 + 2R2


0 0 −2 2

 
0 0 5 5
 
1 2 −2 0
 
 
0 −1 0 1  5
= R4 → R4 + R3

2

0 0 −2 2 

 
0 0 0 10

= 1 × (−1) × (−2) × 10 = 20.

 
2 −1 0 0
 
−1 2 −1 0 
(ii) det  .
 
 0 −1 2 −1
 
0 0 −1 −2

19
 Solution:
   
0  2 −1 0 0

 2 −1 0
 
3
  
−1 2 −1 0  0 −1 0 

1

=
  2  R2 → R2 + R1
2

 0 −1 2 −1 0 −1 2 −1

  
 
0 0 −1 −2 0 0 −1 −2

 
2
 −1 0 0 
 3 
0 −1 0 
 
2  R3 → R3 + 2
=  4 R2
0 0

−1 3

 3 
0 0 −1 −2
 
2 −1 0 0
 
3
 
−1 0 
 
0 3
=
 2  R4 → R4 +

R3
4 4
0 0 −1
 3

0 5 
0 0
4
 
3 4 5
= 2 × × × = 5.
2 3 4

5. Find the determinants of:

 
1
  
(a) a rank one matrix A = 
4 2 −1
 2
2
 
4 4 8 8
 
0 1 2 2
(b) the upper triangular matrix U = 
 

0 0 2 6
 
0 0 0 2

(c) the lower triangular matrix U T .

(d) the inverse matrix U −1 .

 
0 0 0 2
 
0 0 2 6
(e) the reverse-triangular matrix that results from row exchanges, M = 
 

0 1 2 2
 
4 4 8 8

Solution:

20
      
1   2 −1 2 2 −1 2
 
     
(a) A =  4
 
 2 −1 2 = 8 −4 8 . Hence, det(A) = 8 −4 8 =
   
 
2 4 −2 4  4 −2 4
 
2
 −1 2
 
0
 0 0 = 0
 
0 0 0
(b) | U |= 4 × 1 × 2 × 2 = 16

(c) | U T |=| U |= 16.

1 1
(d) | U −1 |= = .
|U | 16
 
0 0 0 2
 
 
0 0 2 6
(e) | M |=   = 4 × 1 × 2 × 2 = 16.
 
0 1 2 2
 
 
4 4 8 8

Assignments
Exercise-4.2, Q. 2,6,13.

21