Taylor’s Theorem and Numerical Derivative

MAT307 Numerical Analysis Forward and Backward Difference Formula

Central Difference Formula

MAT307 Numerical Analysis

Numerical Differentiation - Part I

Dr. Mustafa Ağgül

[email protected]

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

The derivative of a function f (x) at a point x = x0 is defined by

f (x0 + h) − f (x0 )
f 0 (x0 ) = lim
h→0 h
Using this formula for ”small enough” h values, the formula
f (x0 + h) − f (x0 )
f 0 (x0 ) ≈
h
is simply an approximation to the numerical derivative of f (x) at x = x0 .
Similarly, derivative at a point x = x0 can be defined by approaching from
either the right or left. Also, the derivative from the left can be given by
f (x0 ) − f (x0 − h)
f 0 (x0 ) = lim
h→0 h
In this case, an approximation to the derivative of f (x) at x = x0 is

f (x0 ) − f (x0 − h)
f 0 (x0 ) ≈
h

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

As can be seen from our last two approaches, convergence to numerical

derivative does not occur with a single formula. We can use the Taylor
series expansion of the f (x) function to further generalize these formulas.
In the most general sense, Taylor series expansion of the function f (x)
around a point x0 :

f 0 (x0 ) f 00 (x0 ) f (n) (x0 )

f (x) = f (x0 )+(x−x0 ) +(x−x0 )2 +· · ·+(x−x0 )n +. . .
1! 2! n!
Substituting h = x − x0 in the above equation1 , we obtain

f 0 (x0 ) f 00 (x0 ) f (n) (x0 )

f (x0 + h) = f (x0 ) + h + h2 + · · · + hn + ...
1! 2! n!
According to Taylor’s Remainder Theorem, if we truncate this series from
any term, the error of the remaining part corresponds to a value that the
next expression can take in the range [x0 , x0 + h].2
1
WLOG assume x > x0 .
2
This value is attained at an arbitrary number in the range [x0 , x0 + h].
Dr. Mustafa Ağgül Hacettepe University
 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Consider the approximation

f (x0 + h) ≈ f (x0 ) + hf 0 (x0 ).

f 00 (c)
The error in this approximation is h2 , for some c ∈ [x0 , x0 + h].
2!
Substituting this error back in the approximation, we get an exact
equation instead:
f 00 (c)
f (x0 + h) = f (x0 ) + hf 0 (x0 ) + h2 .
2!
0
Solving this equation for f (x0 ) gives
f (x0 + h) − f (x0 ) f 00 (c)
f 0 (x0 ) = +h .
h 2!
Definition (Forward Difference Formula)
f (x0 + h) − f (x0 )
f 0 (x0 ) ≈ is an approximation to the derivative with
00
h
f (c)
an error h .
2!

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Similarly, substituting h = x0 − x, we obtain

f 0 (x0 ) f 00 (x0 ) f (n) (x0 )

f (x0 − h) = f (x0 ) − h + h2 − · · · + (−1)n hn +...
1! 2! n!
Truncating this series after the second term gives, for some
c ∈ [x0 − h, x0 ],
f 0 (x0 ) f 00 (c)
f (x0 − h) = f (x0 ) − h + h2 .
1! 2!
Again solving for f 0 (x0 ) gives
f (x0 ) − f (x0 − h) f 00 (c)
f 0 (x0 ) = +h .
h 2!
Definition (Backward Difference Formula)
f (x0 ) − f (x0 − h)
f 0 (x0 ) ≈ is an approximation to the derivative with
00
h
f (c)
an error h .
2!

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Theorem (The error due to the forward and backward

difference formulas)
Let the function f be twice continuously differentiable function. Also let
the second derivative attain its maximum value M on the interval
[x0 , x0 + h], i.e., M= max |f 00 (x)|.
x∈[x0 ,x0 +h]
00
Then, f (c) ≤ M , for all c ∈ [x0 , x0 + h], and hence, the error due to
forward/backward difference formula has an upper bound:

f 00 (c) M
Error = h ≤h .
2 2

Remark
If we know the maximum value of the second derivative of the function
on the interval [x0 , x0 + h], we can control the error by scaling h values.

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Now, let us consider both series expansions at the same time.

000
f 0 (x0 ) f 00 (x0 ) f (x0 ) f (4) (c1 )
f (x0 + h) = f (x0 ) + h + h2 + h3 + h4
1! 2! 3! 4!
000
f 0 (x0 ) f 00 (x0 ) f (x0 ) f (4) (c2 )
f (x0 − h) = f (x0 ) − h + h2 − h3 + h4
1! 2! 3! 4!
Subtracting the second series from the first, we obtain
000
f 0 (x0 ) f (c)
f (x0 + h) − f (x0 − h) = 2h + 2h3 .
1! 3!
One more time, solving for f 0 (x0 ) gives
f (x0 + h) − f (x0 − h) f 000 (c)
f 0 (x0 ) = + h2 .
2h 3!
Definition (Central Difference Formula)
f (x0 + h) − f (x0 − h)
f 0 (x0 ) ≈ is an approximation to the derivative
000
2h
f (c)
with an error h2 .
3!
Dr. Mustafa Ağgül Hacettepe University
 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Theorem (The error due to the central difference formula)

Let the function f be three times continuously differentiable function.
Also let the third derivative attain its maximum value M on the interval
[x0 − h, x0 + h], i.e.,
M= max |f 000 (x)|.
x∈[x0 −h,x0 +h]

Then, f 000 (c) ≤ M , for all c ∈ [x0 − h, x0 + h], and hence, the error due
to central difference formula has an upper bound:

f 000 (c) M
Error = h2 ≤ h2 .
3! 3!

Remark
If we know the maximum value of the third derivative of the function on
the interval [x0 − h, x0 + h], we can control the error by scaling h values.

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Remark
The error due to the forward/backward difference formula is given by
some constant times a small number h, while it is some other constant
times h2 for the central difference formula. Neglecting the constants, and
comparing the numbers h and h2 , we observe that h2 < h for
0 < h << 1, and hence, the error due to the central difference is much
less than the error caused by the forward/backward difference.
Technically, the error due to the forward/backward difference formula is
said to be O(h) (first order of accuracy), while it is O(h2 ) (second order
of accuracy) with the central difference3 .

3
The forward and backward difference formulas are O(h) accurate. It means
that when the value of h is halved, the error will reduce by half, as well. Also,
since the central difference formula is O(h2 ) accurate, cutting the value of h to
its half will reduce the error to its quarter.
Dr. Mustafa Ağgül Hacettepe University
 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Example
Find the forward, backward and central difference approximations of the
derivative of f (x) = x3 , at x0 = 1. Take h = 0.1 and h = 0.05, and
compute absolute errors and upper bound of the error for each.
For h = 0.1:
f (1 + h) − f (1) f (1.1) − f (1)
Forward Dif. f 0 (1) ≈ =
h 0.1
1.331 − 1
= = 3.31
0.1
f (1) − f (1 − h) f (1) − f (0.9)
Backward Dif. f 0 (1) ≈ =
h 0.1
1 − 0.729
= = 2.71
0.1
f (1 + h) − f (1 − h) f (1.1) − f (0.9)
Central Dif. f 0 (1) ≈ =
2h 0.2
1.331 − 0.7291
= = 3.01
0.2

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Example
Find the forward, backward and central difference approximations of the
derivative of f (x) = x3 , at x0 = 1. Take h = 0.1 and h = 0.05, and
compute absolute errors and upper bound of the error for each.
For h = 0.05:
f (1 + h) − f (1) f (1.05) − f (1)
Forward Dif. f 0 (1) ≈ =
h 0.05
1.157625 − 1
= = 3.1525
0.05
f (1) − f (1 − h) f (1) − f (0.95)
Backward Dif. f 0 (1) ≈ =
h 0.05
1 − 0.857375
= = 2.8525
0.05
f (1 + h) − f (1 − h) f (1.05) − f (0.95)
Central Dif. f 0 (1) ≈ =
2h 0.1
1.157625 − 0.857375
= = 3.0025
0.1

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Example
The exact value of the derivative at x0 = 1 is needed for the accuracy
analysis:
f 00 (x) = 3x2 ⇒ f 0 (1) = 3.

Abs. Error h = 0.1 h = 0.05

Forward Difference |3 − 3.31| = 0.31 |3 − 3.1525| = 0.1525
Backward Difference |3 − 2.71| = 0.29 |3 − 2.8525| = 0.1475
Central Difference |3 − 3.01| = 0.01 |3 − 3.0025| = 0.0025

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Example
For the upper bounds of errors, we need the second and third derivatives
of f (x) = x3 :
f 00 (x) = 6x and f 000 (x) = 6.

Bound for Error h = 0.1 h = 0.05

Forward Difference (6.6)(0.1)/2 = 0.335 (6.3)(0.05)/2 = 0.1575
Backward Difference (6)(0.1)/2 = 0.3 (6)(0.05)/2 = 0.15
Central Difference (6)(0.1)2 /6 = 0.01 (6)(0.05)2 /6 = 0.0025

Dr. Mustafa Ağgül Hacettepe University

 Taylor’s Theorem and Numerical Derivative
MAT307 Numerical Analysis Forward and Backward Difference Formula
Central Difference Formula

Remark
As can be seen from the tables, the reduction of the h value by half cuts
the errors caused by the forward and backward difference formulas by
half. On the other hand, the central difference formula reduced its error
to 1/4 when the h value reduced to its half. These computation results
numerically verify our theoretical findings.

Dr. Mustafa Ağgül Hacettepe University