Kathmandu University

Dhulikhel, Kavre

Environmental Modelling

ENVS 404

Lab Report – 3: Model Evaluation

Submitted by: Submitted to:

Ashim Sharma Dr. Kundan Lal Shrestha
Roll no. 25 Department of Environmental
ENE 4th year Science and Engineering

3rd August, 2021

Model Evaluation

Introduction

Model Evaluation is the subordinate part of the model development process. It is the phase
in which it is decided whether the model performs better. Without model evaluation, we
would have no idea of how different from reality the model simulation would be, we cannot
be certain that the model behaves in the way we think it should and we cannot determine
which parts of the model are most likely to affect the results. There are four different
methods for model evaluation. They are:

I. Graphical analysis
a) Plots to reveal the accuracy of the simulation should
 show the result that is needed from the model
 include the simulated and measured values on the same plot
 show errors in the measurements as error bars and variations in simulations as
error bars or as a band of potential results
 use axes that highlight the acceptable error un the simulation
 avoid confusing and complicated presentation
b) Plots to illustrate the behavior and importance if the model component should
 only include as much complexity as can be interpreted by the eye
 include any measurements that are available
 show the importance of the components as changes in the distribution of the
results
II. Quantitative Analysis
a. A thorough quantitative analysis should include both
 an analysis of coincidence (difference)
 an analysis of association (trends)
b. A quantitative analysis compares the simulation to the independent measurements.
c. Coincidence can be expressed as
 the total difference (root mean squared error)
 the bias (relative error or mean difference)
  the error excluding error due to variations in the measurements (lack of fit)
d. The significance of coincidence can be determined by
 a comparison to the 95% confidence interval, or
 a direct comparison to the P value obtained in the t test or the F test
e. The association can be expressed as the sample correlation coefficient.
f. The significance of the association can be determined using a t-test.
III. Sensitivity Analysis
a. Common types of sensitivity analysis are
 a one-at-a-time sensitivity analysis (input varied one by one)
 a factorial analysis (more than one parameter adjusted at one time)
b. Sensitivity may be expressed
 as the correlation coefficient between the inputs and the outputs
 by regression analysis
 as the sensitivity coefficient
 as the sensitivity index
IV. Uncertainty Analysis
a. Common types of uncertainty analysis are
 a one-at-a-time uncertainty analysis
 a differential analysis
b. A variation in the input parameters is represented
 as a fixed percentile about the mean
 using a probability density function and Monte Carlo sampling
c. Uncertainty can be expressed
 as the importance index
 as the relative deviation ratio
Problems

1. Crop Yield Using Nitrogen Fertilizer

Mean measured values was plotted against Modelled values in Microsoft excel to obtain the
above graph. This scatter plot has a R2 value of 0.645. This graph helps us to determine the
accuracy of our simulation. We have also calculated the coincidence and association values
in Microsoft Excel.

Co-incidence:

Total difference= -63.67

Total size of difference= 639.667

Average size of difference = 14.876

Average square of the size of the differences=1077.36

Root mean squared deviation=32.8231

Root mean squared error=20.8807

Standard error=5.71533
Total Bias= -639.67

Association

Sample correlation coefficient, 𝑟 = 0.80

𝑟2= 0.64

Significance of association, 𝐹= 4.08

Based on the above data, we can say that this model provides a good estimate for the
optimum nitrogen application rate. Also, the value of R2 is 0.64 and closer the value of R2 to
1, the more simulated value is equal to measured value.
From the above graphs we can say that the model is quite consistent for spring wheat and
shows good coincidence and association values. It is also consistent for winter wheat as the
simulated values show good association but the coincidence is reduced. The model is
inconsistent for winter oilseed rape and the simulated values show lower coincidence and
association than both spring wheat and winter wheat.
Here, series 1 represents the plot between crop yield and measured value of optimum
nitrogen and series 2 represents the plot between crop yield and the modelled value of
optimum nitrogen.

Overall, the model is quite okay but improvement can be made to reduce the errors. The
simulations for spring wheat and winter wheat are significantly associated and coincident
with the measured values. The simulations for winter oilseed rape are neither highly
coincident nor associated with the measured values.
2. Population Growth

Examine the behavior of a simple model of the growth of a rabbit population in a warren
during summer. The number of rabbits is estimated using the following model:

𝑅 = 𝑅𝑠𝑡𝑎𝑟𝑡 + exp(𝑘1 ∗ 𝑡) + exp(𝑘2 ∗ 𝑡)

Results

The change in rabbit population for 10 weeks was plotted using Python. From the above
graph, we can see that the population of rabbit increases exponentially with time.

Sensitivity Analysis
For sensitivity analysis, the values of Rstart, k1 and k2 were changed within the range ± 20%
with step size of 5%. The sensitivity index for Rstart, k1 and k2 are calculated to be 0.27, 0.39
and 0.002 respectively. From the above graphs we can see that rabbit population is sensitive
to the changes in value of Rstart and k1 but it is not sensitive to the changes in k2. The model
can be simplified by removing the exponential expression including k2 as changing it has no
significant impact on the model.
Source code

import numpy as np

import matplotlib.pyplot as plt

r_start = 500 #measured no of. rabbits

k1 = 0.5

k2 = 0.02

t = np.linspace(0,10,20)

r_cal =[]

for i in t:

r= r_start + np.exp(k1*i) + np.exp(k2*i)

r_cal.append(r)

plt.plot(t,r_cal,'purple')

plt.ylabel('No. of Rabbits')

plt.xlabel('Time (in weeks)')

plt.show()

Source code for Sensitivity Analysis

a. sensitivity for change in R_start

import numpy as np

import matplotlib.pyplot as plt

r_start = 500

k1 = 0.5

k2 = 0.02

t = 10

ch = np.arange(-20,21,5) #percetange change in r_start

r_cal=[]
for i in ch:

r_ch=r_start +(i/100)*r_start

r= r_ch + np.exp(k1*t) + np.exp(k2*t)

r_cal.append(r)

plt.plot(ch,r_cal,'purple')

plt.ylabel('No. of Rabbits')

plt.xlabel('Change in R_start(%)')

plt.show()

#Sensitivity index

s_index = (max(r_cal)-min(r_cal))/max(r_cal)

print('Sensitivity index for R_start=',s_index)

b. Sensitivity for change in k1

import numpy as np

import matplotlib.pyplot as plt

r_start = 500

k1 = 0.5

k2 = 0.02

t = 10

ch = np.arange(-20,21,5) #percetange change in k1

r_cal=[]

for i in ch:

k1_ch= k1 +(i/100)*k1

r= r_start + np.exp(k1_ch*t) + np.exp(k2*t)

r_cal.append(r)

plt.plot(ch,r_cal,'purple')
plt.ylabel('No. of Rabbits')

plt.xlabel('Change in k1 (%)')

plt.show()

#Sensitivity index

s_index = (max(r_cal)-min(r_cal))/max(r_cal)

print('Sensitivity index for k1=',s_index)

c. Sensitivity for change in k2

import numpy as np

import matplotlib.pyplot as plt

r_start = 500

k1 = 0.5

k2 = 0.02

t = 10

ch = np.arange(-20,21,5) #percetange change in k2

r_cal=[]

for i in ch:

k2_ch= k2 +(i/100)*k2

r= r_start + np.exp(k1*t) + np.exp(k2_ch*t)

r_cal.append(r)

plt.plot(ch,r_cal,'purple')

plt.ylabel('No. of Rabbits')

plt.ylim(500, 660)

plt.xlabel('Change in k2 (%)')

plt.show()

#Sensitivity index
s_index = (max(r_cal)-min(r_cal))/max(r_cal)

print('Sensitivity index for k2=',s_index)