Time Series Forecasting Performance Measures With Python

01/10/2019 Time Series Forecasting Performance Measures With Python
Navigation
Click to Take the FREE Time Series Crash-Course
Search...
Time Series Forecasting Performance Measures

With Python
by Jason Brownlee on February 1, 2017 in Time Series
Tweet Share Share
Last Updated on August 21, 2019
Time series prediction performance measures provide a summary of the skill and capability of the
forecast model that made the predictions.
There are many different performance measures to choose from. It can be confusing to know which
measure to use and how to interpret the results.
In this tutorial, you will discover performance measures for evaluating time series forecasts with
Python.
Time series generally focus on the prediction of real values, called regression problems. Therefore the
performance measures in this tutorial will focus on methods for evaluating real-valued predictions.
After completing this tutorial, you will know:
Basic measures of forecast performance, including residual forecast error and forecast bias.
Time series forecast error calculations that have the same units as the expected outcomes such as
mean absolute error.
Widely used error calculations that punish large errors, such as mean squared error and root mean
squared error.
Discover how to prepare and visualize time series data and develop autoregressive forecasting models
in my new book, with 28 step-by-step tutorials, and full python code.
Let’s get started.
Jun/2019: Fixed typo in forecast bias (thanks Francisco).
Your Start in Machine Learning
https://machinelearningmastery.com/time-series-forecasting-performance-measures-with-python/ 1/14
Your Start in Machine ×

Learning
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
Email
Time Series Forecasting Performance Address
Measures With Python
Photo by Tom Hall, some rights reserved.
START MY EMAIL COURSE

Forecast Error (or Residual Forecast Error)
The forecast error is calculated as the expected value minus the predicted value.
This is called the residual error of the prediction.
1 forecast_error = expected_value - predicted_value
The forecast error can be calculated for each prediction, providing a time series of forecast errors.
The example below demonstrates how the forecast error can be calculated for a series of 5 predictions
compared to 5 expected values. The example was contrived for demonstration purposes.
1 expected = [0.0, 0.5, 0.0, 0.5, 0.0]

2 predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
3 forecast_errors = [expected[i]-predictions[i] for i in range(len(expected))]
4 print('Forecast Errors: %s' % forecast_errors)
Running the example calculates the forecast error for each of the 5 predictions. The list of forecast
errors is then printed.
1 Forecast Errors: [-0.2, 0.09999999999999998, -0.1, -0.09999999999999998, -0.2]
The units of the forecast error are the same as the units of the prediction. A forecast error of zero
indicates no error, or perfect skill for that forecast.

Stop learning Time Series Forecasting the slow way!
Take my free 7-day email course and discover how to get started (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Start Your FREE Mini-Course Now!
Mean Forecast Error (or Forecast Bias)

Mean forecast error is calculated as the average of the forecast error values.
1 mean_forecast_error = mean(forecast_error)
Forecast errors can be positive and negative. This means that when the average of these values is
Learning
calculated, an ideal mean forecast error would be zero.
A mean forecast error value other than zero suggests without
a tendency
mathof
orthe
fancymodel to over forecast
degrees.
(negative error) or under forecast (positive error). As such, thehow
Find out mean forecast
in this error
free and is also called the
practical course.
forecast bias.
Email Address
The forecast error can be calculated directly as the mean of the forecast values. The example below
demonstrates how the mean of the forecast errors can be calculated manually.
1 expected = [0.0, 0.5, 0.0, 0.5, 0.0]
2 predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
3 forecast_errors = [expected[i]-predictions[i] for i in range(len(expected))]
4 bias = sum(forecast_errors) * 1.0/len(expected)
5 print('Bias: %f' % bias)
Running the example prints the mean forecast error, also known as the forecast bias.
In this case the result is negative, meaning that we have over forecast.
1 Bias: -0.100000
The units of the forecast bias are the same as the units of the predictions. A forecast bias of zero, or a
very small number near zero, shows an unbiased model.
Mean Absolute Error

The mean absolute error, or MAE, is calculated as the average of the forecast error values, where all of
the forecast values are forced to be positive.
Forcing values to be positive is called making them absolute. This is signified by the absolute function
abs() or shown mathematically as two pipe characters around the value: |value|.
1 mean_absolute_error = mean( abs(forecast_error) )
Where abs() makes values positive, forecast_error is one or a sequence of forecast errors, and mean()
calculates the average value.
We can use the mean_absolute_error() function from the scikit-learn library to calculate the mean
absolute error for a list of predictions. The example below demonstrates this function.
1 from sklearn.metrics import mean_absolute_error

2 expected = [0.0, 0.5, 0.0, 0.5, 0.0]
3 predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
4 mae = mean_absolute_error(expected, predictions)
5 print('MAE: %f' % mae)
Running the example calculates and prints the mean absolute error for a list of 5 expected and
predicted values.
1 MAE: 0.140000
These error values are in the original units of the predicted values. A mean absolute error of zero
indicates no error.
Mean Squared Error Learning
The mean squared error, or MSE, is calculated as the average of the squared forecast error values.
Squaring the forecast error values forces them to be positive; it also has the effect of putting more
weight on large errors.
Very large or outlier forecast errors are squared, which Email

in turnAddress
has the effect of dragging the mean of
the squared forecast errors out resulting in a larger mean squared error score. In effect, the score gives
worse performance to those models that make large wrong STARTforecasts.
MY EMAIL COURSE
1 mean_squared_error = mean(forecast_error^2)
We can use the mean_squared_error() function from scikit-learn to calculate the mean squared error for
a list of predictions. The example below demonstrates this function.
1 from sklearn.metrics import mean_squared_error

2 expected = [0.0, 0.5, 0.0, 0.5, 0.0]
3 predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
4 mse = mean_squared_error(expected, predictions)
5 print('MSE: %f' % mse)
Running the example calculates and prints the mean squared error for a list of expected and predicted
values.
1 MSE: 0.022000
The error values are in squared units of the predicted values. A mean squared error of zero indicates
perfect skill, or no error.
Root Mean Squared Error

The mean squared error described above is in the squared units of the predictions.
It can be transformed back into the original units of the predictions by taking the square root of the
mean squared error score. This is called the root mean squared error, or RMSE.
1 rmse = sqrt(mean_squared_error)
This can be calculated by using the sqrt() math function on the mean squared error calculated using the
mean_squared_error() scikit-learn function.
1 from sklearn.metrics import mean_squared_error

2 from math import sqrt
3 expected = [0.0, 0.5, 0.0, 0.5, 0.0]
4 predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
5 mse = mean_squared_error(expected, predictions)
6 rmse = sqrt(mse)
7 print('RMSE: %f' % rmse)
Running the example calculates the root mean squared error.
1 RMSE: 0.148324
The RMES error values are in the same units as the predictions. As with the mean squared error, an
RMSE of zero indicates no error. Your Start in Machine ×
Learning
Further Reading
Below are some references for further reading on timewithout
series math
forecast errordegrees.
or fancy measures.
Section 3.3 Measuring Predictive Accuracy, Practical Time Series Forecasting with R: A Hands-On
Guide.
Email Address
Section 2.5 Evaluating Forecast Accuracy, Forecasting: principles and practice
scikit-learn Metrics API
Section 3.3.4. Regression metrics, scikit-learn API START
GuideMY EMAIL COURSE
Summary
In this tutorial, you discovered a suite of 5 standard time series performance measures in Python.
Specifically, you learned:
How to calculate forecast residual error and how to estimate the bias in a list of forecasts.
How to calculate mean absolute forecast error to describe error in the same units as the
predictions.
How to calculate the widely used mean squared error and root mean squared error for forecasts.
Do you have any questions about time series forecast performance measures, or about this tutorial
Ask your questions in the comments below and I will do my best to answer.
Want to Develop Time Series Forecasts with Python?

Develop Your Own Forecasts in Minutes
...with just a few lines of python code
Discover how in my new Ebook:
Introduction to Time Series Forecasting With Python
It covers self-study tutorials and end-to-end projects on topics like: Loading

data, visualization, modeling, algorithm tuning, and much more...
Finally Bring Time Series Forecasting to

Your Own Projects
Skip the Academics. Just Results.
SEE WHAT'S INSIDE

Learning
Tweet Share Share
About Jason Brownlee

Email Address
Jason Brownlee, PhD is a machine learning specialist who teaches developers how to get results
with modern machine learning methods via hands-on tutorials.
View all posts by Jason Brownlee START MY EMAIL COURSE
How to Decompose Time Series Data into Trend and Seasonality How to Work Through a Time Series Forecast Project
34 Responses to Time Series Forecasting Performance Measures With

Python
REPLY
Peter Marelas February 1, 2017 at 2:24 pm #
I’ve seen MAPE used a few times to evaluate our forecasting models. Do you see this used
often and when would you use one over the other?
REPLY
Jason Brownlee February 2, 2017 at 1:55 pm #
Hi Peter, MAPE is a good metric and I do see it used.
I prefer RMSE myself.
REPLY
Ian February 3, 2017 at 3:21 am #
First line of code of Forecast Error should be forecast_error = expected_value “-”

predicted_value.
I believe this is a typo.
REPLY
Jason Brownlee February 3, 2017 at 10:10 am #
Yes, that was a typo. Fixed. Thanks Ian.

Learning
REPLY
Jasem February 6, 2017 at 9:39 pm #
Dr.Jason, You can master applied Machine Learning

Can you provide us simple way to split the data with 10 fold cross validation for train and test set with
large csv file. Then apply different algorithms to train model after that test model to check how does
model accurate. we also want to see ROC curve to combine different algorithms
Email Address
my second question does ROC curve show precision of model?? can you show me a mathematical
formula for ROC curve?
REPLY
Sorry, I do not have the capacity to prepare this example for you.
You can learn more about ROC curves here:

http://machinelearningmastery.com/assessing-comparing-classifier-performance-roc-curves-2/
REPLY
Devakar Kumar Verma August 8, 2017 at 6:44 pm #
What should be range of values for all different measures of performance for a acceptable
model.
REPLY
Jason Brownlee August 9, 2017 at 6:25 am #
Good question, it really depends on your problem and the units of your variable.
REPLY
Devakar Kumar Verma August 9, 2017 at 2:14 pm #
Suppose, variable values range from Your

0-100, theninwhat
Start will beLearning
Machine range?
REPLY
If you have accuracy scores between 0 and 100, maybe 60% is good because the
problem is hard, maybe 98% is good because the problem is easy.
I cannot answer this question generically, sorry.
A good way to figure out if a model is skillful is to compare it to a lot of other models or
against a solid base line model (e.g. relative measure of good).
Irati October 10, 2017 at 8:43 pm # Your Start in Machine ×

REPLY
Hi Jason, Learning
And what about if we perform multivariate time series You can master applied Machine Learning
forecasting?
Imagine we forecast 3 time series with the same model,without math or
how would youfancy degrees.
provide the results? per time
series? the mean of the errors ? Find out how in this free and practical course.
Thanks for your time

Email Address
START MY EMAIL COURSE REPLY

Jason Brownlee October 11, 2017 at 7:52 am #
You can decide how to evaluate the skill of the model, perhaps RMSE across all forecasted
data points.
See this post for multivariate inputs:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
See this post for multi-step forecast:

https://machinelearningmastery.com/multi-step-time-series-forecasting-long-short-term-memory-
networks-python/
REPLY
Carlos May 19, 2018 at 1:45 am #
Good evening, one question , if i want to get max error, how could it be?
REPLY
Jason Brownlee May 19, 2018 at 7:43 am #
What do you mean by max error?
REPLY
Kate August 15, 2018 at 11:33 pm #
Hi, thanks for the post. If I understand correctly, the method mentioned here is useful for
correcting predictions if the ground truths of the test examples are readily available and are included in
the correction process. I was wondering if there are similar approaches for situations where there is a
noticeable trend for residuals in your training/testing data, and I’d like to create a model utilizing these
trends in an environment where ground truths for new examples are not available?
REPLY
ARIMA and ETS models can handle the trend in your data.

REPLY
Parth Gadoya October 31, 2018 at 6:32 pm #
Learning
Hi Sir,
I am forecasting sales for each product on each retail store. I want the accuracy of more than 70% on
85% of store-product combination. So, I am calculating Absolute Percentage Error for each forecast.
But I have lots of zeros and I am unable to evaluate the model completely.
According to my internet search, I found that Mean Absolute Scaled Error is a perfect measure for sales
Email Address
forecasting. But I didn’t found any concrete explanations on how to use and calculate it. As I am
working with multiple stores and multiple products, I have multiple time series in dataset. I have all the
predictions but don’t know how to evaluate? START MY EMAIL COURSE
Please give some details on how to do this and calculate MASE for multiple time series.
Thank you very much in advance.
REPLY
Jason Brownlee November 1, 2018 at 6:04 am #
Sorry, I don’t have material on MASE.
Perhaps search on scholar.google.com for examples?
REPLY
Parth Gadoya November 2, 2018 at 5:49 pm #
Thanks for the suggestion.
REPLY
Carla December 7, 2018 at 1:58 am #
Hi,
in “Statistical ans Machine Learning Forecasting Methods: Concerns and ways forward” by Spyros,
Makridakis, they used this Code for sMAPE. Add this as def and use it in the same way as you use
mse. I assume it should work.
1 def sMAPE(a, b):
2 """
3 Calculates sMAPE
4 :param a: actual values
5 :param b: predicted values
6 :return: sMAPE
7 """
8 a = np.reshape(a, (-1,))
9 b = np.reshape(b, (-1,))
10 return np.mean(100*2.0 * np.abs(a - b) / (np.abs(a) + np.abs(b))).item()
REPLY
bobby November 7, 2018 at 9:03 am #
Hi,
Your
Do you know any error metrics that punish longer lasting errorsStart
in time in Machine
series more than large magnitude
×
errors? Learning
thanks,
bobby
Jason Brownlee November 7, 2018 at 2:46 pmEmail

#
Address REPLY
What do you mean longer lasting errors?

REPLY
Daniël Muysken December 4, 2018 at 12:06 am #
Hey, I was wondering if you know of an error measure that is not so sensitive to outliers? I have
some high peaks in my timeseries that are difficult to predict and I want this error to not carry to much
weight, when evaluating my prediction.
REPLY
Jason Brownlee December 4, 2018 at 6:03 am #
Perhaps Median Absolute Error?
REPLY
Atharva February 4, 2019 at 5:50 pm #
hey, can u tell me that how can i know the accuracy of my model from rmse value
REPLY
You cannot calculate accuracy for a regression problem, I explain this more here:
https://machinelearningmastery.com/classification-versus-regression-in-machine-learning/
REPLY
Abid Mehmood February 20, 2019 at 8:53 pm #
How to know which error(RMSE,MSE, MAE) can we use in our time series predictions?
REPLY
You can talk to project stakeholders and discover what they would like to know about the
performance of a model on the problem – then choose a metric accordingly.
If unsure, use RMSE as the units will be in the scale of the target variable and it’s easy to
understand. Your Start in Machine ×
Learning
REPLY
Dav June 8, 2019 at 2:59 am # without math or fancy degrees.
Hi
Email Address
Once again, great articles and sorry, I just asked you a question on another topic as well.
Tracking Error = Standard deviation of difference between Actual and Predicted values
I am thinking about using Tracking Error to measure Time Series Forecasting Performance. Any reason I
shouldn’t use it?
Thanks
Dav
REPLY
Jason Brownlee June 8, 2019 at 7:03 am #
I’m not familiar with it, sorry.
REPLY
Francisco June 28, 2019 at 5:59 pm #
Hi Jason,
I’m confused with the Forecast bias: “A mean forecast error value other than zero suggests a tendency
of the model to over forecast (positive error) or under forecast (negative error)”
actual – prediction > 0 if prediction is below, and it’d understand that’s under forecast, but in your
example the bias is negative and the prediction is above:
expected = [0.0, 0.5, 0.0, 0.5, 0.0]

predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
Is there a mistake somewhere or maybe I’m missing or not understanding something?
Thanks a lot Your Start in Machine Learning
REPLY
Jason Brownlee June 29, 2019 at 6:45 am #
Yes, I have it the wrong way around, thanks.
Negative is over forecast, positive is under forecast.
Fixed.
REPLY
duderino July 20, 2019 at 5:52 am #
Your
I really enjoyed reading your post, thank you for Start
this. one in Machine
question if I may: ×
Learning
let’s say we are working with a dataset when you are forecasting population growth (number of people)
and your dataset’s most recent value shows roughly 37mil population.
Assuming we do all of the forecasting and calculationswithout
correctly,
mathand
orI fancy
(we) are currently sitting at
degrees.
Mean Absolute Error: 52,386
Mean Squared Error: 3,650,276,091
Root Mean Squared Error: 60,417 Email Address
(and just for fun) Mean Absolute Percentage Error: 0.038
How does one interpret these numbers when working with a dataset
START of this
MY EMAIL scale? I’ve read that “closer
COURSE
to zero is best” but I feel like the size of my dataset means that 60,417 is actually a pretty good number,
but I’m not sure.
(not sure if this is enough data to go off of or not)
REPLY
Jason Brownlee July 20, 2019 at 10:58 am #
A model has skill if it outperforms a naive forecast:

https://machinelearningmastery.com/faq/single-faq/how-to-know-if-a-model-has-good-
performance
Does that help?
Leave a Reply
Name (required)
Email (will not be published) (required)

Website Learning
SUBMIT COMMENT Find out how in this free and practical course.
Email Address
Welcome! I'm Jason Brownlee PhD and I help developers get results with machine learning.
Read More
Picked for you:
How to Create an ARIMA Model for Time Series Forecasting in Python
How to Convert a Time Series to a Supervised Learning Problem in Python
Time Series Forecasting as Supervised Learning
11 Classical Time Series Forecasting Methods in Python (Cheat Sheet)
How To Backtest Machine Learning Models for Time Series Forecasting
Loving the Tutorials?

The Time Series Forecasting with Python EBook is where I keep the Really Good stuff.
SEE WHAT'S INSIDE
© 2019 Machine Learning Mastery Pty. Ltd. All Rights Reserved.

Address: PO Box 206, Vermont Victoria 3133, Australia. | ACN: 626 223 336.
RSS | Twitter | Facebook | LinkedIn
Privacy | Disclaimer | Terms | Contact | Sitemap | Search

Learning
Email Address

Time Series Forecasting Performance Measures With Python

Uploaded by

Copyright:

Available Formats

Time Series Forecasting Performance Measures With Python

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Time Series Forecasting Performance Measures With Python

Uploaded by

Copyright:

Available Formats

01/10/2019 Time Series Forecasting Performance Measures With Python

Click to Take the FREE Time Series Crash-Course

Time Series Forecasting Performance Measures

Tweet Share Share

Last Updated on August 21, 2019

After completing this tutorial, you will know:

Let’s get started.

Jun/2019: Fixed typo in forecast bias (thanks Francisco).

Your Start in Machine Learning

Your Start in Machine ×

START MY EMAIL COURSE

This is called the residual error of the prediction.

1 forecast_error = expected_value - predicted_value

1 expected = [0.0, 0.5, 0.0, 0.5, 0.0]

1 Forecast Errors: [-0.2, 0.09999999999999998, -0.1, -0.09999999999999998, -0.2]

Your Start in Machine Learning

Start Your FREE Mini-Course Now!

Mean Forecast Error (or Forecast Bias)

Mean Absolute Error

1 mean_absolute_error = mean( abs(forecast_error) )

1 from sklearn.metrics import mean_absolute_error

Very large or outlier forecast errors are squared, which Email

1 from sklearn.metrics import mean_squared_error

Root Mean Squared Error

1 from sklearn.metrics import mean_squared_error

Running the example calculates the root mean squared error.

Speciﬁcally, you learned:

Want to Develop Time Series Forecasts with Python?

Your Start in Machine Learning

It covers self-study tutorials and end-to-end projects on topics like: Loading

Finally Bring Time Series Forecasting to

SEE WHAT'S INSIDE

Your Start in Machine ×

About Jason Brownlee

34 Responses to Time Series Forecasting Performance Measures With

Hi Peter, MAPE is a good metric and I do see it used.

I prefer RMSE myself.

Your Start in Machine Learning

First line of code of Forecast Error should be forecast_error = expected_value “-”

Yes, that was a typo. Fixed. Thanks Ian.

Your Start in Machine ×

Dr.Jason, You can master applied Machine Learning

You can learn more about ROC curves here:

Suppose, variable values range from Your

I cannot answer this question generically, sorry.

Irati October 10, 2017 at 8:43 pm # Your Start in Machine ×

Thanks for your time

START MY EMAIL COURSE REPLY

See this post for multivariate inputs:

See this post for multi-step forecast:

What do you mean by max error?

Your Start in Machine ×

Thank you very much in advance.

Sorry, I don’t have material on MASE.

Perhaps search on scholar.google.com for examples?

Thanks for the suggestion.

Jason Brownlee November 7, 2018 at 2:46 pmEmail

What do you mean longer lasting errors?

Perhaps Median Absolute Error?

I’m not familiar with it, sorry.