INTRODUCTION TO PROBABILITY

AND STATISTICS
FOURTEENTH EDITION

Chapter 13
Multiple Regression
Analysis
 INTRODUCTION
 We extend the concept of simple linear regression as we
investigate a response y which is affected by several
independent variables, x1, x2, x3,…, xk.
 Our objective is to use the information provided by the
xi to predict the value of y.
EXAMPLE
 Let y be a student’s college achievement,
measured by his/her GPA. This might be a
function of several variables:
 x1 = rank in high school class
 x2 = high school’s overall rating
 x3 = high school GPA
 x4 = SAT scores
 We want to predict y using knowledge of
x1, x2, x3 and x4.
EXAMPLE
 Let y be the monthly sales revenue for a
company. This might be a function of
several variables:
 x1 = advertising expenditure
 x2 = time of year
 x3 = state of economy
 x4 = size of inventory
 We want to predict y using knowledge of
x1, x2, x3 and x4.
SOME QUESTIONS
 How well does the model fit?
 How strong is the relationship between y and the
predictor variables?
 Have any assumptions been violated?
 How good are the estimates and predictions?

We collect information using n observations on

the response y and the independent variables,
x1, x2, x3, …xk.
THE GENERAL
LINEAR MODEL
 y = b0+ b1x1 + b2x2 +…+ bkxk + e
 where
✓ y is the response variable you want to predict.
✓ b0, b1, b2,..., bk are unknown constants
✓ x1, x2,..., xk are independent predictor variables,
measured without error.

T HE RANDOM ERROR
The deterministic part of the model,
E(y) = b0+ b1x1 + b2x2 +…+ bkxk ,
 describes average value of y for any fixed values of
x1, x2,..., xk . The population of measurements is
generated as y deviates from the line of means
by an amount e.
We assume
✓ e are independent
✓ Have a mean 0 and common variance s2 for any
set x1, x2,..., xk .
✓ Have a normal distribution.
EXAMPLE
 Consider the model E(y) = b0+ b1x1 + b2x2
 This is a first order model (independent
variables appear only to the first power).
 b0 = y-intercept = value of E(y) when
x1=x2=0.
 b1 and b2 are the partial regression
coefficients—the change in y for a one-unit
change in xi when the other independent
variables are held constant.
 Traces a plane in three dimensional space.
 THE METHOD OF
LEAST SQUARES
 The best-fitting prediction equation is
calculated using a set of n measurements
(y, x1, x2 ,… xk) as
yˆ = b0 + b1 x1 + ... + bk xk
 We choose our estimates b0, b1,…, bk to
estimate b0, b1,…, bk to minimize

SSE = ( y − y )
ˆ 2

= ( y − b0 − b1 x1 − ... − bk xk ) 2
EXAMPLE
A computer database in a small community contains
the listed selling price y (in thousands of dollars),
the amount of living area x1 (in hundreds of square
feet), and the number of floors x2, bedrooms x3, and
bathrooms x4, for n = 15 randomly selected
residences currently on the market.
Property y x1 x2 x3 x4 Fit a first order
1 69.0 6 1 2 1 model to the data
2 118.5 10 1 2 2 using the method
of least squares.
3 116.5 10 1 3 2
… … … … … …
15 209.9 21 2 4 3
 E XAMPLE
The first order model is
E(y) = b0+ b1x1 + b2x2 + b3x3 + b4x4
fit using Minitab with the values of y and
the four independent variables entered into
five columns of the Minitab
Regression equation
worksheet.
Regression Analysis: ListPrice versus SqFeet, NumFlrs, Bdrms, Baths
The regression equation is
ListPrice = 18.8 + 6.27 SqFeet - 16.2 NumFlrs - 2.67 Bdrms + 30.3 Baths

Predictor Coef SE Coef T P

Constant 18.763 9.207 2.04 0.069
SqFeet 6.2698 0.7252 8.65 0.000
NumFlrs -16.203 6.212 -2.61 0.026
Bdrms -2.673 4.494 Partial
-0.59 0.565
Baths 30.271 6.849 4.42 0.001
regression
coefficients
THE ANALYSIS OF VARIANCE
 The total variation in the experiment is
measured by the total sum of squares:
Total SS= S yy = ( y − y ) 2

The Total SS is divided into two parts:

✓SSR (sum of squares for regression):
measures the variation explained by using
the regression equation.
✓SSE (sum of squares for error):
measures the leftover variation not
explained by the independent variables.
THE ANOVA TABLE
Total df = n -1 Mean
Squares k MSR = SSR/k
Regressionndf
–1 =– k = n – k -1 MSE = SSE/(n-k-1)
Error df =
Source df SS MS F
Regression k SSR SSR/k MSR/MSE
Error n – k -1 SSE SSE/(n-k-1)
Total n -1 Total SS
THE REAL ESTATE
PROBLEM
Another portion of the Minitab printout
shows the ANOVA Table, with n = 15
and k = 4.
MSE
S = 6.84930 R-Sq = 97.1% R-Sq(adj) = 96.0%

Analysis of Variance
Source DF SS MS F P
Regression 4 15913.0 3978.3 84.80 0.000
Residual Error 10 469.1
Sequential
46.9
Sums of
Total 14 16382.2 squares: conditional
Source DF Seq SS
contribution of each
SqFeet 1 14829.3 independent variable to
NumFlrs 1 0.9
Bdrms 1 166.4
SSR given the variables
Baths 1 916.5 already entered into the
model.
 TESTING THE USEFULNESS
OF THE MODEL
• The first question to ask is whether the
regression model is of any use in predicting y.
• If it is not, then the value of y does not change,
regardless of the value of the independent
variables, x1, x2 ,…, xk. This implies that the
partial regression coefficients, b1, b2,…, bk are
all zero.
H 0 : b1 = b 2 = ... = b k = 0 versus
H a : at least one b i is not zero
THE F TEST
 You can test the overall usefulness of the
model using an F test. If the model is
useful, MSR will be large compared to
the unexplained variation, MSE.

To test H 0 : model is useful in predicting y is equivalent to

H 0 : b1 = b 2 = ... = b k = 0
MSR
Test Statistic : F =
MSE
Reject H 0 if F  F with k and n - k-1 df .
 MEASURING THE
STRENGTH
OF THE RELATIONSHIP
• If the independent variables are useful in
predicting y, you will want to know how well
the model fits.
• The strength of the relationship between x
and y can be measured using:
Multiple coefficient of determination :
SSR
R =
2

Total SS
TESTING THE PARTIAL
REGRESSION COEFFICIENTS
• Is a particular independent variable useful
in the model, in the presence of all the
other independent variables? The test
statistic is function of bi, our best estimate
of bi.
H 0 : b i = 0 versus H a : b i  0
bi − 0
Test statistic : t =
SE(bi )
which has a t distribution with error df = n – k –
1.
THE REAL ESTATE
PROBLEM
Is the overall model useful in predicting list
price? How much of the overall variation in
the response is explained by the
regression model?
S = 6.84930 R-Sq = 97.1% R-Sq(adj) = 96.0%

Analysis of Variance
Source DF SS MS F P
Regression 4 15913.0 3978.3 84.80 0.000
Residual Error 10 469.1 46.9
2
R = .971 indicates that
Total 14 16382.2
F = MSR/MSE = 84.80 with
97.1%
Source of the
DFoverall
Seq SS p-value = .000 is highly
SqFeet 1 14829.3
variation
NumFlrs
is explained
1 0.9
significant. The model is
byBdrms
the regression
1 166.4 very useful in predicting the
Baths 1 916.5
model. list price of homes.
MEASURING THE STRENGTH
OF THE RELATIONSHIP
• Since Total SS = SSR + SSE, R2 measures
✓ the proportion of the total variation in the
responses that can be explained by using the
independent variables in the model.
✓ the percent reduction the total variation by
using the regression equation rather than just
using the sample mean y-bar to estimate y.
2
SSR MSR R /k
R =
2
and F = =
Total SS MSE (1 − R 2 ) /(n − k − 1)
THE REAL ESTATE PROBLEM
In the presence of the other three independent
variables, is the number of bedrooms significant in
predicting the list price of homes? Test using  = .05.
Regression Analysis: ListPrice versus SqFeet, NumFlrs, Bdrms, Baths
The regression equation is
ListPrice = 18.8 + 6.27 SqFeet - 16.2 NumFlrs - 2.67 Bdrms + 30.3 Baths

Predictor Coef SE Coef T P

Constant 18.763 9.207 2.04 0.069
SqFeet 6.2698 0.7252 8.65 0.000
NumFlrs -16.203 6.212 -2.61 0.026
Bdrms -2.673 4.494 -0.59 0.565
Baths 30.271 6.849 4.42 0.001

To test H0: b3 = 0, the test statistic is t = -0.59 with p-value = .565.

The p-value is larger than .05 and H0 is not rejected.
We cannot conclude that number of bedrooms is a valuable predictor in the presence of
the other variables.
Perhaps the model could be refit without x3.
COMPARING
REGRESSION MODELS
• The strength of a regression model is
measured using R2 = SSR/Total SS. This
value will only increase as variables are
added to the model.
• To fairly compare two models, it is better to
use a measure that has been adjusted
using df:
 MSE 
R (adj) = 1 −
2
 100%
 Total SS/( n-1) 
CHECKING THE
REGRESSION ASSUMPTIONS
• Remember that the results of a regression
analysis are only valid when the necessary
assumptions have been satisfied.
✓ e are independent
✓ Have a mean 0 and common variance
s2 for any set x1, x2,..., xk .
✓ Have a normal distribution.
DIAGNOSTIC TOOLS
• We use the same diagnostic tools
used in Chapter 11 and 12 to check
the normality assumption and the
assumption of equal variances.
1. Normal probability plot of
residuals
2. Plot of residuals versus fit or
residuals versus variables
NORMAL PROBABILITY PLOT
✓ If the normality assumption is valid,
the plot should resemble a straight
line, sloping upward to the right.
✓ If not, you will often see the pattern
fail in the tails of the graph.
Normal Probability Plot of the Residuals
(response is List Price)
99

95
90

80
70
Percent

60
50
40
30
20

10

1
-15 -10 -5 0 5 10 15
Residual
 RESIDUALS VERSUS FITS
✓ If the equal variance assumption is valid,
the plot should appear as a random
scatter around the zero center line.
✓ If not, you will see a pattern in the
residuals. Residuals Versus the Fitted Values
(response is List Price)

10

5
Residual

-5

-10

-15
50 75 100 125 150 175 200
Fitted Value
ESTIMATION AND
PREDICTION
• Once you have
✓ determined that the regression line is
useful
✓ used the diagnostic plots to check for
violation of the regression assumptions.
• You are ready to use the regression line to
✓ Estimate the average value of y for
a given value of x
✓ Predict a particular value of y for a
given value of x.
ESTIMATION AND
PREDICTION
• Enter the appropriate values of x1, x2, …, xk
in Minitab. Minitab calculates

yˆ = b0 + b1 x1 + b2 x2 + ... + bk xk
• and both the confidence interval and the
prediction interval.
• Particular values of y are more difficult to
predict, requiring a wider range of values
in the prediction interval.
THE REAL ESTATE PROBLEM
 Estimate the average list price for a home with 1000
square feet of living space, one floor, 3 bedrooms and
two baths with a 95% confidence interval.

We estimate that the average

list price will be between
$110,860 and $124,700 for a
home like this.
Predicted Values for New Observations
New Obs Fit SE Fit 95% CI 95% PI
1 117.78 3.11 ( 110.86, 124.70) ( 101.02, 134.54)

Values of Predictors for New Observations

New Obs SqFeet NumFlrs Bdrms Baths
1 10.0 1.00 3.00 2.00
USING REGRESSION MODELS
When you perform multiple regression analysis,
use a step-by step approach:
1. Obtain the fitted prediction model.
2. Use the analysis of variance F test and R 2 to
determine how well the model fits the data.
3. Check the t tests for the partial regression
coefficients to see which ones are contributing
significant information in the presence of the others.
4. If you choose to compare several different models,
use
R 2(adj) to compare their effectiveness.
5. Use diagnostic plots to check for violation of the
regression assumptions.
A POLYNOMIAL MODEL
• A response y is related to a single independent
variable x, but not in a linear manner. The
polynomial model is:
y = b 0 + b1 x + b 2 x + ... + b k x + e
2 k

 When k = 2, the model is quadratic:

y = b 0 + b1 x + b 2 x + e
2

• When k = 3, the model is cubic:

y = b 0 + b1 x + b 2 x + b 3 x + e
2 3
EXAMPLE
A market research firm has observed the sales (y) as a
function of mass media advertising expenses (x) for 10
different companies selling a similar product.
Company 1 2 3 4 5 6 7 8 9 10
Expenditure, x 1.0 1.6 2.5 3.0 4.0 4.6 5.0 5.7 6.0 7.0

Sales, y 2.5 2.6 2.7 5.0 5.3 9.1 14.8 17.5 23.0 28.0

Scatterplot of y vs x
30 Since there is only one
25
independent variable, you
20
could fit a linear, quadratic,
15
y

10
or cubic polynomial model.
5
Which would you pick?
0
1 2 3 4 5 6 7
x
 TWO POSSIBLE CHOICES
A straight line model: y = b0 + b1x + e
A quadratic model: y = b0 + b1x + b2x2 + e
Here is the Minitab printout for the straight line:
Overall F test is highly
significant, as is the t-test of
the slope. R2 = .856 suggests
Regression Analysis: y versus x a good fit. Let’s check the
The regression equation is residual plots…
y = - 6.47 + 4.34 x
Predictor Coef SE Coef T P
Constant -6.465 2.795 -2.31 0.049
x 4.3355 0.6274 6.91 0.000
S = 3.72495 R-Sq = 85.6% R-Sq(adj) = 83.9%
Analysis of Variance
Source DF SS MS F P
Regression 1 662.46 662.46 47.74 0.000
Residual Error 8 111.00 13.88
Total 9 773.46
 EXAMPLE
Residuals Versus the Fitted Values
(response is y)
5.0
There is a strong pattern of
2.5
a “curve” leftover in the
residual plot.
Residual

0.0

-2.5 This indicates that there is

-5.0
a curvilinear relationship
0 5 10 15 20 25 unaccounted for by your
Fitted Value
straight line model.
You should have used the
Use Minitab to fit the quadratic model!
quadratic model:
y = b0 + b1 x + b2 x 2 + e
THE QUADRATIC MODEL

Regression Analysis: y versus x, x-sq

The regression equation is
y = 4.66 - 3.03 x + 0.939 x-sq

Predictor Coef SE Coef T P

Constant 4.657 2.443 1.91 0.098
x -3.030 1.395 -2.17 0.067
x-sq 0.9389 0.1739 5.40 0.001
S = 1.75204 R-Sq = 97.2% R-Sq(adj) = 96.4%

Analysis of Variance
Source DF SS MS F P
Regression 2 751.98 375.99 122.49 0.000
Residual Error 7 21.49 3.07
Total 9 773.47

Overall F test is highly significant, as is the t-test of the quadratic

term b2. R2 = .972 suggests a very good fit.
Let’s compare the two models, and check the residual plots.
WHICH MODEL TO USE?
Use R2(adj) to compare the models:
The straight line model: y = b0 + b1x + eR 2 (adj) = 83.9%
The quadratic model: y = b0 + b1x + b2x2 + e
Residuals Versus the Fitted Values
R 2 (adj) = 96.4%
(response is y)
3

2
The quadratic model is
1
better.
Residual

-1
There are no patterns in
the residual plot,
-2
indicating that this is the
0 5 10 15 20 25 30
Fitted Value correct model for the
data.
USING QUALITATIVE
VARIABLES
• Multiple regression requires that the
response y be a quantitative variable.
• Independent variables can be either
quantitative or qualitative.
• Qualitative variables involving k
categories are entered into the model by
using k-1 dummy variables.
• Example: To enter gender as a
variable, use
• xi = 1 if male; 0 if female
EXAMPLE
Data was collected on 6 male and 6 female
assistant professors. The researchers recorded their
salaries (y) along with years of experience (x1). The
professor’s gender enters into the model as a
dummy variable: x2 = 1 if male; 0 if not.
Professor Salary, y Experience, x1 Gender, x2 Interaction, x1x2
1 $50,710 1 1 1
2 49,510 1 0 0
… … … … …
11 55,590 5 1 5
12 53,200 5 0 0
EXAMPLE
We want to predict a professor’s salary based on
years of experience and gender. We think that
there may be a difference in salary depending on
whether you are male or female.
The model we choose includes experience (x1),
gender (x2), and an interaction term (x1x2) to
allow salary’s for males and females to behave
differently.

y = b 0 + b1 x1 + b 2 x2 + b 3 x1 x2 + e
MINITAB OUTPUT What is the regression
equation
Is the for useful
overall model males?in For
We use Minitab to fit females?
thepredicting
model. y?
The overall F test is F = 346.24
For males, x2 = 1,
Regression Analysis: y versus x1, x2, x1x2 with p-value = .000. The value of
The regression equation is R2 =y.992
= 49459.7 + 1229.13x
indicates that the
1
y = 48593 + 969 x1 + 867 x2 + 260 x1x2 model fits very well.
For females, x2 = 0,
Predictor Coef SE Coef T P
Constant 48593.0 207.9 y = 48593.0
233.68 0.000+ 969.0x1
x1 969.00 63.67 15.22 0.000
x2 866.7 305.3 Two
2.84different
0.022 straight line
x1x2 260.13 87.06 2.99 0.017
models.
S = 201.344 R-Sq = 99.2% R-Sq(adj) = 98.9%
Is there a difference in the relationship between salary and
Analysis of Variance
years of experience,
Source DF depending
SS on the gender
MS of the
F P
professor?
Regression 3 42108777 14036259 346.24 0.000
Residual Error 8 324315 40539
Yes. The individual11t-test42433092
Total for the
interaction term is t = 2.99
with p-value = .017. This indicates a significant interaction
between gender and years of experience.
EXAMPLE
Have any of the regression assumptions been
violated, or have we fit the wrong model?
Residuals Versus the Fitted Values Normal Probability Plot of the Residuals
(response is y) (response is y)
300 99

200 95
90

100 80
70
Residual

Percent
60
0 50
40
30
-100
20

10
-200
5

-300
49000 50000 51000 52000 53000 54000 55000 56000
1
-400 -300
It does not appear from
-200 -100 0 100 200 300 400 500
Fitted Value
the diagnostic plots that
Residual

there are any violations

of assumptions.
The model is ready to be
used for prediction or
estimation.
TESTING SETS OF
PARAMETERS
• Suppose the demand y may be related to five
independent variables, but that the cost of measuring
three of them is very high.
• If it could be shown that these three contribute little or
no information, they can be eliminated.
• You want to test the null hypothesis
• H0 : b3 = b4 = b5 = 0—that is, the independent variables
x3, x4, and x5 contribute no information for the
prediction of y—versus the alternative hypothesis:
• Ha : At least one of the parameters b3, b4, or b5
differs from 0 —that is, at least one of the variables x3,
x4, or x5 contributes information for the prediction of y.
TESTING SETS OF
PARAMETERS
• To explain how to test a hypothesis concerning a
set of model parameters, we define two models:
• Model One (reduced model)
E ( y ) = b 0 + b1 x1 + b 2 x2 +  + b r xr

Model Two (complete model)

E ( y ) = b 0 + b1 x1 + b 2 x2 +  + b r xr + b r +1 xr +1 + b r + 2 xr + 2 +  + b k xk
• terms in model 1 additional terms in model 2
TESTING SETS OF
PARAMETERS
The test of the hypothesis
H0 : b3 = b4 = b5 = 0
Ha : At least one of the bi differs from 0
uses the test statistic
F=
(SSE1 − SSE 2 ) (k − r )
MSE 2

where F is based on df1 = (k - r ) and df2 =

n -(k + 1).
The rejection region for the test is identical to
other analysis of variance F tests, namely F > F.
STEPWISE
✓
R EGRESSION
A stepwise regression analysis fits a variety
of models to the data, adding and deleting
variables as their significance in the
presence of the other variables is either
significant or nonsignificant, respectively.
✓ Once the program has performed a sufficient
number of iterations and no more variables
are significant when added to the model, and
none of the variables are nonsignificant
when removed, the procedure stops.
✓ These programs always fit first-order
models and are not helpful in detecting
curvature or interaction in the data.
SOME CAUTIONS
✓ Causality: Be careful not to deduce a
causal relationship between a response y
and a variable x.
✓ Multicollinearity: Neither the size of a
regression coefficient nor its t-value
indicates the importance of the variable as
a contributor of information. This may be
because two or more of the predictor
variables are highly correlated with one
another; this is called multicollinearity.
MULTICOLLINEARITY
✓ Multicollinearity can have these effects
on the analysis:
✓ The estimated regression coefficients
will have large standard errors,
causing imprecision in confidence and
prediction intervals.
✓ Adding or deleting a predictor variable
may cause significant changes in the
values of the other regression
coefficients.
MULTICOLLINEARITY
✓ How can you tell whether a regression analysis
exhibits multicollinearity?
✓ The value of R 2 is large, indicating a good
fit, but the individual t-tests are
nonsignificant.
✓ The signs of the regression coefficients are
contrary to what you would intuitively expect
the contributions of those variables to be.
✓ A matrix of correlations, generated by the
computer, shows you which predictor
variables are highly correlated with each
other and with the response y.
KEY CONCEPTS
I. The General Linear Model
1. y = b 0 + b1 x1 + b 2 x2 + + b k xk + e
2. The random error e has a normal distribution with
mean 0 and variance s2.

II. Method of Least Squares

1. Estimates b 0, b 1, …, b k for b 0, b 1, …, b k , are
chosen to minimize SSE, the sum of squared
deviations about the regression
yˆ = b0 +line
b1 x1 + b2 x2 + + bk xk .
2. Least-squares estimates are produced by computer.
KEY CONCEPTS
III. Analysis of Variance
1. Total SS = SSR + SSE, where Total SS = Syy.
The ANOVA table is produced by computer.
2. Best estimate of s2 is

SSE
MSE =
n − k −1
IV. Testing, Estimation, and Prediction
1. A test for the significance of the regression,
H0 : b1 = b2 =  = bk = 0, can be implemented using
the
analysis of variance F test:
MSR
F=
MSE
KEY CONCEPTS
2. The strength of the relationship between x
and y can be measured using
SSR
R =
2

Total SS
which gets closer to 1 as the relationship
gets stronger.
3. Use residual plots to check for
nonnormality, inequality of variances, and
an incorrectly fit model.
4. Significance tests for the partial regression
coefficients can be performed using the
Student’s t test with error d f = n − k − 1: bi − b i
t=
SE (bi )
KEY CONCEPTS
5. Confidence intervals can be generated by
computer to estimate the average value of
y, E(y), for given values of x1, x2, …, xk.
Computer-generated prediction intervals
can be used to predict a particular
observation y for given value of x1, x2, …, xk.
For given x1, x2, …, xk, prediction intervals
are always wider than confidence intervals.
KEY CONCEPTS
V. Model Building
1. The number of terms in a regression model cannot
exceed the number of observations in the data set and
should be considerably less!
2. To account for a curvilinear effect in a quantitative
variable, use a second-order polynomial model. For a
cubic effect, use a third-order polynomial model.
3. To add a qualitative variable with k categories, use (k
− 1) dummy or indicator variables.
4. There may be interactions between two qualitative
variables or between a quantitative and a qualitative
variable. Interaction terms are entered as bxixj .
5. Compare models using R 2(adj).