Straight-Line Regression Analysis

Copyright@Tieming Ji
Spring 2013
University of Missouri at Columbia
1 / 13
Who invented regression methods?
Figure: Karl Pearson
1857 1936
Figure: Francis Galton
1822 1911
2 / 13
History
Glaton discovered Galton Mark ngerprints.
Bring mathematical rigor into the science of biology.
Pairs of (reputed to be) highly intelligent fathers and sons
to study the inheritance of genius.
Pairs of fathers and sons to study inheritance of height.
3 / 13
4 / 13
How would you describe the relationship between the heights of the
fathers and the heights of their sons?
G
G
G
G
G G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G G
G
G G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G G
G
G
G G
G G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G G
G
G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
GG
G
G G
G
G
G
G
G
G G
GG G G
G
G
G
G
G G
G G
G
G
G G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G G G
G
G G G
G G G
G
G
G
G
G
G G
G
G
G
G G G
G
G
G
G
G
G
G
G
G G
G
G GG
G
G G
G
G
G
G
G G
G
G
G G G
G
G
G G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G
G
G G G G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
GG
G
G
G
G
G
GG G
G
G G
G G
G
G
G
G
G
G
G G
G G
G
G
G
G
G G
G
G G
G
G G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G
G G G G
G
GG
G
G G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
GG
G
G
G
G G
G G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G G
G G G
G
G
G
G
G
G
G
G G
G
G G
G
G
G G
G
G
G
G
G
G
G
G G GG
G G
G G
G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G G G
G
G G
G
G G
G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G G
G
G
G
G G
G
G
G
G
G G
G
G
G G
G
G
G
G
G
G
GG
G G
G
G
G G G G
G G
G
G
G
G
G G
G
GG
G
G
G
G
G
G
G
G G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G
G
G G
G
G
G
G
G G
G
G
G
G G
G G
G
G G
G
G
G
G
G G
G
G
G GGG G
G
G
G
G
G
G
G
G
G
G G G GG
G
G
G
G
G
G GG
G
G
G
G
G G G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G G
G
G G
G
G
GG
G G
G
G
G
G
G G G
G G
G
G
G G
G
G G
G G
G
G G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
60 65 70 75
6
0
6
5
7
0
7
5
Father's height (inches)
S
o
n
'
s

h
e
i
g
h
t

(
i
n
c
h
e
s
)
Let i index the i th pair of father and
son (i = 1, , n).
y
i
: The height of the i th son.
x
i
: The height of the i th father.
Notice the variation among sons
heights given the xed height of the
father, a straight line regression model
is:
y
i
=
0
+
1
x
i
+ e
i
,
where e
i
i .i .d.
N(0,
2
).
5 / 13
y
i
=
0
+
1
x
i
+ e
i
,
e
i
i .i .d.
N(0,
2
).
What are the assumptions of the model?
Randomization
Linearity
Independence, normality, and homoscedasticity assumption
for e
i
s (and also y
i
s implied from e
i
s).
y
i
s are independently distributed, and
y
i
|x
i
N(
0
+
1
x
i
,
2
).
6 / 13
X: random variable (r.v.) associated with fathers height.
x
1
, , x
n
are observations of X.
Y: r.v. associated with sone height.
y
1
, , y
n
are observations of Y given values of x
i
s.
We have:

Y|x
=
0
+
1
x
Y|x =
0
+
1
x + E
7 / 13
Fitting the Best Straight Line
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
GG
G
G G
G
G
G
G
G
G
G
GG G G
G
G
G
G
G G
G G
G
G
G G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G G
G
G
G
G G G
G
G
G
G
G
G
G
G
G G
G
G GG
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
GG G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
GG
G
G
G
G G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G G
G G G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G
G
G
G
G G
G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G G
G
G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G G
G
G
G
G G G
G G
G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G
GG G
G
G
G
G
G
G
G
G
G
G G G GG
G
G
G
G
G
G
GG
G
G
G
G
G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G G
G
G
G
G
G G
G
G G
G
G
G G
G
G G
G G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
60 65 70 75
6
0
6
5
7
0
7
5
Father's height (inches)
S
o
n
'
s

h
e
i
g
h
t

(
i
n
c
h
e
s
)
8 / 13
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
GG
G
G G
G
G
G
G
G
G
G
GG G G
G
G
G
G
G G
G G
G
G
G G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G G
G
G
G
G G G
G
G
G
G
G
G
G
G
G G
G
G GG
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
GG G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
GG
G
G
G
G G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G G
G G G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G
G
G
G
G G
G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G G
G
G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G G
G
G
G
G G G
G G
G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G
GG G
G
G
G
G
G
G
G
G
G
G G G GG
G
G
G
G
G
G
GG
G
G
G
G
G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G G
G
G
G
G
G G
G
G G
G
G
G G
G
G G
G G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
60 65 70 75
6
0
6
5
7
0
7
5
Father's height (inches)
S
o
n
'
s

h
e
i
g
h
t

(
i
n
c
h
e
s
)
G
9 / 13
Observations: y
1
, y
2
, , y
n
.
Estimates: y
1
, y
2
, , y
n
.
Least Square Method (LSM) nds

0
and

1
such that
SSE =
n

i =1
(y
i
y
i
)
2
=
n

i =1
(y
i
(

0
+

1
x
i
))
2
is minimized for all
0
and
1
.
Denote e
i
= y
i
(

0
+

1
x
i
), then y
i
=

0
+

1
x
i
+ e
i
. So
LSM nds

0
and

1
such that

n
i =1
e
2
i
is minimized.
10 / 13
By LSM,

1
=

n
i =1
(x
i
x)(y
i
y)

n
i =1
(x
i
x)
2
(1)

0
= y

1
x (2)
Thus,
y
i
=

0
+

1
x
i
(3)
= y +

1
(x
i
x) (4)
11 / 13
Properties of LS estimates:
Theorem:

0
and

1
are unbiased estimators of
0
and
1
. In
addition, they have the minimum variance among all unbiased
(linear) estimators of
0
and
1
.
That is,

0
and

1
are BLUE.
12 / 13
Do the tting results surprise you?
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G G
G
G G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G G
G
G
G G
G G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
GG
G
G G
G
G
G
G
G
G G
GG G G
G
G
G
G
G G
G G
G
G
G G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G G
G
G
G G G
G G G
G
G
G
G
G
G G
G
G
G
G G G
G
G
G
G
G
G
G
G
G G
G
G GG
G
G
G
G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G G G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
GG G
G
G G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G
GG
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
GG
G
G
G
G G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G G
G G G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G
G
G G
G G
G
G
G G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G G
G
G G
G
G
G
G
G
GG
G
G
G
G
G
G
G G G
G G
G
G
G
G
G
G
G
G
G
G
G G
G
G G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G G
G
G
G G G G
G G
G
G
G
G
G
G
G
GG
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G G
G
G
G
G
G
G
G G
G
G
G
G
G G
G
G
G
G
G
G G
G
G G
G
G
G
G
G
G
G
G
G G
GG G
G
G
G
G
G
G
G
G
G
G G G GG
G
G
G
G
G
G GG
G
G
G
G
G G G
G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G G
G
G G
G
G
G
G
G G
G
G
G
G
G G
G
G G
G
G
G G
G
G G
G G
G
G G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G G
G
G
G
G
G
G
G
G G G G
G
G
G
G
G
G
G
G
G
G
G
G G
G
60 65 70 75
6
0
6
5
7
0
7
5
Father's height (inches)
S
o
n
'
s

h
e
i
g
h
t

(
i
n
c
h
e
s
)
n=1078
Fitted
Diagonal
Origin of the name Regression.
13 / 13