User Manual

Section 4: Using Matrix Operations 111
Least-Squares Using Normal Equations
The unconstrained least-squares problem is known in statistical literature as multiple linear
regression. It uses the linear model



p
j
jj
rxby
1
Here, b
1,
…, b
p
are the unknown parameters, x
l
, ..., x
p
are the independent (or explanatory)
variables, y is the dependent (or response) variable, and r is the random error having
expected value E(r) = 0, variance σ
2
.
After making n observations of y and x
1
, x
2
, ..., x
p
, this problem can be expressed as
y = Xb + r
where y is an n-vector, X is an n × p matrix, and r is an n-vector consisting of the unknown
random errors satisfying E(r) = 0 and Cov(r) = E(rr
T
) = σ
2
I
n
.
If the model is correct and X
T
X has an inverse, then the calculated least-squares solution
yXXXb
TT 1
)(
ˆ


has the following properties:
 E(
b
ˆ
) = b, so that
b
ˆ
is an unbiased estimator of b.
 Cov(
b
ˆ
) = E((
b
ˆ
− b)
T
(
b
ˆ
− b)) = σ
2
(X
T
X)
–l
, the covariance matrix of the estimator
b
ˆ
.
 E(
r
ˆ
) = 0, where
r
ˆ
= y − X
b
ˆ
is the vector of residuals.

22
)()||
ˆ
(||E  pn
F
 bXy
, so that
)/(||
ˆ
||
ˆ
22
pn
F
 r 
is an unbiased estimator for
σ
2
. You can estimate Cov(
b
ˆ
) by replacing σ
2
by
2
ˆ

.
The total sum of squares
2
||||
F
y
can be partitioned according to
2
||||
F
y
= y
T
y
= (y − X
b
ˆ
+ X
b
ˆ
)
T
(y − X
b
ˆ
+ X
b
ˆ
)
= (y − X
b
ˆ
)
T
(y − X
b
ˆ
) - 2
b
ˆ
T
X
T
(y − X
b
ˆ
) + (X
b
ˆ
)
T
(X
b
ˆ
)
=
22
||
ˆ
||||
ˆ
||
FF
bXbXy 
=

















SquaresofSum
Regression
SquaresofSum
Residual
.
When the model is correct,
 
2222
||||||
ˆ
||E   pp
FF
XbbX