Module 33

Statistical Inference Problems: Point Estimation

() Module 33 Statistical Inference Problems: Point Estimation 1/1

Introduction to Statistical Inference
The basic situation in statistical inference problems is as follows:

We seek information about characteristics of a collection of elements,

called population;

Due to various considerations (such as time, cost etc.) we may not

wish or be able to study each individual element of the population;

Our object is to draw conclusions about the unknown population

characteristics on the basis of information on characteristics of a
suitably selected sample from the population;

Formally, let the r.v. X (which may be vector valued) describes the
characteristics of the population under investigation and let F (·) be
the d.f. of X ;
() Module 33 Statistical Inference Problems: Point Estimation 1/1
Parametric Statistical Inference:

Here the r.v.X has a d.f. F ≡ Fθ (·) with a known functional form
(except perhaps for the parameter θ, which may be a vector valued);

Let Θ be the set of possible values of the unknown parameter θ. In

problems of parametric statistical inference, the statistician’s job is
to decide, on the basis of a suitably selected sample (generally a
random sample) from Fθ (·), which member or members of the family
{Fθ (·) : θ ∈ Θ} can represent the d.f. of X ;

() Module 33 Statistical Inference Problems: Point Estimation 2/1

Nonparametric Statistical Inference:

Here we know nothing about the functional form of the d.f. F (·)
(except perhaps that F (·) is, say, continuous or discrete);

Our goal is to make inferences about the unknown d.f. F (·);

() Module 33 Statistical Inference Problems: Point Estimation 3/1

In statistical inference problems, the statistician can observe n
independent observations on X , the r.v. describing the population
under investigation, i.e., the statistician observes n values x 1 , . . . , x n
assumed by the r.v. X ;

Each x i can be regarded as the value assumed by a r.v. X i ,

i = 1, . . . , n, having the d.f. F (·);

The observed values (x 1 , . . . , x n ) are then the values assumed by

(X 1 , . . . , X n );

The set {X 1 , . . . , X n } is then a random sample of size n taken from

the population having d.f. F (·);

The observed value (x 1 , . . . , x n ) is called a realization of the random

sample.
() Module 33 Statistical Inference Problems: Point Estimation 4/1
Definition 1

(a) The space of the possible values of the random sample (X 1 , . . . , X n )

is called the sample space. We will denote the sample space by χ.
Generally the sample space χ is the same as the support SX of the
distribution of random sample or its interior.

(b) In the parametric statistical inference problems, the set Θ of possible

values of the unknown parameter θ is called the parameter space.

() Module 33 Statistical Inference Problems: Point Estimation 5/1

Some Parametric Statistical Inference Problems

Consider the following example.

Example 1.
A manager wants to make inferences about the mean lifetime of a
brand of an electric bulb manufactured by a certain company;

Here the population under investigation consists of lifetimes of all the

electric bulbs produced by that company;

Suppose that the r.v. X represents the lifetime of a typical electric

bulb manufactured by the company, i.e., the r.v. X describes the
given population;

() Module 33 Statistical Inference Problems: Point Estimation 6/1

Probability modelling on the past experience with testing of similar
electric bulbs indicates that X has an exponential distribution with
mean θ, i.e., X has the d.f.

0, if x < 0
FX (x|θ) = − θx ;
1 − e , if x ≥ 0
But the value of θ ∈ Θ = (0, ∞) is not evident from the past
experience and the manager wants to make inferences about the
unknown parameter θ ∈ Θ;

Here Θ = (0, ∞) is the parameter space. Due to various

considerations (e.g., time, cost etc.), the statistician can not obtain
the lifetimes of all the bulbs produced by the company;

One way to obtain information about the unknown θ is to do testing,

under identical conditions, on a number, say n, of electric bulbs
produced by the company;
() Module 33 Statistical Inference Problems: Point Estimation 7/1
This leads to observing a realization x = (x1 , . . . , xn ) of a random
sample X = (X1 , . . . , Xn ) from the population;

Here, X = (X1 , . . . , Xn ) ∈ χ = R+ = {(t1 , . . . , tn ) : 0 ≤ ti < ∞, i =

1, . . . , n} and χ is the sample space;

On the basis of the realization x of the random sample X , the

manager may want answer several questions concerning unknown θ.
Some of these may be:

() Module 33 Statistical Inference Problems: Point Estimation 8/1

(a) How to obtain a point estimate of θ? This is an example of a point
estimation problem;

(b) How to obtain an appropriate interval in which the unknown θ lies

with certain confidence? This is an example of a confidence interval
estimation problem of finding an appropriate random interval
(depending on X ) for the unknown θ such that the given random
interval contains the true θ with given confidence (probability);

(c) To verify the claim (hypothesis) that θ ∈ Θ0 , where Θ0 ⊂ Θ. This is

an example of hypothesis testing problem.

() Module 33 Statistical Inference Problems: Point Estimation 9/1

Point Estimation Problems

X : a r.v. defined on a probability space (Ω, F, P);

X has a d.f. F (·|θ), the functional form of which is known and θ ∈ Θ

is unknown; here Θ is the parameter space;

The basic situation in point estimation problems is as follows:

We observe r.v.s X 1 , . . . , X n (say, a random sample) from the

population described by the d.f. F (·|θ);

based on random sample X 1 , . . . , X n we seek an approximation (or an

estimate) of θ (or some function of θ).

() Module 33 Statistical Inference Problems: Point Estimation 10 / 1

Definition 2.

(a) Let g (θ) (possibly a vector valued) be a function of θ which we want

to estimate. Then g (θ) is called the estimand.

(b) Let Λ = {g (θ) : θ ∈ Θ} ⊆ Rq be the range of possible values of the

estimand g (θ). A statistic δ ≡ δ(X ) is said to be an estimator of
g (θ) if δ maps the sample space χ into Rq ; here X = (X 1 , . . . , X n ).

() Module 33 Statistical Inference Problems: Point Estimation 11 / 1

Definition 2 (continued)

(c) Let x = (x 1 , . . . , x n ) be a sample realization of X = (X 1 , . . . , X n )

and let δ ≡ δ(X ) be an estimator of g (θ). Then δ(x) is called an
estimate of g (θ) (i.e., an estimate is a realization of an estimator);

Note: An estimator is a r.v.. To cover more general situations, in the

definition of an estimator we allow it to assume values outside Λ, the set
of possible values of the estimand g (θ) (although it may look absurd).

() Module 33 Statistical Inference Problems: Point Estimation 12 / 1

Example 2.

Let X1 , . . . , Xn be a random sample from a Poisson(θ) distribution,

where θ ∈ Θ = (0, ∞);

Let the estimand be g (θ) = θ;

Then δ1 (X ) = X is an estimator of g (θ), so also is δ2 (X ) = S 2 .

By the definition, δ3 (X ) = (−1)X1 X1 is also an estimator of g (θ), but

it is absurd since it can assume negative values whereas the estimand
g (θ) is positive.

We will now discuss two commonly used methods of parametric point

estimation, namely the Method of Moments and the Method of
Maximum Likelihood.
() Module 33 Statistical Inference Problems: Point Estimation 13 / 1
The Method of Moments

X1 , . . . , Xn : a random sample of size n from a population having

distribution function Fθ (x), x ∈ R (i.e., the r.v. describing the
population has the d.f. Fθ (·)), where θ = (θ1 , . . . , θp ) ∈ Θ is an
unknown parameter;

Suppose that, for k = 1, . . . , p, mk = Eθ (X1k ) exists and is finite.

Here and elsewhere Eθ0 (·) (Pθ0 (·)) represents that the expectation
(probability) is calculated under the d.f. Fθ0 (·), θ0 ∈ Θ. Let
mk = hk (θ), k = 1, . . . , p;

Define
n
1X k
Ak = Xi , k = 1, . . . , p.
n
i=1

() Module 33 Statistical Inference Problems: Point Estimation 14 / 1

Definition 3.

(a) mk (k = 1, . . . , p) is called the k th population moment (about origin)

of the d.f. Fθ (·), θ ∈ Θ;

(b) Ak (k = 1, . . . , p) is called the k th sample moment (about origin)

based on the random sample X1 , . . . , Xn ;

(c) The method of moments consists of equating Ak with

hk (θ1 , . . . , θp ), for k = 1, . . . , p, and solving for θ1 , . . . , θp . The value
(θ1 , . . . , θp ) = (θˆ1 , . . . , θˆp ), say, so obtained is called the method of
moment estimator (M.M.E.) of θ = (θ1 , . . . , θp );

() Module 33 Statistical Inference Problems: Point Estimation 15 / 1

Definition 3 (continued)

(d) Let g : Θ → Λ be a mapping of Θ onto Λ. If θ̂ is the M.M.E. of θ,

then g (θ̂) is called the M.M.E. of g (θ).

Remark 1. (a) The method of moments is not applicable when mk

(k = 1, . . . , p) do not exist (e.g., for the Cauchy distribution with median
θ).

(b) M.M.E. may not exist when the underlying equations do not have a
solution. Also the M.M.E. may not be unique as the underlying equations
may have more than one solution.

() Module 33 Statistical Inference Problems: Point Estimation 16 / 1

Example 3.

Let X1 , . . . , Xn be a random sample from a Poisson(θ) distribution, where

θ ∈ Θ = (0, ∞) is unknown. Then θ̂ = X is the M.M.E. of θ.

Solution: We have m1 = E (X1 ) = θ. Thus M.M.E. θ̂ is the solution of

equation
n
1X
θ̂ = Xi = X .
n
i=1

() Module 33 Statistical Inference Problems: Point Estimation 17 / 1

Example 4.
Let X1 , . . . , Xn (n ≥ 2) be a random sample from N(µ, σ 2 ) distribution,
where θ = (µ, σ 2 ) ∈ Θ = {(z1 , z2 ) : −∞ < z1 < ∞, z2 > 0} is unknown.
Then θ̂ = (X , n−1 2
n S ) is the M.M.E. of θ.

Solution: We have

m1 = E (X1 ) = µ and m2 = E (X12 ) = σ 2 + µ2 .

Thus M.M.E. θ̂ = (µ̂, σ̂ 2 ) is the solution of

n n
1X 2 1X 2
2
µ̂ = Xi , σ̂ + µ̂ = Xi
n n
i=1 i=1

n n
1X 2 2 1X n−1 2
⇒ µ̂ = X σ̂ 2 = Xi − X = (Xi − X )2 = S .
n n n
i=1 i=1

() Module 33 Statistical Inference Problems: Point Estimation 18 / 1

The Method of Maximum Likelihood

X1 , . . . , Xn : a random sample of size n from a population having

p.d.f. (or p.m.f.) fθ (x), x ∈ R (i.e., the r.v. describing the population
has the p.d.f. (or p.m.f.) fθ (·)), where θ = (θ1 , . . . , θp ) ∈ Θ is an
unknown parameter;

Then the joint p.d.f. of X = (X1 , . . . , Xn ) is

n
Y
fX (x|θ) = fθ (xi ), θ ∈ Θ.
i=1

Definition 4. For a given sample realization x = (x1 , . . . , xn ) of the

observation on X = (X1 , . . . , Xn ), the function

Lx (θ) = fX (x|θ),

considered as a function of θ ∈ Θ, is called the likelihood function.

() Module 33 Statistical Inference Problems: Point Estimation 19 / 1
Remark 2.

In the discrete case, given the sample realization x = (x1 , . . . , xn ),

Lx (θ0 ) = fX (x|θ0 )

is the probability of obtaining the observed sample x, when θ0 ∈ Θ is the

true value of θ. Therefore, intuitively, it is appealing to find θ̂ ≡ θ̂(x)
(provided it exists) such that Lx (θ̂) = supθ∈Θ Lx (θ), since if such a θ̂ exists
then it is more probable that x came from the distribution with p.d.f. (or
p.m.f.) fX (·|θ̂) than from any of the other distribution fX (·|θ),
θ ∈ Θ − {θ̂}. A similar argument can also be given for absolutely
continuous distributions.

() Module 33 Statistical Inference Problems: Point Estimation 20 / 1

Definition 5.

(a) For a given sample realization x, the maximum likelihood estimate

(m.l.e.) of the unknown parameter θ is the value θ̂ ≡ θ̂(x) (provided
it exists) such that
Lx (θ̂) = sup Lx (θ).
θ∈Θ

(b) Let g : Θ → Λ be a mapping of Θ into Λ. Define, for λ ∈ Λ,

Θλ = {θ ∈ Θ : g (θ) = λ}. Then, for a given sample realization x, the
function
Mx (λ) = sup Lx (θ),
θ∈Θλ

considered as a function of λ ∈ Λ, is called the likelihood function

induced by g (θ).

() Module 33 Statistical Inference Problems: Point Estimation 21 / 1

(c) For a given sample realization x, the maximum likelihood estimate
(m.l.e.) of the estimand g (θ) is the value λ̂ ≡ λ̂(x) (provided it
exists) such that
Mx (λ̂) = sup Mx (λ),
λ∈Λ

where Mx (λ) is as defined in (b) above.

(d) The estimator (a r.v.) corresponding to the m.l.e. is called the

maximum likelihood estimator (M.L.E.).

() Module 33 Statistical Inference Problems: Point Estimation 22 / 1

Remark 3.

(a) (Maximum likelihood estimate may not be unique). Let

x = (x1 , . . . , xn ) be a sample realization based on a random sample
from U(θ − 21 , θ + 21 ) distribution, where θ ∈ Θ = (−∞, ∞) is an
unknown parameter. Then, for x(1) = min{x1 , . . . , xn } and
x(n) = max{x1 , . . . , xn },

1, if x(n) − 21 ≤ θ ≤ x(1) + 1

Lx (θ) = 2 .
0, otherwise
1 1
Clearly any estimate δ(x) such that x(n) − 2 ≤ δ(x) ≤ x(1) + 2 is a
x(1) +x(n)
m.l.e. In particular δ ∗ (x) = 2 is a m.l.e. of θ.

() Module 33 Statistical Inference Problems: Point Estimation 23 / 1

(b) Maximum likelihood estimate may be absurd. Let
x = (0, 0, . . . , 0) be a sample realization based on a random sample of
size n from a Bin(1, θ) distribution, where θ ∈ Θ = (0, 1) is unknown.
In this case
Lx (θ) = (1 − θ)n , 0 < θ < 1.
and θ̂ = x = 0 is the m.l.e., while θ̂ does not belong to Θ.

(c) Since Lx (θ) and ln Lx (θ) attain their maximum for same values of θ,
sometimes it is more convenient to work with ln Lx (θ).

() Module 33 Statistical Inference Problems: Point Estimation 24 / 1

(d) If Θ is an open rectangle in Rp and Lx (θ) is a positive and
differentiable function of θ (i.e., the first order partial derivatives exist
in the components of θ), then if a m.l.e. θ̂ exists, it must satisfy

∂ 
ln Lx (θ) θ=θ̂ = 0, j = 1, . . . , p; θ = (θ1 , . . . , θp )

∂θj
∂ 
⇔ Lx (θ) θ=θ̂ = 0, j = 1, . . . , p.

∂θj

() Module 33 Statistical Inference Problems: Point Estimation 25 / 1

Result 1.

(Invariance of the m.l.e.) Suppose that Θ ⊆ Rp . Let g : Θ → Λ be a

mapping of Θ into Λ, where Λ is a region in Rq (1 ≤ q ≤ p). If θ̂ ≡ θ̂(x)
is the m.l.e. of θ and θ̂(X ) ∈ Θ with probability one, then g (θ̂) is the
m.l.e. of g (θ).

Proof: We have Θλ = {θ ∈ Θ : h(θ) = λ}, λ ∈ Λ and

Mx (λ) = sup Lx (θ), λ ∈ Λ.

θ∈Θλ

Clearly {Θλ : λ ∈ Λ} forms a partition of Θ. Now

θ̂ is m.l.e. of θ ∈ Θ ⇒ θ̂ ∈ Θ ⇒ θ̂ ∈ Θλ , for some λ ∈ Λ.

() Module 33 Statistical Inference Problems: Point Estimation 26 / 1

Let θ̂ ∈ Θλ̂ , where λ̂ ∈ Λ. Then h(θ̂) = λ̂ (by definition of Θλ̂ ). Also,
since θ̂ ∈ Θλ̂ ,

Lx (θ̂) ≤ sup Lx (θ) = Mx (λ̂) ≤ supMx (λ) = sup Lx (θ) = Lx (θ̂)

θ∈Θλ̂ λ∈Λ θ∈Θ

⇒ Mx (λ̂) = supMx (λ)

λ∈Λ

⇒ λ̂ = h(θ̂) is an m.l.e. of h(θ).

() Module 33 Statistical Inference Problems: Point Estimation 27 / 1

Regularity Conditions R1 :

(a) The parameter space Θ is an open interval in R (finite, infinite or

semi-finite);

(b) The support SX = {x : fX (x|θ) > 0} does not depend on θ;

∂
(c) For any x ∈ SX and any θ ∈ Θ, the derivative ∂θ fX (x|θ), θ ∈ Θ,
exists and is finite and
Z ∞ Z ∞
··· fX (x|θ)dx = 1, θ ∈ Θ,
−∞ −∞

can be differentiated under the integral (or summation) sign, so that

Z ∞ Z ∞ Z ∞ Z ∞
∂ d
··· fX (x|θ)dx = ··· fX (x|θ)dx = 0, ∀ θ ∈ Θ,
−∞ −∞ ∂θ dθ −∞ −∞

with integrals replaced by the summation sign in the discrete case;

() Module 33 Statistical Inference Problems: Point Estimation 28 / 1
(d) For any x ∈ SX and any θ ∈ Θ, the second partial derivative
∂2
f (x|θ), θ ∈ Θ, exists and is finite and
∂θ2 X
Z ∞ Z ∞
∂
··· fX (x|θ)dx = 0, θ ∈ Θ,
−∞ −∞ ∂θ

can be differentiated under the integral (summation) sign, so that

∞ ∞ ∞ ∞
∂2 d2
Z Z Z Z
··· fX (x|θ)dx = ··· fX (x|θ)dx = 0, ∀θ ∈ Θ,
−∞ −∞ ∂θ2 dθ2 −∞ −∞

with integrals replaced by the summation sign in the discrete case.

() Module 33 Statistical Inference Problems: Point Estimation 29 / 1

Remark 4.
(a) Using advanced mathematical arguments, it can be shown that the
regularity conditions R1 are satisfied for a large family of distributions,
including the exponential family of distributions having associated
p.d.f.s (or p.m.f.s) of the form

fX (x|θ) = c(θ)h(x)e r (θ)T (x) , x ∈ χ, θ ∈ Θ,

for some functions h(·), c(·), r (·) and T (·) and an open interval
Θ ⊆ R.

() Module 33 Statistical Inference Problems: Point Estimation 30 / 1

(b) For x ∈ SX ,
 2  2
∂ 1 ∂
Ψ(x, θ) = ln fX (x|θ) = fX (x|θ)
∂θ fX (x|θ) ∂θ

represents the relative rate at which the p.d.f. (or p.m.f.) fX (x|θ)
changes at x. The average of this rate is denoted by
 2 !
∂
I (θ) = Eθ ln fX (X |θ) , θ ∈ Θ ⊆ R.
∂θ

The large value of I (θ0 ) indicates that it is easier to distinguish θ0

from the neighboring values of θ0 and therefore more accurately θ can
be estimated if true θ = θ0 . The quantity I (θ), θ ∈ Θ, is called the
Fisher’s information that X contains about the parameter θ. Note
that I (θ) is a function of θ ∈ Θ.

() Module 33 Statistical Inference Problems: Point Estimation 31 / 1

(c) Let X1 , . . . , Xn be a random sample with common p.d.f./p.m.f.
f (·|θ), θ ∈ Θ ⊆ R, and let X = (X1 , . . . , Xn ). Then
n
Y
fX (x|θ) = f (xi |θ), θ ∈ Θ.
i=1

Let i(θ) and I (θ), respectively, denote the Fisher’s information

contained in the single observation, say X1 , and the whole sample
X = (X1 , . . . , Xn ). Then, for θ ∈ Θ,
 2 !
∂
I (θ) = Eθ ln fX (X |θ)
∂θ
 !2 
n
X ∂
= Eθ  ln f (Xi |θ) 
∂θ
i=1

() Module 33 Statistical Inference Problems: Point Estimation 32 / 1

 n  2 !
X ∂
= Eθ ln f (Xi |θ)
∂θ
i=1
 
n X
n
X ∂ ∂ 
+Eθ  ln f (Xi |θ) ln f (Xj |θ)
 ∂θ ∂θ 
i=1 j=1
i6=j
 2 !
∂
= nEθ ln f (X1 |θ)
∂θ
= ni(θ),

since X1 , . . . , Xn are i.i.d., and

() Module 33 Statistical Inference Problems: Point Estimation 33 / 1

   Z ∞
∂ ∂
Eθ ln f (X1 |θ) = f (x|θ)dx
∂θ −∞ ∂θ
Z ∞
d
= f (x|θ)dx
dθ −∞

= 0, ∀ θ ∈ Θ.

() Module 33 Statistical Inference Problems: Point Estimation 34 / 1

Result 2.

Let X1 , X2 , . . . be a sequence of i.i.d. one-dimensional r.v.s with common

∂
Fisher’s information i(θ) = Eθ (( ∂θ ln f (X1 |θ))2 ), θ ∈ Θ ⊆ R, where f (·|θ),
θ ∈ Θ, is the common p.d.f. (or p.m.f.) of the sequence X1 , X2 , . . ., and Θ
is an open interval in R. Let θ̂n be the unique M.L.E. of θ based on
X1 , . . . , Xn . Then, under regularity conditions R1 , as n → ∞,
√
 
d 1 p
n(θ̂n − θ) → Y ∼ N 0, and θ̂n → θ.
i(θ)

() Module 33 Statistical Inference Problems: Point Estimation 35 / 1

Corollary 1.

Under the regularity conditions R1 , let ĝn ≡ ĝn (X ) be the M.L.E. of

one-dimensional estimand g (θ), where g (·) is a differentiable function.
Then, under regularity conditions R1 , as n → ∞,

√ (g 0 (θ))2
 
d p
n(ĝn − g (θ)) → W ∼ N 0, and ĝn → g (θ), θ ∈ Θ.
i(θ)

() Module 33 Statistical Inference Problems: Point Estimation 36 / 1

Example 5.
Let X1 , . . . , Xn (n ≥ 2) be a random sample from N(µ, σ 2 ) distribution,
where θ = (µ, σ 2 ) ∈ Θ = (−∞, ∞) × (0, ∞) is unknown. Show that the
maximum likelihood estimator of θ is (µ̂, σ̂ 2 ) = (X , n−1 2
n S ).

Proof: For a given sample realization x = (x1 , . . . , xn )

n  
1 1 2
√ e − 2σ2 (xi −µ)
Y
Lx (θ) =
i=1
σ 2π
n 1 Pn 2
= (2πσ 2 )− 2 e − 2σ2 i=1 (xi −µ) , θ ∈ Θ.
Then
n
n n 1 X
ln Lx (θ) = − ln(2π) − ln(σ 2 ) − 2 (xi − µ)2 ,
2 2 2σ
i=1
n
∂ 1 X
ln Lx (θ) = (xi − µ)
∂µ σ2
i=1

() Module 33 Statistical Inference Problems: Point Estimation 37 / 1

 n
∂ n 1 X
ln Lx (θ) = − + (xi − µ)2
∂σ 2 2σ 2 2σ 4
i=1
∂ n
ln Lx (θ) = − 2
∂µ2 σ
n
∂ n 1 X
ln Lx (θ) = − (xi − µ)2
∂(σ 2 )2 2σ 4 σ 6
i=1
n
∂ 1 X
ln Lx (θ) = − 4 (xi − µ).
∂µ∂σ 2 σ
i=1
n−1 2
Clearly θ̂ = (µ̂, σ̂ 2 ) = (X , n S )
is the unique critical point. Also
 
∂ n
2
ln Lx (θ) = − 2
∂µ θ=θ̂ σ̂
  n
∂ n 1 X
ln Lx (θ) = − (xi − x)2
∂(σ 2 )2 θ=θ̂ 2σ̂ 4 σ̂ 6
i=1

() Module 33 Statistical Inference Problems: Point Estimation 38 / 1

 n
= −
  2σ̂ 4
∂
ln Lx (θ) = 0.
∂µ∂σ 2 θ=θ̂

It follows that θ̂ = (µ̂, σ̂ 2 ) = (X , n−1 2

n S ) is the m.l.e. of θ.

() Module 33 Statistical Inference Problems: Point Estimation 39 / 1

Example 6.
Let X1 , . . . , Xn be a random sample from Bin(m, θ) distribution, where
θ ∈ Θ = (0, 1) is unknown and m is a known positive integer. Show that
δM (X ) = X is the M.L.E. of θ.
Solution. For a sample realization x ∈ χ = {0, 1, . . . , m}n
n   
Y m xi m−xi
Lx (θ) = θ (1 − θ)
xi
i=1
n  
!
Y m Pn Pn
= θ i=1 xi (1 − θ)mn− i=1 xi .
xi
i=1

First, let θ ∈ (0, 1).

n n n
  ! !
X m X X
ln Lx (θ) = ln + xi ln θ + mn − xi ln(1 − θ)
xi
i=1 i=1 i=1

() Module 33 Statistical Inference Problems: Point Estimation 40 / 1

 Pn
mn − ni=1 xi
P
∂ i=1 xi
ln Lx (θ) = −
∂θ θ 1−θ
∂ x̄
ln Lx (θ) > 0 ⇔ θ<
∂θ m
x̄
⇔ is the M.L.E. of θ
m
X̄
⇒ δM (X ) = is the M.L.E. of θ.
m

() Module 33 Statistical Inference Problems: Point Estimation 41 / 1

Example 7.
Let X1 , . . . , Xn be a random sample from U(θ1 , θ2 ) distribution, where
θ = (θ1 , θ2 ) ∈ Θ = {(z1 , z2 ) : −∞ < z1 < z2 < ∞} is unknown. Show
that δ M (X ) = (X(1) , X(n) ) is the M.L.E. of θ.
Solution.
(
1
fXi (x) = θ2 −θ1 , if θ1 < x < θ2
, i = 1, . . . , n.
0, otherwise
Let x be the fixed realization. Then
n
Y
Lx (θ) = Lx (θ1 , θ2 ) = fX (x|θ) = fXi (xi |θ)
i=1
(
1
(θ2 −θ1 )n , x(1) > θ1 , x(n) < θ2
= .
0, otherwise
Here Lx (θ) is an increasing function of θ1 and decreasing function of θ2 .
Thus δ M (X ) = (X(1) , X(n) ) is the M.L.E. of θ.
() Module 33 Statistical Inference Problems: Point Estimation 42 / 1
Example 8. ( M.L.E. and M.M.E. may be different)

Let X ∼ U(0, θ), where θ ∈ Θ = (0, ∞) is unknown. Show that the

M.M.E. of θ is δMME (X ) = 2X , whereas the M.L.E. of θ is
δMLE (X ) = X .

Solution. Since Eθ (X ) = 2θ , it follows that δMME (X ) = 2X is the

M.M.E. of θ. Also, for a fixed realization x > 0,
(
1
, if θ > x
Lx (θ) = fX (x|θ) == θ .
0, if 0 < θ ≤ x

Clearly Lx (θ) is maximized at θ = x. Thus the M.L.E. of θ is

δMLE (X ) = X .

() Module 33 Statistical Inference Problems: Point Estimation 43 / 1

Properties of Estimators

Unbiased Estimators
Suppose that the estimand g (θ) is real-valued.
Definition 6.
(a) An estimator δ(X ) is said to be an unbiased estimator of g (θ) if
Eθ (δ(X )) = g (θ), ∀ θ ∈ Θ.

(b) An estimator which is not unbiased for estimating g (θ) is called a

biased estimator of g (θ).

(c) The quantity Bθ (δ) = Eθ (δ(X )) − g (θ), θ ∈ Θ, is called the bias of

the estimator δ(X ).

() Module 33 Statistical Inference Problems: Point Estimation 44 / 1

Remark 5.

(a) Note that Bθ (δ) is a function of θ ∈ Θ.

(b) Note that for an unbiased estimator δ(X ), Bθ (δ) = 0, ∀ θ ∈ Θ.

(c) An unbiased estimator, if evaluated a large number of times, on the

average equals the true value of the estimand. Thus, the property of
unbiasedness is a reasonable property for an estimator to have.

() Module 33 Statistical Inference Problems: Point Estimation 45 / 1

Example 8. ( Unbiased estimators may not exist)

. Let X ∼ Bin(n, θ), where θ ∈ Θ = (0, 1) is unknown and n is a known

positive integer. Show that the unbiased estimators for the estimand
g (θ) = 1θ do not exist.

Solution. On contrary suppose there exists an estimator δ(X ) such that

1
Eθ (δ(X )) = , ∀θ∈Θ
θ
n  
X n j 1
i.e., δ(j) θ (1 − θ)n−j = , ∀0<θ<1
j θ
j=0
n  
X n j
⇒θ δ(j) θ (1 − θ)n−j = 1, ∀ 0 < θ < 1,
j
j=0

which is not possible since, as θ → 0, L.H.S. → 0, whereas R.H.S. → 1.

() Module 33 Statistical Inference Problems: Point Estimation 46 / 1

Example 9. ( Unbiased estimator may be absurd)
Let X ∼ Poisson(θ), where θ ∈ Θ = (0, ∞) is unknown, and let the
estimand be g (θ) = e −3θ . Show that δ(X ) = (−2)X is the unique
unbiased estimator of g (θ) (here δ(X ) = (−2)X takes both positive and
negative values, whereas the estimand g (θ) is always positive).
Solution. An estimator δ(X ) is unbiased for estimating g (θ) = e −3θ iff
Eθ [δ(X )] = g (θ), ∀ θ ∈ Θ
∞
X e −θ θj
⇔ δ(j) = e −3θ , ∀ θ > 0
j!
j=0
∞
X δ(j)θj
⇔ = e −2θ , ∀ θ > 0
j!
j=0
∞ ∞
X δ(j)θj X (−2)j θj
⇔ = , ∀θ>0
j! j!
j=0 j=0

() Module 33 Statistical Inference Problems: Point Estimation 47 / 1

The L.H.S. and R.H.S. are power series in θ and they match in an open
interval. Thus,

δ(j) (−2)j
= , j = 0, 1, 2, . . .
j! j!
⇒ δ(j) = (−2)j , j = 0, 1, 2, . . .

Thus δ(X ) = (−2)X is the unique unbiased estimator of g (θ) = e −3θ .

() Module 33 Statistical Inference Problems: Point Estimation 48 / 1

Example 10. ( M.M.E. and M.L.E. may not be
unbiased)
Let X ∼ U(0,
√ θ), where θ ∈ Θ = (0, ∞) is unknown, and let the estimand
be g (θ) = θ.√Show that M.M.E. and√ the M.L.E. of g (θ) are
δMME (X ) = 2X and δMLE (X ) = X , respectively, and
√
2 2
Eθ (δMME (X )) = 3 g (θ), θ ∈ Θ, Eθ (δMLE (X )) = 23 g (θ), θ ∈ Θ.
Solution. For the sample realization x > 0, the likelihood function
(
1
, if θ > x
LX (θ) = fX (x|θ) = θ
0, otherwise,
is minimized at θ = θ̂ = x. Thus the MLE
√ of θ is X and by the invariance
property of MLEs, the MLE of g (θ) = θ is
√
δMLE (X ) = X .
Z θ√
x 2√
Eθ [δMLE (X )] = dx = θ 6= θ.
0 θ 3
() Module 33 Statistical Inference Problems: Point Estimation 49 / 1
Also MME of θ is given by (since E (X ) = 2θ )

θ̂MME
= X ⇒ θ̂MME = 2X .
2
√
Thus the MME of g (θ) = θ is
√
δMME (X ) = 2X
√
√ 2 2√
⇒ Eθ [δMME (X )] = 2Eθ (X ) = θ 6= θ.
3

() Module 33 Statistical Inference Problems: Point Estimation 50 / 1

Example 11 ( Typically, there are many unbiased
estimators for a given estimand)

Let X1 , . . . , Xn be a random sample from a N(θ, 1) distribution, where

θ ∈ Θ = (−∞, ∞) is unknown, and let the estimand be g (θ) = θ. Then
X +X
δM (X ) = X , δi (X ) = Xi , δi,j = i 2 j , i, j ∈ {1, 2, . . . , n}, i 6= j
δi,j,k (X ) = Xi + Xj − Xk , i, j, k ∈ {1, . . . , n}, etc., are all unbiased for
estimating g (θ).

() Module 33 Statistical Inference Problems: Point Estimation 51 / 1

As seen in the above example, typically, there are many unbiased
estimators for a given estimand. Therefore, it is useful to have some
criterion for comparing unbiased estimators. One criterion which is often
used is the variance of the unbiased estimator δ(·) (denoted by Vθ (δ) to
emphasize the dependence on θ ∈ Θ). If δ1 ≡ δ1 (X ) and δ2 ≡ δ2 (X ) are
two unbiased estimators of g (θ) and if

Vθ (δ1 ) = Eθ ((δ1 (X ) − g (θ))2 ) < Vθ (δ2 ) = Eθ ((δ2 (X ) − g (θ))2 ), ∀ θ ∈ Θ,

then (δ1 (X ) − g (θ))2 is, on the average, less than (δ2 (X ) − g (θ))2 , which
indicates that δ1 is nearer to g (θ) than δ2 . For this reason we define:

() Module 33 Statistical Inference Problems: Point Estimation 52 / 1

Definition 7.
An unbiased estimator δ1 is said to be better than the unbiased estimator
δ2 if Vθ (δ1 ) ≤ Vθ (δ2 ), ∀ θ ∈ Θ, with strict inequality for at least one
θ ∈ Θ.

Definition 8.
In an estimation problem where the M.L.E. exists, an estimator (not
necessarily unbiased) which depends on observation X = (X1 , . . . , Xn ) only
through the M.L.E. (i.e., an estimator which is a function of the M.L.E.
alone) is called an estimator based on the M.L.E..

() Module 33 Statistical Inference Problems: Point Estimation 53 / 1

Under fairly general conditions, it can be shown that the estimators which
are not based on the M.L.E. are not desirable, i.e., given any unbiased
estimator δ, which is not based on the M.L.E., there exists an unbiased
estimator based on the M.L.E., say δM , such that δM is better than that δ.
Thus, to find the best unbiased estimators one should consider only those
estimators which are based on the M.L.E. Under fairly general conditions,
it can also be shown that there is only one unbiased estimator based on
the M.L.E., and that estimator is the best unbiased estimator. Therefore,
in finding a sensible unbiased estimator for an estimand g (θ), we typically
start with the M.L.E. of g (θ). If it is unbiased, then we have found the
estimator we want. If it is not unbiased, we modify it to make it unbiased.

() Module 33 Statistical Inference Problems: Point Estimation 54 / 1

Example 12. Let X1 , . . . , Xn be a random sample from a Poisson(θ)
distribution, where θ ∈ Θ = (0, ∞) is unknown, and let the estimand be
g (θ) = Pθ (X = 0) = e −θ . Then the M.L.E. of g (θ) is δM (X ) = e −X and
the unbiased estimator based on the M.L.E. is δU (X ) = (1 − n1 )nX .
Solution. Let T = ni=1 Xi so that T ∼ Poisson(nθ) and X = Tn . We
P

want the estimator δ(X ) = δ( Tn ) such that

Eθ (δ(X )) = e −θ , ∀ θ > 0

∞
X j e −nθ (nθ)j
⇔ δ( ) = e −θ , ∀ θ > 0
n j!
j=0
∞
X j nj
⇔ δ( ) θj = e (n−1)θ , ∀ θ > 0
n j!
j=0
∞ ∞
X j nj X (n − 1)j
⇔ δ( ) θj = θj , ∀ θ > 0
n j! j!
j=0 j=0
() Module 33 Statistical Inference Problems: Point Estimation 55 / 1
 j 1
⇔ δ( ) = (1 − )j , j = 0, 1, . . . .
n n
It follows that the unbiased estimator based on the M.L.E. is
δU (X ) = (1 − n1 )nX .

() Module 33 Statistical Inference Problems: Point Estimation 56 / 1

Example 13.

Let X1 , . . . , Xn be a random sample from a N(µ, σ 2 ) distribution, where

θ = (µ, σ 2 ) ∈ Θ = {(z1 , z2 ) : −∞ < z1 < ∞, z2 > 0} is unknown, and let
the estimand be g (θ) = σ 2 . Show that the M.L.E. of (µ, σ 2 ) is
2
(X , (n−1)S
n ) and the unbiased estimator of g (θ) based on the M.L.E. is S 2 .

() Module 33 Statistical Inference Problems: Point Estimation 57 / 1

 Thank you for your patience

() Module 33 Statistical Inference Problems: Point Estimation 58 / 1