2
$\begingroup$

From Bayesian Data Analysis 3rd Edition [Gelman et. al], they give this as an example when introducing non-informative priors:

"We return to the problem of estimating the mean θ of a normal model with known variance $σ^2$ , with a $N(μ_0 , τ_0^2 )$ prior distribution on $θ$. If the prior precision, $1/τ_0^2$, is small relative to the data precision, $n/σ^2$, then the posterior distribution is approximately as if $τ_0^2 = ∞$:

$$p(θ|y) ≈ N(θ|y, σ^2 /n)$$

Putting this another way, the posterior distribution is approximately that which would result from assuming $p(θ)$ is proportional to a constant for $θ ∈ (−∞, ∞)$. Such a distribution is not strictly possible, since the integral of the assumed $p(θ)$ is infinity, which violates the assumption that probabilities sum to 1. In general, we call a prior density $p(θ)$ proper if it does not depend on data and integrates to 1. (If $p(θ)$ integrates to any positive finite value, it is called an unnormalized density and can be renormalized—multiplied by a constant—to integrate to 1.) The prior distribution is improper in this example, but the posterior distribution is proper, given at least one data point."

In particular, I don't really understand the bold part. It doesn't look to me like the marginal $p(θ)$ is proportional to a constant. Is my understanding correct that the integral is infinity because if the variance of the posterior is approximately $∞$, then $θ$ is equally likely to be found anywhere, and the pdf is uniform on $(-∞, ∞)$. Also, why is the posterior distribution proper given at least one data point?

I don't understand the intuition here

$\endgroup$
2

1 Answer 1

5
$\begingroup$

I cannot really help with the "intuition" but here are some items of explanation:

It doesn't look to me like the marginal $p(θ)$ is proportional to a constant.

If the posterior $p(\theta|y)$ is approximately $\mathcal N(θ|y, σ^2 /n)$, it means it is approximately equal to the likelihood since $$f(y|\theta)=\frac{\exp(-n\{\theta-y\}^2/2\sigma^2)}{\sqrt{2\pi\sigma^2/n}}=f(\theta|y)$$by symmetry of the Normal density. Thus, $p(\theta|y)\approx f(y|\theta)\propto f(y|\theta)\times c$ where $c$ is an arbitrary constant and by Bayes theorem $$p(\theta)f(y|\theta)\propto p(\theta|y) \propto f(y|\theta)\times c$$ where all the proportionality symbols are in terms of functions of $\theta$ (as $y$ is observed, hence fixed).

Is my understanding correct that the integral is infinity because if the variance of the posterior is approximately $∞$, then $θ$ is equally likely to be found anywhere, and the pdf is uniform on $(−∞,∞)$.

A positive function with an infinite mass, $$\int p(\theta)\,\text{d}\theta=\infty$$ cannot be interpreted as a probability density. Hence the probability of $\theta$ to be anywhere does not exist if $p(\theta)$ is improper. (It is improperly called a prior since it is not a probability density.) It is also improper to call a constant prior uniform because this is not a Uniform probability distribution. Note that (i) the integral of the function is infinity because it is constant, not because the variance is infinite (as there exist plenty of true probability distributions with an infinite variance). And (ii) the variance of the prior, not the posterior, is infinite, although it is again improper to call it a variance since it is not a true probability distribution.

why is the posterior distribution proper given at least one data point?

It happens that with this specific improper prior, $p(\theta)=1$, the integral $$\int f(y_1|\theta)\,\text{d}\theta=1$$ is finite. The proof is direct since $f(y_1|\theta)=f(\theta|y_1)$ in this specific case. There exist infinitely other cases that require an arbitrary number $m$ of observations for the posterior to exist. And yet infinitely other cases that never produce posterior distributions, no matter the number of observations. Note also that the very name of noninformative priors is itself debatable.

$\endgroup$
1
  • 1
    $\begingroup$ @Glen_b Thank you, I fail to spot these typos, more & more... $\endgroup$
    – Xi'an
    Commented Jun 22, 2020 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.