(a) Let X₁, X₂, ⋯, Xₙ be i.i.d. random variables following N(μ, σ²). Find the maximum likelihood estimator for σ² and examine…

(a) MLE of σ², bias, consistency and MSE

MLE of σ². With μ and σ² both unknown, the log-likelihood is

log L = −(n/2) log(2π) − (n/2) log σ² − Σ(Xᵢ − μ)²/(2σ²).

• ∂log L/∂μ = Σ(Xᵢ − μ)/σ² = 0 gives μ̂ = X̄.
• ∂log L/∂σ² = −n/(2σ²) + Σ(Xᵢ − μ)²/(2σ⁴) = 0; substituting μ̂ gives σ̂² = (1/n) Σ(Xᵢ − X̄)².

At (X̄, σ̂²) the Hessian is diagonal with entries −n/σ̂² and −n/(2σ̂⁴), both negative, so this is a maximum. By invariance, the MLE of σ is √σ̂².

Unbiasedness. Write SS = Σ(Xᵢ − X̄)². For a normal sample SS/σ² ~ χ²ₙ₋₁, so E(SS) = (n − 1)σ² and V(SS) = 2(n − 1)σ⁴. Hence

E(σ̂²) = (n − 1)σ²/n ≠ σ², bias = −σ²/n.

The MLE is biased (it under-estimates σ²), though asymptotically unbiased.

Unbiased estimator and its consistency. Correcting the divisor gives

s² = SS/(n − 1), E(s²) = σ².

V(s²) = 2(n − 1)σ⁴/(n − 1)² = 2σ⁴/(n − 1). Since E(s²) = σ² and V(s²) → 0 as n → ∞, Chebyshev's inequality gives P(|s² − σ²| > ε) ≤ 2σ⁴/[(n − 1)ε²] → 0. So s² is consistent for σ². (σ̂² is consistent too, since its bias and variance both vanish.)

Mean squared errors.

• MSE(s²) = V(s²) = 2σ⁴/(n − 1)
• V(σ̂²) = 2(n − 1)σ⁴/n², so MSE(σ̂²) = 2(n − 1)σ⁴/n² + σ⁴/n² = (2n − 1)σ⁴/n²

Then MSE(s²) − MSE(σ̂²) = σ⁴(3n − 1)/[n²(n − 1)] > 0 for every n ≥ 2. The biased MLE has the smaller MSE: its small bias is more than repaid by its lower variance. For example, n = 5 gives 0.36σ⁴ against 0.50σ⁴. More generally, among estimators c·SS the MSE σ⁴[2(n − 1)c² + (c(n − 1) − 1)²] is minimised at c = 1/(n + 1), giving 2σ⁴/(n + 1). Unbiasedness and minimum MSE therefore need not coincide.

(b) Bayes estimators under a normal prior

Likelihood and prior. Since x̄ is sufficient for θ,

L(θ | x) ∝ exp[−n(θ − x̄)²/(2σ²)], and π(θ) ∝ exp[−(θ − μ)²/(2τ²)].

Posterior. Multiplying and completing the square in θ:

π(θ | x) ∝ exp{−½[(n/σ² + 1/τ²)θ² − 2θ(nx̄/σ² + μ/τ²)]}

So θ | x ~ N(μₙ, σₙ²) (the normal prior is conjugate), with

• Posterior precision: 1/σₙ² = n/σ² + 1/τ², i.e. σₙ² = σ²τ²/(nτ² + σ²)
• Posterior mean: μₙ = (nx̄/σ² + μ/τ²)/(n/σ² + 1/τ²) = (nτ²x̄ + σ²μ)/(nτ² + σ²)

(i) Squared error loss, L(θ, d) = (θ − d)². The posterior expected loss E[(θ − d)² | x] = V(θ | x) + (μₙ − d)² is minimised at d = E(θ | x). The Bayes estimator is the posterior mean:

θ̂_B = (nτ²x̄ + σ²μ)/(nτ² + σ²) = w x̄ + (1 − w)μ, where w = nτ²/(nτ² + σ²).

(ii) Absolute error loss, L(θ, d) = |θ − d|. d ↦ E|θ − d| has derivative P(θ < d | x) − P(θ > d | x), which is zero where the posterior probability is split equally. The Bayes estimator is therefore the posterior median. The normal posterior is symmetric about μₙ, so its median equals its mean:

θ̂_B = μₙ = (nτ²x̄ + σ²μ)/(nτ² + σ²), the same estimator as in (i).

The posterior (Bayes) risk under squared error is the posterior variance σₙ² = σ²τ²/(nτ² + σ²). This is smaller than both σ²/n (the variance of x̄) and τ² (the prior variance), because data and prior information are pooled.

Interpretation. The estimator is a precision-weighted average of the sample mean and the prior mean. As n → ∞ or τ² → ∞ (vague prior), w → 1 and θ̂_B → x̄. As τ² → 0 (confident prior), θ̂_B → μ.

(c) SPRT for the Poisson mean and its OC function

The SPRT. Wald's sequential probability ratio test for H₀: θ = θ₀ against H₁: θ = θ₁ takes observations one at a time. After the m-th observation it computes

λₘ = Πᵢ₌₁ᵐ f(xᵢ, θ₁)/f(xᵢ, θ₀).

With constants B < 1 < A, the rule is:

• accept H₀ if λₘ ≤ B;
• reject H₀ (accept H₁) if λₘ ≥ A;
• continue sampling if B < λₘ < A.

For prescribed risks α and β, Wald's approximations are A ≈ (1 − β)/α and B ≈ β/(1 − α). The test terminates with probability one. Among all tests with the same α and β it minimises the expected sample size under both H₀ and H₁ (Wald–Wolfowitz optimality). Using these approximate boundaries, the actual error probabilities α′ and β′ satisfy α′ + β′ ≤ α + β, so the risks are essentially preserved. In practice log λₘ is accumulated as a running sum of Zᵢ = log[f(xᵢ, θ₁)/f(xᵢ, θ₀)].

SPRT for Poisson(θ) (take θ₁ > θ₀). With f(x, θ) = e^(−θ)θˣ/x! and Sₘ = Σᵢ₌₁ᵐ xᵢ:

λₘ = e^(−m(θ₁ − θ₀)) (θ₁/θ₀)^(Sₘ)

log λₘ = Sₘ log(θ₁/θ₀) − m(θ₁ − θ₀).

Dividing log B < log λₘ < log A by log(θ₁/θ₀) > 0, sampling continues while

[log B + m(θ₁ − θ₀)]/log(θ₁/θ₀) < Sₘ < [log A + m(θ₁ − θ₀)]/log(θ₁/θ₀).

H₀ is accepted when Sₘ falls on or below the lower line and rejected when it reaches the upper line. The boundaries in the (m, Sₘ) plane are two parallel straight lines with common slope s = (θ₁ − θ₀)/log(θ₁/θ₀) and intercepts log B/log(θ₁/θ₀) (negative) and log A/log(θ₁/θ₀).

OC function. L(θ) = P_θ(accept H₀). For each θ, choose h = h(θ) ≠ 0 such that E_θ[(f(X, θ₁)/f(X, θ₀))ʰ] = 1. Wald's approximation is

L(θ) = (Aʰ − 1)/(Aʰ − Bʰ).

For the Poisson case:

E_θ[e^(−h(θ₁ − θ₀))(θ₁/θ₀)^(hX)] = e^(−h(θ₁ − θ₀)) · exp{θ[(θ₁/θ₀)ʰ − 1]} = 1.

This gives θ = h(θ₁ − θ₀)/[(θ₁/θ₀)ʰ − 1]. The OC curve is traced parametrically: let h run over the real line, compute θ(h) and L(θ(h)).

Checks.

• h = 1: θ = (θ₁ − θ₀)/(θ₁/θ₀ − 1) = θ₀, and L = (A − 1)/(A − B) = 1 − α.
• h = −1: θ = θ₁, and L = (A⁻¹ − 1)/(A⁻¹ − B⁻¹) = β.
• h → 0: θ → s = (θ₁ − θ₀)/log(θ₁/θ₀), and L → log A/(log A − log B).
• h → +∞: θ → 0 and L → 1. h → −∞: θ → ∞ and L → 0.

ASN function. Here Z = log[f(X, θ₁)/f(X, θ₀)] = X log(θ₁/θ₀) − (θ₁ − θ₀), so E_θ(Z) = θ log(θ₁/θ₀) − (θ₁ − θ₀). The average sample number is

E_θ(N) ≈ [L(θ) log B + (1 − L(θ)) log A]/[θ log(θ₁/θ₀) − (θ₁ − θ₀)].

At θ₀ and θ₁ this is usually well below the fixed sample size that a Neyman–Pearson test with the same α and β would need, which is the practical attraction of the SPRT. The ASN is largest near θ = s, where the evidence favours neither hypothesis and the path of Sₘ drifts between the two lines.