(a) How will you justify the usage of the principle of least squares in estimating the parameters of a linear regression model?…

(a) Justifying Least Squares; Deriving β̂ = (X′X)⁻¹X′y

The principle of least squares is justified because it chooses the parameter vector that makes the fitted regression plane closest to the observed data in the Euclidean sense. The residual sum of squares is

RSS(β) = ∑ᵢ (yᵢ - xᵢ'β)² = (y - Xβ)'(y - Xβ).

Minimizing this does not require normality. Under the Gauss-Markov assumptions E(ε)=0, Var(ε)=σ²I, and X full column rank, the least-squares estimator is the best linear unbiased estimator. If ε is normal, it is also the maximum-likelihood estimator. Geometrically, Xβ̂ is the orthogonal projection of y on the column space of X.

Expand RSS:

RSS(β) = y'y - 2β'X'y + β'X'Xβ.

Differentiate with respect to β:

∂RSS/∂β = -2X'y + 2X'Xβ.

The second derivative 2X'X is positive definite when X has full column rank, so the stationary point is a minimum. Set the derivative to zero:

X'Xβ = X'y.

These are the normal equations. If X'X is nonsingular,

β̂ = (X'X)⁻¹X'y.

Condition of validity: X is of full column rank p, so X'X is invertible.

(b)(i) Distribution of (X₁ − X₂, X₂ − X₃)′

Let

Y = [X₁ - X₂; X₂ - X₃] = A X, where A = [[1, -1, 0], [0, 1, -1]].

If X ~ N₃(μ, Σ), then by the affine transformation property of the multivariate normal distribution,

Y ~ N₂(Aμ, AΣA').

Here μ = (μ₁, μ₂, μ₃)'. Thus

E(Y) = (μ₁ - μ₂, μ₂ - μ₃)'.

Let Σ = (σᵢⱼ). Then

Var(X₁ - X₂) = σ₁₁ + σ₂₂ - 2σ₁₂,

Var(X₂ - X₃) = σ₂₂ + σ₃₃ - 2σ₂₃,

Cov(X₁ - X₂, X₂ - X₃) = σ₁₂ - σ₁₃ - σ₂₂ + σ₂₃.

Hence

Y ~ N₂( (μ₁ - μ₂, μ₂ - μ₃)', [[σ₁₁ + σ₂₂ - 2σ₁₂, σ₁₂ - σ₁₃ - σ₂₂ + σ₂₃], [σ₁₂ - σ₁₃ - σ₂₂ + σ₂₃, σ₂₂ + σ₃₃ - 2σ₂₃]] ).

(b)(ii) Partial Correlation ρ₁₂·₃ in Terms of Simple Correlations

The partial correlation between X₁ and X₂ eliminating X₃ is the correlation between the residuals of X₁ and X₂ after regression on X₃. For the trivariate normal distribution,

Cov(X₁, X₂ | X₃) = σ₁₂ - σ₁₃σ₂₃/σ₃₃,

Var(X₁ | X₃) = σ₁₁ - σ₁₃²/σ₃₃ = σ₁₁(1 - ρ₁₃²),

Var(X₂ | X₃) = σ₂₂ - σ₂₃²/σ₃₃ = σ₂₂(1 - ρ₂₃²).

Therefore

ρ₁₂·₃ = [σ₁₂ - σ₁₃σ₂₃/σ₃₃] / √[(σ₁₁ - σ₁₃²/σ₃₃)(σ₂₂ - σ₂₃²/σ₃₃)].

Divide numerator and denominator by √(σ₁₁σ₂₂). Since ρ₁₂ = σ₁₂/√(σ₁₁σ₂₂), ρ₁₃ = σ₁₃/√(σ₁₁σ₃₃), and ρ₂₃ = σ₂₃/√(σ₂₂σ₃₃),

ρ₁₂·₃ = (ρ₁₂ - ρ₁₃ρ₂₃) / √((1 - ρ₁₃²)(1 - ρ₂₃²)).

This requires |ρ₁₃| < 1 and |ρ₂₃| < 1.

(c) Showing (n₁n₂/n)D² ~ T²(p, n − 2)

Let x̄₁ and x̄₂ be the sample mean vectors of the two independent samples, and let S₁ and S₂ be their sample covariance matrices. Define the pooled covariance matrix

S = [(n₁ - 1)S₁ + (n₂ - 1)S₂]/(n - 2), n = n₁ + n₂.

The Mahalanobis distance is

D² = (x̄₁ - x̄₂)' S⁻¹ (x̄₁ - x̄₂).

We must show that

T² = (n₁n₂/n)D² ~ T²(p, n - 2).

Since the samples are independent and come from Nₚ(μ, Σ),

x̄₁ ~ Nₚ(μ, Σ/n₁), x̄₂ ~ Nₚ(μ, Σ/n₂).

Hence

d = x̄₁ - x̄₂ ~ Nₚ(0, Σ(1/n₁ + 1/n₂)) = Nₚ(0, Σ n/(n₁n₂)).

Let

A = (n₁ - 1)S₁ + (n₂ - 1)S₂.

By the additivity property of Wishart matrices,

A ~ Wₚ(Σ, n - 2),

and A is independent of d. Also S = A/(n - 2).

Now define

y = √(n₁n₂/n) d.

Then y ~ Nₚ(0, Σ). Also

T² = (n₁n₂/n)d'S⁻¹d

= (n₁n₂/n)(n - 2)d'A⁻¹d

= (n - 2)y'A⁻¹y.

By the standard Hotelling T² theorem, if y ~ Nₚ(0, Σ) independently of A ~ Wₚ(Σ, ν), then ν y'A⁻¹y ~ T²(p, ν). Taking ν = n - 2,

(n₁n₂D²)/n ~ T²(p, n − 2).

This is the central Hotelling T² distribution under equality of the two population mean vectors.

(d) Inclusion Probabilities and the Stratified Design

The six possible samples, each with probability 1/6, are

S₁ = {a,b,d}, S₂ = {a,b,e}, S₃ = {a,d,e},

S₄ = {b,c,d}, S₅ = {b,c,e}, S₆ = {c,d,e}.

Count appearances of each unit:

a appears in S₁,S₂,S₃, so π(a) = 3/6 = 1/2.

b appears in S₁,S₂,S₄,S₅, so π(b) = 4/6 = 2/3.

c appears in S₄,S₅,S₆, so π(c) = 3/6 = 1/2.

d appears in S₁,S₃,S₄,S₆, so π(d) = 4/6 = 2/3.

e appears in S₂,S₃,S₅,S₆, so π(e) = 4/6 = 2/3.

Thus

π(a)=1/2, π(b)=2/3, π(c)=1/2, π(d)=2/3, π(e)=2/3.

For second-order inclusion probabilities π(ij), count samples containing both i and j:

π(ab)=2/6=1/3, π(ac)=0, π(ad)=2/6=1/3, π(ae)=2/6=1/3,

π(bc)=2/6=1/3, π(bd)=2/6=1/3, π(be)=2/6=1/3,

π(cd)=2/6=1/3, π(ce)=2/6=1/3, π(de)=2/6=1/3.

This is a stratified design with strata

U₁ = {a,c}, sample size 1,

U₂ = {b,d,e}, sample size 2.

Selecting one unit from U₁ and two units from U₂ by simple random sampling without replacement gives six possible samples, each with probability (1/2)(1/3)=1/6. These are exactly S₁ to S₆. Also π(a)=π(c)=1/2; π(b)=π(d)=π(e)=2/3; within U₁, π(ac)=0; within U₂, π(bd)=π(be)=π(de)=1/3; between strata, the product rule gives 1/3.

Hence the design is stratified with strata {a,c} and {b,d,e}.

(e)(i) The Design Is Connected and Balanced

Let rows of N correspond to treatments and columns to blocks. Each row sum is 3, so each treatment occurs r = 3 times. Each column sum is 3, so each block contains k = 3 treatments.

For any two distinct treatments, the number of blocks containing both is the dot product of the corresponding rows. For example,

row₁·row₂ = 1·1 + 1·1 + 1·0 + 0·1 = 2.

Similarly every off-diagonal row dot product equals 2. Thus every pair of treatments occurs together λ = 2 times. Therefore the design is balanced.

Now

N N' = [[3,2,2,2],[2,3,2,2],[2,2,3,2],[2,2,2,3]] = I + 2J.

The C-matrix is

C = rI - (1/k)N N' = 3I - (1/3)(I + 2J)

= (8/3)I - (2/3)J.

For the vector 1, J1 = 4·1, so C1 = (8/3)1 - (2/3)(4·1) = 0.

For any vector v orthogonal to 1, Jv = 0, so Cv = (8/3)v.

Thus C has rank 3 = v - 1, with only one zero eigenvalue corresponding to the overall mean contrast. Hence all treatment contrasts are estimable and the design is connected.

Therefore the design is connected and balanced.

(e)(ii) Efficiency Factor E = 8/9

For a balanced incomplete block design, the efficiency factor is

E = λv/(rk).

Here v = 4, r = 3, k = 3, and λ = 2. Hence

E = (2 × 4)/(3 × 3) = 8/9.

Equivalently, the non-zero eigenvalue of C is 8/3, so each canonical efficiency factor is

(8/3)/r = (8/3)/3 = 8/9.

Since all non-zero canonical efficiency factors are equal, their harmonic mean is also 8/9.

Therefore E = 8/9.