Postscript version of these notes

STAT 350: Lecture 16

Reading: There is no really relevant part of the text.

Theory of F and t tests

Independence: If $U_1, U_2, \ldots U_k$ are random variables then we call $U_1, \ldots,U_k$ independent if

$$P(U_1 \in A_1 , \ldots , U_k \in A_k) = P(U_1 \in A_1 ) \times \cdots \times P(U_k \in A_k)$$

for any sets $A_1, \ldots , A_k$.

We usually either:

• Assume independence because there is no physical way for the value of any of the random variables to influence any of the others.

  OR

• We prove independence.

How do we prove independence? We use the notion of a joint density.

We say $U_1, \ldots,U_k$ have joint density function $f =f(u_1,\ldots,u_k)$ if

$$P((U_1, \ldots,U_k) \in A) = \idotsint\limits_A f(u_1,\ldots,u_k) du_1 \cdots du_k$$

We are interested here in joint densities because independence of $U_1, \ldots,U_k$ is equivalent to

$$f(u_1,\ldots,u_k) = f_1(u_1) \times \cdots \times f_k(u_k)$$

for some densities $f_1 ,\ldots,f_k$. In this case f_i is the density of U_i.

[ ASIDE: notice that for an independent sample the joint density is the likelihood function!]

Application to Normals: Standard Case

If

$$Z = \left[ \begin{array}{c} Z_1 \\ \vdots \\ Z_n\end{array} \right] \sim MVN(0,I_{n \times n})$$

then the joint density of Z, denoted $f_Z(z_1,\ldots,z_n)$ is

$$f_Z(z_1,\ldots,z_n) = \phi(z_1) \times \cdots \times \phi(z_n)$$

where

$$\phi(z_i) = \frac{1}{\sqrt{2\pi}} e^{-z_i^2/2}$$

So
$$\begin{align*}f_Z & = (2 \pi)^{-n/2} \exp\left\{ -\frac{1}{2} \sum_{i=1}^n z_i^2 \right\} \\ & = (2 \pi)^{-n/2} \exp\left\{ -\frac{1}{2}z^Tz\right\} \end{align*}$$
where

$$z=\left[ \begin{array}{c} z_1 \\ \vdots \\ z_n \end{array} \right]$$

Application to Normals: General Case

If $X=AZ+\mu$ and A is invertible then for any set $B \in R^n$ we have
$$\begin{align*}P(X \in B) & = P(AZ+\mu \in B) \\ & =P(Z \in A^{-1}(B-\mu)) \\ &... ...2 \pi)^{-n/2} \exp\left\{ -\frac{1}{2}z^Tz\right\} dz_1 \cdots dz_n \end{align*}$$
Make the change of variables $x=Az+\mu$ in this integral to get

$$P(X \in B) = \idotsint\limits_B (2 \pi)^{-n/2} \exp\left\{-\f... ...^T \left( A^{-1} (x-\mu)\right) \right\} J(x) dx_1 \cdots dx_n$$

where J(x) denotes the Jacobian of the transformation

$$J(x) = J(x_1,\ldots,x_n) = \left\vert {\rm det}\left( \frac{\... ...t\vert = \left\vert {\rm det}\left( A^{-1} \right) \right\vert$$

Algebraic manipulation of the integral then gives

$$P(X \in B) = \idotsint\limits_B (2 \pi)^{-n/2} \exp\left\{-\f... ... (x-\mu) \right\} \vert{\rm det } A^{-1} \vert dx_1\cdots dx_n$$

where
$$\begin{align*}\Sigma & = AA^T \\ \Sigma^{-1} & = \left( A^{-1}\right)^T \left( ... ...\left({\rm det} A^{-1}\right)^2 \\ & = \frac{1}{{\rm det} \Sigma} \end{align*}$$
The conclusion of this algebra is that the $MVN(\mu,\Sigma)$ density is

$$(2 \pi)^{-n/2} \exp\left\{- \frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right\} ({\rm det} \Sigma)^{-1/2}$$

What if A is not invertible? Ans: there is no density.

How do we apply this density?

Suppose

$$X = \left[\begin{array}{c} X_1 \\ \hline X_2 \end{array}\right]$$

and

$$\Sigma = \left[\begin{array}{c\vert c} \Sigma_{11} & \Sigma_{12} \\ \hline \Sigma_{21} & \Sigma_{22} \end{array}\right]$$

If $\Sigma_{12}=0$ then

1.

$\Sigma_{21} = 0$

2.

In the homework you will verify that

$$\Sigma^{-1} = \left[\begin{array}{c\vert c} \Sigma^{-1}_{11} & 0 \\ \hline 0 & \Sigma^{-1}_{22} \end{array}\right]$$

3.

Writing

$$x = \left[ \begin{array}{c} x_1 \\ \hline x_2 \end{array} \right]$$

and

$$\mu = \left[ \begin{array}{c} \mu_1 \\ \hline \mu_2 \end{array} \right]$$

we find

$$(x-\mu)^T \Sigma^{-1} (x-\mu) = (x_1-\mu_1)^T \Sigma^{-1}_{11} (x_1-\mu_1) + (x_2-\mu_2)^T \Sigma^{-1}_{22} (x_2-\mu_2)$$

4.

So, if $n_1={\rm dim}(X_1)$ and $n_2 = {\rm dim}(X_2)$ we see that
$$\begin{align*}f_X(x_1,x_2) = (2\pi)^{-n_1/2}& \exp\left\{-\frac{1}{2}(x_1-\mu_1)... ...eft\{-\frac{1}{2}(x_2-\mu_2)^T \Sigma^{-1}_{22} (x_2-\mu_2)\right\} \end{align*}$$
so that X₁ and X₂ are independent.

Summary: If ${\rm Cov}(X_1,X_2) = {\rm E}[(X_1-\mu_1)(X_2-\mu_2)^T]=0$ then X₁ is independent of X₂.

Warning: This only works provided

$$X = \left[\begin{array}{c} X_1 \\ \hline X_2 \end{array}\right] \sim MVN(\mu,\Sigma)$$

Fact: However, it works even if $\Sigma$ is singular, but you can't prove it as easily using densities.

Application:

$$\begin{align*}\hat\mu & = X\hat\beta = X(X^TX)^{-1} X^T Y \\ & = X\beta + H \ep... ...= Y - X\hat\beta \\ & = \epsilon -H\epsilon \\ & = (I-H)\epsilon \end{align*}$$
So

$$\left[ \begin{array}{c} \hat\mu \\ \hline \hat\epsilon \end{a... ... \left[ \begin{array}{c} \mu \\ \hline 0 \end{array}\right]}_b$$

Hence

$$\left[ \begin{array}{c} \hat\mu \\ \hline \hat\epsilon \end{a... ...begin{array}{c} \mu \\ \hline 0 \end{array}\right];AA^T\right)$$

Now

$$A = \sigma\left[ \begin{array}{c} H \\ \hline I-H \end{array}\right]$$

so
$$\begin{align*}AA^T & = \sigma^2 \left[ \begin{array}{c} H \\ \hline I-H \end{arr... ...sigma^2 \left[ \begin{array}{cc} H & 0 \\ 0 & I-H\end{array}\right] \end{align*}$$
The 0s prove that $\hat\epsilon$ and $\hat\mu$ are independent. It follows that $\hat\mu^T\hat\mu$, the regression sum of squares (not adjusted) is independent of $\hat\epsilon^T \hat\epsilon$, the Error sum of squares.

Joint Densities

Suppose Z₁ and Z₂ are independent standard normals. In class I said that their joint density was

$$f(z_1,z_2) = \frac{1}{2\pi} \exp(-(z_1^2+z_2^2)/2) \, .$$

Here I want to show you the meaning of joint density by computing the density of a $\chi_2^2$ random variable.

Let U = Z₁²+Z₂². By definition U has a $\chi^2$ distribution with 2 degrees of freedom. The cumulative distribution function of U is

$$F(u) = P(U \le u).$$

For $u \le 0$ this is 0 so take $u \ge 0$. The event that $U\le u$ is the same as the event that the point (Z₁,Z₂) is in the circle centered at the origin and having radius u^1/2, that is, if A is the circle of this radius then

$$F(u) = P((Z_1,Z_2)\in A) \, .$$

By definition of density this is a double integral

$$\int\int_A f(z_1,z_2) \, dz_1\, dz_2 \, .$$

You do this integral in polar co-ordinates. Letting $z_1 = r\cos\theta$ and $z_2 = r\sin\theta$ we see that

$$f(r\cos\theta,r\sin\theta) = \frac{1}{2\pi} \exp(-r^2/2) \, .$$

The Jacobian of the transformation is r so that $dz_1\,dz_2$ becomes $r\, dr\, d\theta$. Finally the region of integration is simply $0 \le \theta \le 2\pi$ and $0 \le r \le u^{1/2}$ so that

$$\begin{eqnarray*}P(U \le u)& =& \int_0^{u^{1/2}}\int_0^{2\pi} \frac{1}{2\pi} \ex... ...xp(-r^2/2) \right\vert _0^{u^{1/2}} \\ & = & 1-\exp(-u/2) \, . \end{eqnarray*}$$

The density of U can be found by differentiating to get

$$f(u) = \frac{1}{2} \exp(-u/2)$$

which is the exponential density with mean 2. This means that the $\chi^2_2$ density is really an exponential density.