Postscript version of this file

STAT 870 Lecture 3

Goals of Today's Lecture:

• Define independence of events, random variables and $\sigma$ fields.

• Illustrate monotone class arguments.

Independence, conditional distributions

Events A and B independent if

$$P(AB) = P(A)P(B) \, .$$

Events A_i, $i=1,\ldots,p$ are independent if

$$P(A_{i_1} \cdots A_{i_r}) = \prod_{j=1}^r P(A_{i_j})$$

for any set of distinct indices $i_1,\ldots,i_r$ between 1 and p.

Example: p=3

$$\begin{eqnarray*}P(A_1A_2A_3) & = & P(A_1)P(A_2)P(A_3) \\ P(A_1A_2) & = & P(A_1... ...\ P(A_1A_3) & = & P(A_1)P(A_3) \\ P(A_2A_3) & = & P(A_2)P(A_3) \end{eqnarray*}$$

Need all equations to be true for independence!

Example: Toss a coin twice. If A₁ is the event that the first toss is a Head, A₂ is the event that the second toss is a Head and A₃ is the event that the first toss and the second toss are different. then P(A_i) =1/2 for each i and for $i \neq j$

$$P(A_i \cap A_j) = \frac{1}{4}$$

but

$$P(A_1 \cap A_2 \cap A_3) = 0 \neq P(A_1)P(A_2)P(A_3) \, .$$

Rvs $X_1,\ldots,X_p$ are independent if

$$P(X_1 \in A_1, \cdots , X_p \in A_p ) = \prod P(X_i \in A_i)$$

for any choice of $A_1,\ldots,A_p$.

$\sigma$-fields ${\cal F}_1,\ldots,{\cal F}_p$ are independent if

$$P( A_1 \cap \cdots \cap A_p ) = \prod P( A_i)$$

for any choice of events $A_1\in {\cal F}_1,\ldots,A_p\in {\cal F}_p$.

Theorem:

1: If X and Y are independent then

F_X,Y(x,y) = F_X(x)F_Y(y)

for all x,y

2: If X and Y are independent and have joint density f_X,Y(x,y) then X and Y have densities, say f_X and f_Y, and for almost every pair (x,y)

$$f_{X,Y}(x,y) = f_X(x) f_Y(y) \, .$$

3: If X, Y independent with marginal densities f_X and f_Y then (X,Y) has joint density f_X,Y(x,y) given by

$$f_{X,Y}(x,y) = f_X(x) f_Y(y) \, .$$

4: If

F_X,Y(x,y) = F_X(x)F_Y(y)

for all x,y then X and Y are independent.

5: If (X,Y) has density f(x,y) and there are functions g(x) and h(y) such that

f(x,y) = g(x) h(y)

for almost all (x,y) then X and Y are independent and they each have a density given by

$$f_X(x) = g(x)/\int_{-\infty}^\infty g(u) du$$

and

$$f_Y(y) = h(y)/\int_{-\infty}^\infty h(u) du \, .$$

Proof:

1: By definition $X \le x$, $Y \le y$ independent so

$$P(X \le x, Y \le y) = P(X \le x)P(Y \le y)$$

2: Suppose $X\in{\Bbb R}^p$, $Y\in{\Bbb R}^q$, $B\subset{\Bbb R}^q$ Borel. Then $\{Y \in A\} =\{(X,Y) \in {\Bbb R}^p \times B\}$. Hence
$$\begin{align*}P(Y \in B) & = P((X,Y) \in {\Bbb R}^p \times B) \\ & = \int_{{\Bbb R}^p \times B} f_{X,Y}(x,y) dx \, dy \\ & \equiv \int_B g(y) dy \end{align*}$$
so Y has density g defined by

$$g(y) = \int_{{\Bbb R}^p} f_{X,Y}(x,y) dx$$

Similarly for X. So for Borel $A\subset{\Bbb R}^p$, $B\subset{\Bbb R}^q$
$$\begin{align*}P(X \in A, Y\in B) &= \int_A\int_B f_{X,Y}(x,y) dydx \\ P(X\in A)... ...t_A f_X(x)dx \int_B f_Y(y) dy \\ &= \int_A\int_B f_X(x)f_Y(y) dydx \end{align*}$$
Using Fubini's theorem here.

Since $P(X \in A, Y\in B) =P(X\in A)P(Y\in B)$ we see that for any sets A and B

$$\int_A\int_B [ f_{X,Y}(x,y) - f_X(x)f_Y(y) ]dydx = 0$$

Hence f_X,Y(x,y) = f_X(x) f_Y(y) ae.

3: For any A and B Borel
$$\begin{align*}P(X \in A, Y \in B) & = P(X\in A)P(Y \in B) \\ &= \int_Af_X(x)dx \int_B f_Y(y) dy \\ &= \int_A\int_B f_X(x)f_Y(y) ]dydx \end{align*}$$
Define g(x,y) = f_X(x)f_Y(y). For $C=A \times B$

$$P( (X,Y) \in C) = \int_C g(x,y)dy dx$$

Now do general Borel C via a monotone class argument.

Monotone Class: collection ${\cal C}$ of subsets of a given set, closed under increasing countable unions and decreasing countable intersections:

1.

If $A_1\subset A_2 \subset \cdots$ are in ${\cal C}$ then $\bigcup_1^\infty A_i \in {\cal C}$.

2.

If $A_1\supset A_2 \supset \cdots$ are in ${\cal C}$ then $\bigcap_1^\infty A_i \in {\cal C}$.

Lemma: The smallest monotone class containing a field $\cal F_o$ is a $\sigma$-field.

Field: like $\sigma$-field but only finite unions and intersections.

Proof of Lemma: Let ${\cal C}$ be smallest monotone class containing $\cal F_o$ (intersection of all monotone classes containing $\cal F_o$). Put ${\cal M} = \{ A\in {\cal C}: A^c \in {\cal C}\}$.

• $\cal M$ is a monotone class.

• Since $\cal F_o$ is a field and a subset of ${\cal C}$, $\cal M$ contains $\cal F_o$.

• Since ${\cal C}$ is smallest monotone class containing $\cal F_o$ and $\cal M$ is another monotone class containing $\cal F_o$ we see ${\cal C} \subset {\cal M}$.

• This means every $A\in {\cal C}$ has the property $A^c\in {\cal C}$. In other words ${\cal C}$ is closed under the operation of taking complements, one of the defining properties of a $\sigma$-field.

• Similar argument for finite unions in notes.

• So ${\cal C}$ is a field. Since ${\cal C}$ is monotone class and field ${\cal C}$ is a $\sigma$-field.

Now apply lemma to 3.

Rectangle: set $C=A \times B$ where A, B Borel.

$$P((X,Y) \in C) = \int _C f_X(x) f_Y(y) dx dy$$ (1)

for each rectangle C.

A finite union of rectangles may be written as a finite disjoint union of rectangles so (1) holds for any such C.

The complement of a rectangle may be rewritten as a finite disjoint union of rectangles.

The collection $\cal M$ of subsets of ${\Bbb R}^{p+q}$ for which (1) holds contains the field of all finite disjoin unions of rectangles.

$\cal M$ is a monotone class by the dominated convergence theorem.

Application of the dominated convergence theorem.

$g(x,y)=f_X(x) f_Y(y)\ge 0$ has integral equal to 1 by Tonelli's theorem. If $C_1 \subset C_2 \subset \cdots$put

$$g_n(x,y) = g(x,y) 1(x \in C_n)$$

Note that $\vert g_n(x,y)\vert \le \vert g(x,y)\vert$ and that g_n converges for almost all x to

$$g_\infty(x,y) = g(x,y) 1(x \in \cup C_n)$$

Thus
$$\begin{align*}\lim_{n\to \infty} \int g_n(x,y) dx dy & = \lim_{n\to \infty} \int... ...int g_\infty(x,y) dx dy \\ & = \int_{\cup C_n} f_X(x) f_Y(y) dx dy \end{align*}$$
At the same time

$$\lim P((X,y) \in C_n) = P((X,Y) \in \cup C_n)$$

so that

$$P((X,Y) \in \cup C_n) = \int_{\cup C_n} f_X(x) f_Y(y) dx dy$$

This and a corresponding argument for intersections show that $\cal M$ is a monotone class containing the field of all finite unions of Borel rectangles.

So $\cal M$ contains smallest $\sigma$ field containing this field.

Every open rectangle is in this field so in $\cal M$.

Every open set is a countable union of open rectangles and so in $\cal M$.

Thus $\cal M$ is a $\sigma$ field containing the open sets and so must contain the Borel $\sigma$ field.

4: Monotone class argument: Homework 1.

5:
$$\begin{align*}P(X \in A, Y \in B) & = \int_A \int_B g(x) h(y) dy dx \\ & = \int_A g(x) dx \int_B h(y) dy \end{align*}$$
Take $B={\Bbb R}^1$ to see that

$$P(X \in A ) = c_1 \int_A g(x) dx$$

where $c_1 = \int h(y) dy$. Def'n of density shows c₁ g is density of X. Then $\int\int f_{X,Y}(xy)dxdy = 1$ so $\int g(x) dx \int h(y) dy = 1$ and $c_1 = 1/\int g(x) dx$. Similar argument for Y.

Theorem 1   If $X_1,\ldots,X_p$ are independent and Y_i =g_i(X_i) then $Y_1,\ldots,Y_p$ are independent. Moreover, $(X_1,\ldots,X_q)$ and $(X_{q+1},\ldots,X_{p})$ are independent.

Conditional probability

Def'n: P(A|B) = P(AB)/P(B) if $P(B) \neq 0$.

Def'n: For discrete rvs X, Y conditional pmf of Y given X is
$$\begin{align*}f_{Y\vert X}(y\vert x) &= P(Y=y\vert X=x) \\ &= f_{X,Y}(x,y)/f_X(x) \\ &= f_{X,Y}(x,y)/\sum_t f_{X,Y}(x,t) \end{align*}$$

Problem for absolutely continuous X: P(X=x) = 0 for all x. Solution is to take a limit

$$P(A\vert X=x) = \lim_{\delta x \to 0} P(A\vert x \le X \le x+\delta x)$$

If X,Y have joint density f_X,Y then with $A=\{ Y \le y\}$ we have
$$\begin{multline*}P(A\vert x \le X \le x+\delta x) \\ = \frac{P(A \cap \{ x \le ... ...\delta x} f_{X,Y}(u,v)dudv }{ \int_x^{x+\delta x} f_X(u) du } \end{multline*}$$
Divide top and bottom by $\delta x$; let $\delta x\to 0$. Denominator converges to f_X(x); numerator converges to

$$\int_{-\infty}^y f_{X,Y}(x,v) dv$$

So we define the conditional CDF of Y given X=x to be

$$P(Y \le y \vert X=x) = \frac{ \int_{-\infty}^y f_{X,Y}(x,v) dv }{ f_X(x) }$$

Differentiate wrt y to get definition of conditional density of Y given X=x:

f_Y|X(y|x) = f_X,Y(x,y)/f_X(x)

or in words ``conditional = joint/marginal''.