Postscript version of these notes

STAT 804 -- Lecture 2

We are investigating assumptions on a discrete time process which will permit us to make reasonable estimates of the parameters. We will look for assumptions which guarantee at least the existence

Definition: A stochastic process $X_t; t = 0, \pm 1, \ldots$ is stationary if the joint distribution of $X_t, \cdots,X_{t+k}$ is the same as the joint distribution of $X_0,\cdots,X_k$ for all t and all k. (Often we call this strictly stationary.)

Definition: A stochastic process $X_t; t = 0, \pm 1, \ldots$ is weakly (or second order) stationary if

$${\rm E}(X_t) \equiv \mu$$

for all t (that is the mean does not depend on t) and

$${\rm Cov}(X_t,X_{t+h}) = {\rm Cov}(X_0,X_{h})\equiv C_X(h)$$

is a function of h only (and does not depend on t).

Remark:

1.

X finite variance, strictly stationary implies X weakly stationary.

2.

X second order stationary and Gaussian implies X strictly stationary.

Definition: X is Gaussian if, for each $t_1,\ldots,t_k$ the vector $(X_{t_1},\ldots,X_{t_k})$ has a Multivariate Normal Distribution.

Examples of Stationary Processes:

1) Strong Sense White Noise: A process $\epsilon_t$ is strong sense white noise if $\epsilon_t$ is iid with mean 0 and finite variance $\sigma^2$.

2) Weak Sense White Noise: $\epsilon_t$ is second order stationary with

$${\rm E}(\epsilon_t) = 0$$

and

$${\rm Cov}(\epsilon_t,\epsilon_s) = \begin{cases} \sigma^2 & s=t \\ 0 & s \neq t \end{cases}$$

In this course we always use $\epsilon_t$ as notation for white noise and $\sigma^2$ as the variance of this white noise. We use subscripts to indicate variances of other things.

Example Graphics:

White noise: iid N(0,1) data

White noise: $X_t = \epsilon_t \cdots \epsilon_{t+9}$

2) Moving Averages: if $\epsilon_t$ is white noise then $X_t = (\epsilon_t + \epsilon_{t-1})/2$ is stationary. (If you use second order white noise you get second order stationary. If the white noise is iid you get strict stationarity.)

Example proof: ${\rm E}(X_t) = \left[ {\rm E}(\epsilon_t) +{\rm E}(\epsilon_{t-1})\right] /2 = 0$ which is constant as required. Moreover:

$${\rm Cov}(X_t,X_s) = \begin{cases} \frac{{\rm Var}(\epsilon_t... ...epsilon_{t+2} + \epsilon_{t+1} & s=t + 2 \\ \vdots \end{cases}$$

Most of these covariances are 0. For instance

$${\rm Cov}(\epsilon_t +\epsilon_{t-1},\epsilon_{t+2} + \epsilo... ...,\epsilon_{t+2} ) +{\rm Cov}(\epsilon_{t-1},\epsilon_{t+1}) =0$$

because the $\epsilon$s are uncorrelated by assumption. The only non-zero covariances occur for s=t and $s=t \pm 1$. Since ${\rm Cov}(\epsilon_t,\epsilon_t)=\sigma^2$we get

$${\rm Cov}(X_t,X_s) = \begin{cases} \sigma^2 & s=t\\ \frac{\sigma^2}{4} & \vert s-t\vert=1 \\ 0 & \text{otherwise} \end{cases}$$

Notice that this depends only on |s-t| so that the process is stationary.

The proof that X is strictly stationary when the $\epsilon$s are iid is in your homework; it is quite different.

Example Graphics:

$X_t = (\epsilon_t + \epsilon_{t-1})/2$

$X_t = \epsilon_t +6\epsilon_{t-1} + 15\epsilon_{t-2} + 20\epsilon_{t-3} +15\epsilon_{t-4} + 6\epsilon_{t-5} + \epsilon_{t-6}$

The trajectory of X can be made quite smooth (compared to that of white noise) by averaging over many $\epsilon$s.

3) Autoregressive Processes:

An AR(1) process X is a process satisfying the equations:

$$X_t = \mu+\rho(X_{t-1}-\mu) + \epsilon_t$$ (1)

where $\epsilon$ is wide noise. If X_t is second order stationary with ${\rm E}(X_t) = \theta$, say, then take expected values of (1) to get

$$\theta = \mu+\rho(\theta-\mu)$$

which we solve to get

$$\theta(1-\rho) = \mu(1-\rho) \, .$$

Thus either $\rho=1$ (which will actually guarantee that X is not stationary) or $\theta=\mu$. Now we calculate variances:
$$\begin{align*}{\rm Var}(X_t) & = {\rm Var}(\mu+\rho(X_{t-1}-\mu)+\epsilon_t) \\ ... ...) + 2\rho {\rm Cov}(X_{t-1},\epsilon_t) + \rho^2 {\rm Var}(X_{t-1}) \end{align*}$$
We now assume that the meaning of (1) is that $\epsilon_t$ is uncorrelated with $X_{t-1},X_{t-2},\cdots$. In the strictly stationary case we are imagining that somehow X_t-1is built up out of past values of $\epsilon_s$ which are independent of $\epsilon_t$. In the weakly stationary case we are imagining that X_t-1is actually a linear function of these past values. In either case this makes ${\rm Cov}(X_{t-1},\epsilon_t) =0$. If X is stationary so that ${\rm Var}(X_t) = {\rm Var}(X_{t-1}) \equiv \sigma^2_X$ then we find

$$\sigma^2_X = \sigma^2+\rho^2 \sigma^2_X$$

whose solution is

$$\sigma^2_X =\frac{\sigma^2}{1-\rho^2}$$

Notice that this variance is negative or undefined unless $\vert\rho\vert<1$. There is no stationary process satisfying (1) for $\vert\rho\vert \ge 1$.

Now for $\vert\rho\vert , 1$ how is X_t determined from the $\epsilon_s$? (We want to solve the equations (1) to get an explicit formula for X_t.) The case $\mu=0$ is notationally simpler. We get
$$\begin{align*}X_t & = \epsilon_t + \rho X_{t-1} \\ & = \epsilon_t + \rho( \eps... ...\epsilon_t + \rho\epsilon_{t-1}+\cdots +\rho^{k-1} + \rho^k X_{t-k} \end{align*}$$
Since $\vert\rho\vert<1$ it seems reasonable to suppose that $\rho^kX_{t-k} \to 0$ and for a stationary series X this is true in the appropriate mathematical sense. This leads to taking the limit as $k\to \infty$ to get

$$X_t = \sum_{j=0}^\infty \rho^j \epsilon_{t-j} \, .$$

Claim: It is a theorem that if $\epsilon$ is a weakly stationary series then $X_t = \sum_{j=0}^\infty \rho^j \epsilon_{t-j}$ converges (technically it converges in mean square) and is a second order stationary solution to the equation (1). If $\epsilon$ is a strictly stationary process then under some weak assumptions about how heavy the tails of $\epsilon$ are $X_t = \sum_{j=0}^\infty \rho^j \epsilon_{t-j}$ converges almost surely and is a strongly stationary solution of (1).

In fact if $\ldots,a_{-2},a_{-1},a_0,a_1,a_2,\ldots$ are constants such that $\sum a_j^2 < \infty$ and $\epsilon$ is weakly stationary (respectively strongly stationary with finite variance) then

$$X_t = \sum_{j=-\infty}^\infty a_j \epsilon_{t-j}$$

is weakly stationary (respectively strongly stationary with finite variance). In this case we call X a linear filter of $\epsilon$.

Example Graphics:

Motivation of the jargon ``filter'' comes from physics. Consider an electric circuit with a resistance R in series with a capacitance C. We apply an ``input'' voltage U(t) across the two elements and measure the voltage drop across the capacitor. We will call this voltage drop the ``output'' voltage and denote the output voltage by X_t. The relevant physical rules are these:

1.

The total voltage drop around the circuit is 0. This drop is -U(t) plus the voltage drop across the resistor plus X(t). (The negative sign is a convention; the input voltage is not a ``drop''.)

2.

The voltage drop across the resistor is Ri(t) where i is the current flowing in the circuit.

3.

If the capacitor starts off with no charge on its plates then the voltage drop across its plates at time t is

$$X(t) = \frac{\int_0^t i(s) \, ds}{C}$$

These rules give

$$U(t) = Ri(t) +\frac{\int_0^t i(s) \, ds}{C}$$

Differentiate the definition of X to get

$$X^\prime(t) = i(t)/C$$

so that

$$U(t) =RCX^\prime(t) +X(t) \, .$$

Multiply by e^t/RC/RC to see that

$$\frac{e^{t/RC} U(t)}{RC} = \left(e^{t/RC}X(t)\right)^\prime$$

whose solution, remembering X(0)=0, is obtained by integrating from 0 to s to get

$$e^{s/RC}X(s) = \frac{1}{RC}\int_0^s e^{t/RC} U(t) \, dt$$

leading to
$$\begin{align*}X(s) & = \frac{1}{RC}\int_0^s e^{(t-s)/RC} U(t) \, dt \\ & = \frac{1}{RC}\int_0^s e^{u/RC} U(t-u) \, du \end{align*}$$
This formula is the integral equivalent of our definition of filter and shows $X = \text{filter}(U)$.