Notes on Random Process and its Applications 
to linear systems 

EE 315 – Winter 2005

Dr. Adil S. Balghonaim 
Random Process

The concept of random variable was defined previously as mapping from the Sample Space S to the real line as shown below 

                          

The concept of  random process can be extended to include time and the outcome will be random functions of time as shown below 

 

The functions  are one realizations of many of the random process  X(t) .

A random process also represents a random variable when time is fixed
              is a random variable

Stationary and Independence

The random process X(t) can be classified as follows:

First-order stationary
A random process is classified as first-order stationary if its first-order probability density function remains equal regardless of any shift in time to its time origin. If we let represent a given value at time , then we define a first-order stationary as one that satisfies the following equation: 

The physical significance of this equation is that our density function,   is completely independent of  and thus any time shift   . The most important result of this statement, and the identifying characteristic of any first-order stationary process, is the fact that the mean is a constant, independent of any time shift. 

Second-order stationary
A random process is classified as second-order stationary if its second-order probability density function does not vary over any time shift applied to both values. In other words, for values  and  then we will have the following be equal for an arbitrary time shift . 

From this equation we see that the absolute time does not affect our functions, rather it only really depends on the time difference between the two variables. 

For a second-order stationary process, we need to look at the autocorrelation function ( will be presented later) to see its most important property. 

Since we have already stated that a second-order stationary process depends only on the time difference, then all of these types of processes have the following property: 

Wide-Sense Stationary (WSS)
A process that satisfies the following:

							

is a Wide-Sense Stationary (WSS)

Second-order stationary  Wide-Sense Stationary

The converse is not true in general

Time Average and Ergodicity
  
    The time average of a quantity is defined as

Here A   is used to denote time average in a manner analogous to  
E   for the statistical average.

The time average is taken over all time because, as applied to random processes, sample functions of processes are presumed to exist for all time.

Let  x(t) be a sample of the random process X(t) were the lower case letter imply a sample function (not random function).

We define the mean value    ( a lowercase letter is used to imply a sample function)  and the time autocorrelation function  as follows:

						

For any one sample function  ( i.e.,  x(t) )  of the random process X(t), the last two integrals simply produce  a number  for the average   and a number  for   for a specific value of .

Since the sample function x(t) is one out of  other samples functions of the random process X(t), the average and the autocorrelation are actually random variables.

By taking the expected value  for   and  , we obtain

we now can get the expectation inside the integral assuming that the process is stationary, which   mean that the mean and autocorrelation are not function of time

we obtain

                           

similarly

 

Now suppose that by some theorem (will be stated later) the random variables  and  have zero variance; that is, 

                     and   are constants

Then 

  

The above indicate that the time average    and   equal the statistical averages  and .

The theorem that allows such thing ( The time average equal the statistical average) is called  “ergodic theorem”

Process that satisfy the ergodic theorem are called ergodic processes

Ergodicity is a very restrictive form of stationarity and it may be difficult to prove, however we assume often a process is ergodic to simplify problems.

In the real world, we are usually forced to work with only one sample functions of a process and therefore must like or not, derive mean value, correlation functions, etc., from the time waveform.

By assuming ergodicity, we may infer the similar statistical characteristic of the process.

Correlation  Function

Autocorrelation Function and Its Properties

The autocorrelation function of a random process X(t) is the correlation  of two random variables   and  defined by the process at times  t1  and  t2  . Mathematically 

Assuming a  second-order stationary process

	

Properties of 		
					
(1)		autocorrelation function is bounded by   	                                    its value at the origin
(2)		autocorrelation function is an even                         	                                    function
(3)		The average power in the process		
(4)	If  and X(t) is ergodic with no periodic components then 

Covariance Function
 The Covariance of two random variables  X and Y was defined as

We extend  the concept of of covariance to random process
	
which can be put in the form	
				
which is similar to the two random variables case

For WSS stationary process X(t) , we get	

    The variance  of X(t)

Linear System with Random Inputs

In application of random process, the Input-Output relation through a linear system can be described as follows:

Here  X(t)  is a random process and h(t) (Deterministic Function) is the impulse response of the linear system ( Filter or any other Linear System )

Now we can look at input output relation as follows:

The Time Domain
The output  in the time domain is the convolution of the Input random process X(t) and the impulse response h(t),

Question:  Can you evaluate this convolution integral ?

Answer:
We can observe that we can not evaluate this convolution integral in general because X(t) is random and there is no mathematical expression for X(t).

The Frequency Domain
The output  in the Frequency Domain is the Product of the Input Fourier Transform of the input  random process X(t) , and the  Fourier Transform of  the impulse response h(t), H(f)

where
     the  Fourier Transform of the input  random 
                                      process  X(t)  is a random process 

       the  Fourier Transform of the deterministic 
                                     impulse response

         the  Fourier Transform of the output  random 
                                      process  Y(t)  is a random process 

Question : Can you evaluate   the  Fourier Transform of the input  random process  X(t)  ?

Answer: In general no , since the function X(t) in general is random and has no mathematical expression.

Question: How can we describe then the behavior of the input random process and the output random process through a linear time-invariant system ?

Answer: Let us look at the autocorrelation function of  the random process X(t) ,  , assuming  a WSS process ( constant mean and the autocorrelation function is deterministic and only a function of  ).

We defined previously the autocorrelation    as 

The auto correlation tell us how the random process is varying.

Is it a slow varying process or a high varying process.

Now we will define  another function that will help us on looking at the behavior of the random process as follows:

Let 
        (or )   be the Fourier Transform of  

         ( or  )						
         ( or )	

Since
        					
 =      Average Power
Then    is Power Spectral Density ( PSD )  of the Random Process X(t) 

Properties of  PSD

(1)										
(2)						
(3)	  is real since     is even function	
   
(4)		 =     Total power	
(5)			DC  power ( power at zero Frequency)		
Now let us look at the input-output linear system shown below in the time domain and Frequency domain assuming  the Random Process X(t) is WSS

Time Domain

The Mean of the output,

The Autocorrelation of the output,

which reduces to

This show that the mean is constant and the autocorrelation of the output  is a function of    

  Y(t) is  WSS

Frequency Domain

where

 is the Fourier Transform of  

 is the Fourier Transform of  

Since

then

                      
Total power of the output