About the linear combination of multivariate normal distributions.

[mangdoo]

How can I prove that the any linear combination of multivariate normal
distribution is also normal?

I can prove it but I'm not sure that this is right or not. The point of my
proof is as follows.

---
The X and Y has the same dimensional random vector, and each random vector is
multivariate normal distribution. If I compute the linear combination of X
and Y,

Z = aX+bY

I can compute the components of random vector Z.

Z = [aX_1+bY_1; aX_2+bY_2; ... ;aX_n+bY_n]

In order to be a multivariate normal distribution, each component in random
vector should be a univariate normal distribution. In this point of view,
each component in random vector Z is a linear combination of univariate
normal distribution.(Because X and Y are multivariate normal distributions.)
Since the linear combination of univariate normal distribution is normal,
each component in random vector Z is also normal. Therefore Z, a linear
combination of multivariate normal distribution, is also normal.
----

My proof is right? If not, please tell me the wrong points. If right, then
how can I calculate the mean and covariance of the linear combination of
multivariate normal distribution.

Your comments are very helpful for me.
Thank you very much.

 

[statdad]

You can clean up your wording this way: For any $$X, Y$$ which are multivariate normal vectors we know that every linear combination of their components is univariate normal. Set $$Z = aX + bY$$, where $$a, b$$ are real: you want to show $$Z$$ is multivariate normal. Argue that since any linear combination of $$Z's$$ components is a linear combination of those of $$X, Y$$ the result holds.

OR

You could use the idea of multivariate moment-generating functions (use characteristic functions if you wish).

The rules for finding the mean vector and covariance matrix for $$aX + bY$$
can be found in textbooks. They are not much different than the rules for scalar random quantities.

 

[Stephen Tashi]

each component in random vector Z is also normal. Therefore Z, a linear
combination of multivariate normal distribution, is also normal.

The wikipedia article on the multivariate normal distribution says that it is possible to have a distribution where the X and Y components are normal and the distribution of (X,Y) is not bivariate normal. So what needs to be examined is your assertion that the normality of all the components proves the multivariate normality of the joint distribution.

 

[mangdoo]

Thank you very much, statdad and Stephen Tashi. :)

Okay. Then how about this idea?

First, in order to prove the linear combination of dependent univariate normal distribution, I write the two random variables in this way.

$$X = a_{11}u_{1} + a_{12}u_{2} + \mu_{x}$$
$$Y = a_{21}u_{1} + a_{22}u_{2} + \mu_{y}$$

$$U_{1}$$ and $$U_{2}$$ are independent standard normal distributions.

So, if I do the linear combination of $$X$$ and $$Y$$, then the equation is like this

$$X+Y = (a_{11} + a_{21})u_{1} + (a_{12}+a_{22})u_{2} + \mu_{x} + \mu_{y}$$

Using the characteristic function, I can easily prove the linear combination of dependent univariate normal distribution is also normal.


Now, I want to expand this into the multivariate normal vector.

$$X = AU + \mu_{x}$$
$$Y = BU + \mu_{y}$$

$$X$$, $$Y$$, $$\mu_{x}$$ and $$\mu_{y}$$ are d dimensional vector. $$A$$ and $$B$$ are the d by d matrix. $$U$$ is d dimensional vector which components are independent univariate standard normal.

So, if $$X$$ and $$Y$$ are combinated linearly, the equation is,

$$X+Y = (A+B)U + \mu_{x} + \mu_{y} = Z$$

Then I can show that the linear combination of the components in $$Z$$ is normal.

Is this right? I feel something missed.

 

[Stephen Tashi]

Thank you very much, statdad and Stephen Tashi. :)

Okay. Then how about this idea?

First, in order to prove the linear combination of dependent univariate normal distribution, I write the two random variables in this way.

I think you mean "independent univariate normal random variables".

$$X = a_{11}u_{1} + a_{12}u_{2} + \mu_{x}$$
$$Y = a_{21}u_{1} + a_{22}u_{2} + \mu_{y}$$

$$U_{1}$$ and $$U_{2}$$ are independent standard normal distributions.

I think you mean that $$U_{1}$$ and $$U_{2}$$ are independent random variables with standard normal distributions.

So, if I do the linear combination of $$X$$ and $$Y$$, then the equation is like this

$$X+Y = (a_{11} + a_{21})u_{1} + (a_{12}+a_{22})u_{2} + \mu_{x} + \mu_{y}$$

Using the characteristic function, I can easily prove the linear combination of dependent univariate normal distribution is also normal.

Again, you mean "independent univariate normal random variables".

Now, I want to expand this into the multivariate normal vector.

$$X = AU + \mu_{x}$$
$$Y = BU + \mu_{y}$$

Are you are using $$X$$ and $$Y$$ to mean something different now?

$$X$$, $$Y$$, $$\mu_{x}$$ and $$\mu_{y}$$ are d dimensional vector. $$A$$ and $$B$$ are the d by d matrix. $$U$$ is d dimensional vector which components are independent univariate standard normal.

Are you asserting that all multivariate distributions have the above form? If so, is that from a definition or from a theorem?

So, if $$X$$ and $$Y$$ are combinated linearly, the equation is,

$$X+Y = (A+B)U + \mu_{x} + \mu_{y} = Z$$

Then I can show that the linear combination of the components in $$Z$$ is normal.

Do you mean that any linear combination of the components of $$Z$$ is a univariate normal or do you mean that it is a multivariate normal?

 

[mangdoo]

I think you mean that $$U_{1}$$ and $$U_{2}$$ are independent random variables with standard normal distributions.

I'm sorry. The expression $$U_{1}$$ and $$U_{2}$$ is wrong. I want to represent the two dependent random variable $$X$$ and $$Y$$ using a standard normal distribution $$u_{1}$$ and $$u_{2}$$.

Are you are using $$X$$ and $$Y$$ to mean something different now?

Are you asserting that all multivariate distributions have the above form? If so, is that from a definition or from a theorem?

I'm sorry again. Now, the $$X$$ and $$Y$$ are multivariate normal distributions which dimension is d. So, I have to express each of them using multivariate standard normal distribution $$U$$ which dimension is also d. $$A$$ and $$B$$ are the matrix for covariance for $$X$$ and $$Y$$. Is this expression is wrong? If so, how can I represent the multivariate normal distribution using the multivariate standard distribution?

Do you mean that any linear combination of the components of $$Z$$ is a univariate normal or do you mean that it is a multivariate normal?

The problem is that the result of the linear combination of multivariate normal distribution, $$Z$$, can be also multivariate normal distribution or not. It does not mean the univariate normal distribution.

 

[Stephen Tashi]

Let me ask you this: Are you assuming that $$X$$ is an $$n$$ dimensional multivariate random variable if and only if $$X$$ can be written as:

$$X = \sum_{i=1}^{i=n} a_i u_i + \mu$$

where the $$a_i$$ are mutually orthogonal unit vectors, the $$u_i$$ are scalars that are independent normally distributed random variables with mean zero, and $$\mu$$ is a constant n-dimensional vector ?

To me, this is plausible. However, you need to cite some reason for it being true. In the articles that I have looked at, the above statement is not the definition of a multivariate random variable. So if you wish to use that statement as an equivalence to the definition, you need to cite some theorem.

 

[mangdoo]

$$X = \sum_{i=1}^{i=n} a_i u_i + \mu$$

The way to represent the multivariate normal distribution like that comes from the book named 'Mathematical statistics'(Peter J.Bickel, Kjell A. Doksum). At the chapter B,(It may be the appendix.) they represent the bivariate normal distribution like this.

$$X = a_{11}u_{1} + a_{12}u_{2} + \mu_{x}$$
$$Y = a_{21}u_{1} + a_{22}u_{2} + \mu_{y}$$

So, I just expand this form to the multivariate normal distribution. I would like to read some articles you said, because I'm not confident of my representation and proof.

In addition, if you prove this problem which means that the linear combination of multivariate normal distribution is also normal, how do you approach the proof?

Thanks for your splendid help.

 

[Stephen Tashi]

So, I just expand this form to the multivariate normal distribution. I would like to read some articles you said, because I'm not confident of my representation and proof.

I think you are correct that it generalizes to case of the multivariate normal distribution. I will look in some books for a theorem. The current Wikipedia article on the multivariate normal distribution http://en.wikipedia.org/wiki/Multivariate_normal_distribution has a section called "Drawing values from the distribution". The algorithm implies that a multivariate normal is equivalent to such a sum.

How would I prove it? I like the idea in your method, so I would pursue that. I don't claim to have written a proof of it myself.

 

[mangdoo]

How would I prove it? I like the idea in your method, so I would pursue that. I don't claim to have written a proof of it myself.

Thank you very much. :)

 

[SW VandeCarr]

In general a multivariate normal distribution should satisfy the following function for the normalized Mahalanobis squared distance function:

$$f(x)=\frac{1}{(2\pi)^{p/2}}|\sum|^{1/2} e^{-1/2(x-\mu)' \sum^{-1}(x-\mu)$$

where x and $$\mu$$ are vectors and $$|\sum|$$ is a positive definite matrix.

http://www.stat.lsu.edu/faculty/moser/exst7037/mvnprop.pdf

 

[Stephen Tashi]

I found this passage in "Statistical Pattern Recognition" by Funkanaga: p17

"Also, it is always possible to find a nonsingular linear transformation which makes the new covariance matrix diagonal. Since a diagonal covariance matrix means uncorrelated variables (independent variables for a normal distribution) , we can always find for a normal distribution a set of axes such that random variables are independent in the new coordinate system. These subjects will be discussed in detail in a later section."

He uses "normal distribution" to mean multivariate normal distribution. Unfortunately, I haven't found the "later section" where anything further is discussed.