Suppose I have a Gaussian probability distribution:(adsbygoogle = window.adsbygoogle || []).push({});

[itex]N_{A}(0,1).[/itex]

A set of values are generated from this distribution to which an arbitrary amount of Gaussian noise, say [itex]N_{B}(0,0.5)[/itex], is added and then the [itex]N_{B}[/itex] values sorted from lowest to highest. These are then digitised by assigning 0 to a value below 0 and then 1 to a value above 0. We end up with values such as:

-3.11359 | 0 | -6.91717 | 0 |

-3.31609 | 0 | -6.70810 | 0 |

-3.67497 | 0 | -6.13978 | 0 |

-2.73024 | 0 | -6.12173 | 0 |

-4.20178 | 0 | -6.07266 | 0 |

-3.38846 | 0 | -5.88277 | 0 |

To calculate the mutual information in bits between this set of data we use the following formula:

[itex]I(A;B) = \sum_{x \in A} \sum_{y \in B}P(x,y)log_{2}(\frac{P(x,y)}{P(x)P(y)}) [/itex]

where [itex]P(x,y)[/itex] is the probability of A = x and B = y. Now in this case P(0,1) = P(1,0) = P(1,1) = 0 so we are left with

[itex]I(A;B) = P(0,0)log_{2}(\frac{P(0,0)}{P(x=0)P(y=0)})[/itex]

Now here is what I don't get. For this Gaussian distribution P(x=0) ≈ 0.5 and P(y=0) ≈ 0.5 and from this set of data P(0,0) = 1. This gives,

[itex]I(A;B) = log_{2}(\frac{1}{0.25}) = 2 [/itex]

From what I understand it is should not possible for the mutual information to be greater than 1. Where am I going wrong with this? I feel as though I am making a very basic error.

Thanks.

**Physics Forums - The Fusion of Science and Community**

The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

# Mutual Information between two Gaussian distributions

Loading...

Similar Threads - Mutual Information between | Date |
---|---|

I Random Variables Mutually independent | Apr 8, 2017 |

I Mutual information between two time series. | May 1, 2016 |

If X,Y,Z mutually independent, is X*Y and X*Z independent? | Apr 21, 2012 |

Mutual information. concave/convex | Nov 20, 2011 |

Calculating Mutual Information | Jun 27, 2011 |

**Physics Forums - The Fusion of Science and Community**