- #1

mathguy2009

- 3

- 0

I had a question about calculating mutual information, a topic to which I am very new. Consider the following hypothetical situation, in which I describe the entire process from what I understand:

Suppose I had a normal die (sides numbered 1-6) that you roll on a leaning table. I draw a line on the table so that the line divides the higher and lower halves of the table. Now, suppose I wanted to calculate the probability that I would roll a 5 on the upper half of the table. So, I begin by drawing a table like such:

-----1----2----3----4----5----6

High

Low

Then I perform 100 trials and record how many times a certain number was rolled on the higher or lower half of the table:

-----1----2----3----4----5----6

High_2____5___7___10___1___8

Low_11___6___8___12__20___10

To calculate the joint probability distribution P(roll, position), I would simply divide by the number of times the die was rolled (100 times) to produce the following table:

-------1------2------3------4------5-------6

High_0.02___0.05___0.07___0.10__0.01___0.08

Low_0.11___0.06___0.08___0.12__0.20___0.10

To calculate the marginal probabilities, I sum the rows and columns:

P(roll) = (sum(col 1), sum(col 2), ..., sum(col 6)) = (0.13, 0.11, 0.15, 0.22, 0.21, 0.18)

P(position) = (sum(row 1), sum(row 2)) = (0.33, 0.67)

Lastly, I calculate the mutual information of the data set (for an article on mutual information, see http://en.wikipedia.org/wiki/Mutual_information) using the following formula, using a base-2 logarithm to obtain an answer in units of bits:

MI(roll#; position) = [itex]\sum[/itex]

_{roll#}[itex]\sum[/itex]

_{position}P(roll#, position)log

_{2}[itex]\frac{P(roll, position)}{P(roll)P(position)}[/itex]

The number I get is 0.1206 bits...I suspect I did something wrong somewhere along the way, since this number is suspiciously small, but I cannot find my mistake. Any suggestions/corrections would be very much appreciated. Thanks in advance!