What Is the Total Probability of Rolling a 6 with Maximum Shannon Entropy?

[ZetaOfThree]

Homework Statement

A 6 sided dice is loaded such that 6 occurs twice as often as 1. What is the total probability of rolling a 6 if the Shannon entropy is a maximum?

Homework Equations

Shannon Entropy:
$$S=-\sum_i p_i \ln{p_i}$$
where $p_i$ is the probability that we roll $i$.

The Attempt at a Solution

We know that $\sum_i p_i = 1$ and we are given that $p_1=p_6/2$. So $$p_2+p_3+p_4+p_5+\frac{3}{2} p_6 =1 \Rightarrow p_5 = 1-p_2-p_3-p_4-\frac{3}{2} p_6$$ There we can write the Shannon entropy as $$S=-\left( \frac{p_6}{2} \ln{\frac{p_6}{2}} + p_2 \ln{p_2}+p_3 \ln{p_3}+p_4 \ln{p_4}+(1-p_2-p_3-p_4-\frac{3}{2} p_6) \ln(1-p_2-p_3-p_4-\frac{3}{2} p_6) + p_6 \ln{p_6}\right)$$
$$=-\left( \frac{3 p_6}{2} \ln{p_6} + p_2 \ln{p_2}+p_3 \ln{p_3}+p_4 \ln{p_4}+(1-p_2-p_3-p_4-\frac{3}{2} p_6) \ln(1-p_2-p_3-p_4-\frac{3}{2} p_6) - \frac{p_6}{2} \ln{2}\right)$$
To find an extremum, we partial differentiate and set it equal to zero:
$$\frac{\partial S}{\partial p_2} = - \left(\ln{p_2} +1 - \ln(1-p_2-p_3-p_4-\frac{3}{2} p_6) -1 \right) = -\left(\ln{p_2} - \ln(1-p_2-p_3-p_4-\frac{3}{2} p_6) \right) =0$$
$$\Rightarrow \ln p_2 = \ln(1-p_2-p_3-p_4-\frac{3}{2} p_6)$$
Differentiating with respect to $p_3$ and $p_4$, we find the same condition, so we conclude that $p_2=p_3=p_4=p_5$. We now write the Shannon entropy as $$S=- \left(\frac{3 p_6}{2} \ln{p_6}+4p_2 \ln{p_2} - \frac{p_6}{2} \ln{2}\right)$$ So $$\frac{\partial S}{\partial p_6}= -\left(\frac{3}{2} \ln{p_6}+\frac{3}{2} - \frac{1}{2} \ln{2}\right) = 0$$
Therefore we find that $p_6 = \frac{2^{1/3}}{e}$. But this is wrong, because if we plug in the probabilities into the Shannon entropy formula, we do not get a maximum. For example, we get a higher Shannon entropy if we plug in $p_1=p_2=p_3=p_4=p_5=1/7$ and $p_6=2/7$. Where did I go wrong? Maybe I found a minimum or something? If so, how do I get the maximum?
Thanks for any help.

 

[BvU]

It looks as if you put $\displaystyle {d p_2\over d p_6}=0$. But isn't there a constraint that $4p_2+{3\over 2}p_6 = 1 \ \Rightarrow \ 4d p_2 + {3\over 2}dp_6 = 0$ ?

 

[ZetaOfThree]

It looks as if you put $\displaystyle {d p_2\over d p_6}=0$.

What step are you referring to? I don't think I used this.

But isn't there a constraint that $4p_2+{3\over 2}p_6 = 1 \ \Rightarrow \ 4d p_2 + {3\over 2}dp_6 = 0$ ?

Yes, you can use that condition to find $p_2$ after $p_6$ is found. So in the case above, we have $p_1 = \frac{1}{2^{2/3}e}$, $p_2=p_3=p_4=p_5= \frac{1}{8} \left(2- \frac{3 \sqrt [3] {2}}{e} \right)$ and $p_6 = \frac{\sqrt[3]{2}}{e}$.

 

[mfb]

The partial derivative gives you an optimum if p₆ is independent of the other probabilities. It is not. You should get the correct result if you express p₂ as function of p₆ and then calculate the partial derivative.

 

[Dick]

What step are you referring to? I don't think I used this.

Yes, you can use that condition to find $p_2$ after $p_6$ is found. So in the case above, we have $p_1 = \frac{1}{2^{2/3}e}$, $p_2=p_3=p_4=p_5= \frac{1}{8} \left(2- \frac{3 \sqrt [3] {2}}{e} \right)$ and $p_6 = \frac{\sqrt[3]{2}}{e}$.

Given that you know $p_2=p_3=p_4=p_5$, and given the two other constraints you could write the entropy as a function of a single variable like $p_6$ and maximize that as a single variable problem. That should be straightforward.

 

[ZetaOfThree]

The partial derivative gives you an optimum if p₆ is independent of the other probabilities. It is not. You should get the correct result if you express p₂ as function of p₆ and then calculate the partial derivative.

Given that you know $p_2=p_3=p_4=p_5$, and given the two other constraints you could write the entropy as a function of a single variable like $p_6$ and maximize that as a single variable problem. That should be straightforward.

Sounds about right. I did that, and got an answer I'm pretty sure is correct. Thank you all for the help!