Show extremum of the Entropy is a maximum

[dipole]

Homework Statement

The Entropy of a probability distribution is given by,

S = -k_B \sum _{i=1}^N p(i)\ln{p(i)}

I've shown that the extremum of such a function is given by,

S' = k_B \ln{N} (which is a positive quantity)

Now I want to show that this is a maximum by showing that

S' - S = k_B \ln{N} + \sum _{i=1}^N p(i)\ln{p(i)} > 0

Homework Equations

The p(i)'s are constrained by

\Sum_{i=1}^N p(i) =1

The Attempt at a Solution

I'm kind of stuck here. The second term is inherently negative, so it's not a priori obvious that S' - S > 0. I would probably want to take the ratio and show \frac{S'}{S} \geq 1 but I'm not sure how to do this.

Any ideas?

 

[Oxvillian]

I suggest writing a Taylor series for S in terms of the p_i and looking at the second-order term.

 

[Dick]

Homework Statement

The Entropy of a probability distribution is given by,

S = -k_B \sum _{i=1}^N p(i)\ln{p(i)}

I've shown that the extremum of such a function is given by,

S' = k_B \ln{N} (which is a positive quantity)

Now I want to show that this is a maximum by showing that

S' - S = k_B \ln{N} + \sum _{i=1}^N p(i)\ln{p(i)} > 0

Homework Equations

The p(i)'s are constrained by

\Sum_{i=1}^N p(i) =1

The Attempt at a Solution

I'm kind of stuck here. The second term is inherently negative, so it's not a priori obvious that S' - S > 0. I would probably want to take the ratio and show \frac{S'}{S} \geq 1 but I'm not sure how to do this.

Any ideas?

You only found one extremum which has a positive entropy (using Lagrange multipliers, I presume). The only other place you could have an extremum is on the boundary. The boundary of your set of p(i) would be the case where one of the p(i) is 1 and the rest are 0. What's the entropy there? You have to think about limits, since log(0) is undefined.

 

[Oxvillian]

The boundary of your set of p(i) would be the case where one of the p(i) is 1 and the rest are 0.

Hello Dick!

The boundary of the space in question is actually much bigger than this - it's the set of points for which at least one p(i) is zero. So doing it this way, some work remains.

 

[Dick]

Hello Dick!

The boundary of the space in question is actually much bigger than this - it's the set of points for which at least one p(i) is zero. So doing it this way, some work remains.

Good point. But if you fix one of your N p(i) to be zero. Then you are in the N-1 case with the remaining p(i). Suggests you use induction. In the case N=2, the boundary is the two points (p(1),p(2)) equal to (1,0) or (0,1).

 

[Oxvillian]

Good point. But if you fix one of your N p(i) to be zero. Then you are in the N-1 case with the remaining p(i). Suggests you use induction. In the case N=2, the boundary is the two points (p(1),p(2)) equal to (1,0) or (0,1).

Correct

But (at the risk of giving away the plot), I think the easiest way to demonstrate the global nature of the max is to notice that you can always increase the entropy by transferring probability from somewhere with a large p(i) to somewhere with a small p(i).

 

[Dick]

Correct

But (at the risk of giving away the plot), I think the easiest way to demonstrate the global nature of the max is to notice that you can always increase the entropy by transferring probability from somewhere with a large p(i) to somewhere with a small p(i).

That is a simpler way to look at it.