How Can Lagrange Multipliers Determine Maximum Shannon Entropy?

[Irishdoug]

Homework Statement

Given a random variable X with d possible outcomes and distribution p(x),prove that the Shannon entropy is maximised for the uniform distribution where all outcomes are equally likely p(x) =1/d

Relevant Equations

$H(X) = - \sum_{x}^{} p(x)log_{2}p(x)$

$log_{2}$ is used as the course is a Quantum Information one.

I have used the Lagrange multiplier way of answering. So I have set up the equation with the constraint that $\sum_{x}^{} p(x) = 1$

So I have:

$L(x,\lambda) = - \sum_{x}^{} p(x)log_{2}p(x) - \lambda(\sum_{x}^{} p(x) - 1) = 0$

I am now supposed to take the partial derivatives with respect to p(x) and $\lambda$, however the derivatives with respect to $\lambda$ will give 0 I believe as we have to constants, 1 and -1.

So $\frac{\partial (- \sum_{x}^{} p(x)log_{2}p(x) - \lambda(\sum_{x}^{} p(x) - 1)) }{\partial p(x)} = -(log_{2}p(x) + \frac{1}{ln_{2}}+\lambda) = 0$

I am unsure what to do with the summation signs, and I am also unsure how to proceed from here. Can I please have some help.

 

[jambaugh]

The partials with respect to $\lambda$ should recover your constraint functions since the $\lambda$ dependent terms in your Lagrangian are only $\lambda$ times your constraint functions. Also consider using an index:

Sample space is $\{ x_1, x_2, \cdots x_d\}$ and $p_k = p(x_k)$

$$L(p_k, \lambda) = -\sum_{k} p_k \log_2(p_k) - \lambda C(p_k)$$
with $C$ your constraint function $C(p_k) = p_1+p_2+\ldots +p_d - 1$ and normalized probabilities equate to $C=0$.

$$\frac{\partial}{\partial p_k} L =\frac{1}{\ln(2)} -\log_2(p_k) -\lambda \doteq 0$$
$$\frac{\partial}{\partial \lambda} L = C(p_k) \doteq 0$$
(using $\doteq$ to indicate application of a constraint rather than an a priori identity.)
This is your $d+1$ equation on your $d+1$ free variables $(p_1, p_2, \ldots ,p_d, \lambda)$.