How Can Lagrange Multipliers Determine Maximum Shannon Entropy?

Irishdoug
Messages
102
Reaction score
16
Homework Statement
Given a random variable X with d possible outcomes and distribution p(x),prove that the Shannon entropy is maximised for the uniform distribution where all outcomes are equally likely p(x) =1/d
Relevant Equations
## H(X) = - \sum_{x}^{} p(x)log_{2}p(x) ##

##log_{2}## is used as the course is a Quantum Information one.
I have used the Lagrange multiplier way of answering. So I have set up the equation with the constraint that ## \sum_{x}^{} p(x) = 1##

So I have:

##L(x,\lambda) = - \sum_{x}^{} p(x)log_{2}p(x) - \lambda(\sum_{x}^{} p(x) - 1) = 0##

I am now supposed to take the partial derivatives with respect to p(x) and ##\lambda##, however the derivatives with respect to ##\lambda## will give 0 I believe as we have to constants, 1 and -1.

So ##\frac{\partial (- \sum_{x}^{} p(x)log_{2}p(x) - \lambda(\sum_{x}^{} p(x) - 1)) }{\partial p(x)} = -(log_{2}p(x) + \frac{1}{ln_{2}}+\lambda) = 0##

I am unsure what to do with the summation signs, and I am also unsure how to proceed from here. Can I please have some help.
 
Physics news on Phys.org
The partials with respect to ##\lambda## should recover your constraint functions since the ##\lambda## dependent terms in your Lagrangian are only ##\lambda## times your constraint functions. Also consider using an index:

Sample space is ##\{ x_1, x_2, \cdots x_d\}## and ##p_k = p(x_k)##

L(p_k, \lambda) = -\sum_{k} p_k \log_2(p_k) - \lambda C(p_k)
with ##C## your constraint function ##C(p_k) = p_1+p_2+\ldots +p_d - 1## and normalized probabilities equate to ##C=0##.

\frac{\partial}{\partial p_k} L =\frac{1}{\ln(2)} -\log_2(p_k) -\lambda \doteq 0
\frac{\partial}{\partial \lambda} L = C(p_k) \doteq 0
(using ##\doteq## to indicate application of a constraint rather than an a priori identity.)
This is your ##d+1## equation on your ##d+1## free variables ##(p_1, p_2, \ldots ,p_d, \lambda)##.
 
Thread 'Need help understanding this figure on energy levels'
This figure is from "Introduction to Quantum Mechanics" by Griffiths (3rd edition). It is available to download. It is from page 142. I am hoping the usual people on this site will give me a hand understanding what is going on in the figure. After the equation (4.50) it says "It is customary to introduce the principal quantum number, ##n##, which simply orders the allowed energies, starting with 1 for the ground state. (see the figure)" I still don't understand the figure :( Here is...
Thread 'Understanding how to "tack on" the time wiggle factor'
The last problem I posted on QM made it into advanced homework help, that is why I am putting it here. I am sorry for any hassle imposed on the moderators by myself. Part (a) is quite easy. We get $$\sigma_1 = 2\lambda, \mathbf{v}_1 = \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} \sigma_2 = \lambda, \mathbf{v}_2 = \begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \\ 0 \end{pmatrix} \sigma_3 = -\lambda, \mathbf{v}_3 = \begin{pmatrix} 1/\sqrt{2} \\ -1/\sqrt{2} \\ 0 \end{pmatrix} $$ There are two ways...
Back
Top