Variational Calc boundary question

[nterrace]

When I try to find extrema of entropy for a pdf p(x), I use a lagrange mulitplier to impose the normalizaiton constraint, grind through the steps, and I get that p(x) is constant. This makes sense, since a flat distribution maximizes the entropy. But why does this method not also give me the function that minimizes the entropy? I assume this would be a delta function, yes? So why does that answer not fall out? I can only guess that the delta function is a sort of "boundary" of the variation space. When dealing with functions, the derivative might not be zero at an extrema that lies on a boundary... Is something analogous to that the reason the variational method I applied doesn't work to minimize entropy?

 

[jvicens]

Thank you for sharing your experience with finding extrema of entropy for a pdf p(x). Your question about why this method does not give you the function that minimizes entropy is a valid one.

First, let's clarify the difference between maximizing and minimizing entropy. When we are finding the extrema of entropy, we are looking for the values of p(x) that give us the highest or lowest possible value for entropy. So, when we use the Lagrange multiplier method, we are actually finding the maximum value of entropy, not the minimum.

Now, why does this method not give us the function that minimizes entropy? This is because the Lagrange multiplier method is designed to find the extrema of a function subject to certain constraints. In this case, the constraint is the normalization constraint. When we solve for the extrema using this method, we are essentially finding the values of p(x) that satisfy the constraint and give us the maximum value of entropy. This does not necessarily mean that it will also give us the minimum value of entropy.

To find the minimum value of entropy, we would need to use a different method. One possible approach is to use the method of steepest descent, where we iteratively adjust the values of p(x) to minimize the entropy. This method can give us the delta function as the solution for minimum entropy.

You are correct in your assumption that the delta function is a sort of "boundary" of the variation space. When we are dealing with functions, the derivative might not be zero at an extrema that lies on a boundary. This is because the delta function is a special case where the function is not differentiable. Therefore, the variational method you applied may not work to minimize entropy because it is designed for differentiable functions.

I hope this helps clarify your question. Keep up the good work in your research on entropy!