<jabberwocky><div class="vbmenu_control"><a href="jabberwocky:;" onClick="newWindow=window.open('','usenetCode','toolbar=no, location=no,scrollbars=yes,resizable=yes,status=no ,width=650,height=400'); newWindow.document.write('<HTML><HEAD><TITLE>Usenet ASCII</TITLE></HEAD><BODY topmargin=0 leftmargin=0 BGCOLOR=#F1F1F1><table border=0 width=625><td bgcolor=midnightblue><font color=#F1F1F1>This Usenet message\'s original ASCII form: </font></td></tr><tr><td width=449><br><br><font face=courier><UL><PRE>"Arnold Neumaier" <Arnold.Neumaier@univie.ac.at> wrote ...\n> r.e.s. wrote:\n> > "Arnold Neumaier" <Arnold.Neumaier@univie.ac.at> wrote ...\n> >>r.e.s. wrote:\n> >>>>"Arnold Neumaier" <Arnold.Neumaier@univie.ac.at> wrote ...\n> >>>>\n> >>>>>The natural least informative distribution on natural numbers\n> >>>>>is a Poisson distribution, but here one has to be at least informed\n> >>>>>about the mean.\n> >\n> >>>The maximum-entropy distribution on the natural numbers, constrained\n> >>>only to have a given mean, is geometric, not Poisson. (It\'s a shifted\n> >>>version of the usual geometric distribution if 0 is included in the\n> >>>naturals. This is just the Boltzmann distribution for energy-levels\n> >>>that are the natural numbers scaled by an appropriate constant.)\n> >>\n> >>The maximum entropy solution\n\n> S(rho)= <-log(rho(x))> = max!\n\n> >>for a distribution with density rho(x) depends on whether\n> >>we define densities rho of a random natural number x by\n> >> <f(x)> = sum_n rho(n) f(n) [A]\n> >>or\n> >> <f(x)> = sum_n rho(n) f(n)/n! [B]\n> >>corresponding to different choices of priors.\n<snip>\n\nWithout the measure-theoretic framework, it was difficult to see\nwhat was meant, so thanks for the detailed explanation below. Yes,\nof course constrained maximization of\n- integral dmu(x) rho(x) log(rho(x)) [1]\nleads to different densities rho with respect to the two different\nmeasures mu, which in the present problem can be characterized by\ntheir behavior on the singleton sets {n} (n in {0,1,2,...}):\n\n(A) mu_A({n}) = 1\n(B) mu_B({n}) = 1/n!\n\nJaynes\' "finite sets policy" (see below) might help to see why\n(A) is the correct choice here.\n\n\n> > However, the original point remains independent of priors ...\n> > If a Poisson distribution and a geometric distribution on the same\n> > set have the same mean, then the geometric distribution necessarily\n> > has greater Shannon entropy than does the Poisson distribution.\n>\n> Yes.\n>\n> > A reasonable interpretation of this is that the geometric distribution\n> > represents a state of knowledge that incorporates less information\n> > than does a state corresponding to the Poisson, *regardless* of how\n> > those states come about.\n>\n> You only proved that it incorporates less \'Shannon entropy\'.\n> But the identification of \'information\' and \'Shannon entropy\' is\n> dubious for situations with infinitely many alternatives.\n> Shannon assumes in his analysis that in the absence of knowledge,\n> all alternatives are equally likely, which makes no sense\n> in the infinite case (and may even be debated in the finite case).\n\nIt\'s wise to distrust pat answers in the infinite case, but in the\n**finite** case, the question has been effectively settled, IMO.\nIn the simplest version, if one must assign "state of knowledge"\nprobabilities concerning just two alternatives, any nonuniform\nassignment clearly represents information that favors one of the\ntwo alternatives. The same reasoning extends to other finite sets,\nor you could appeal to an invariance argument such as you mention\nbelow.\n\nFor distributions on *infinite* spaces, it\'s useful to consider\nJaynes\' "finite sets policy". His last book was published\nposthumously, and the following is from a pre-publication version:\n\n"Our conclusion -- based on some forty years of mathematical\nefforts and experience with real problems -- is that, at least\nin probability theory, an infinite set should be thought of\nonly as the limit of a specific (i.e. unambiguously specified)\nsequence of finite sets. Likewise, an improper pdf has meaning\nonly as the limit of a well-defined sequence of proper pdf\'s.\n... Indeed, experience to date shows that almost any attempt\nto depart from our recommended `finite sets\' policy has the\npotentiality for generating a paradox, in which two equally\nvalid methods of reasoning lead us to contradictory results."\n\n(He devoted an entire chapter to illustrating how errors easily\narise when this policy is not followed.)\n\nIf we apply a finite-sets policy to the present problem, together\nwith your view of mu as a prior, then we should choose which of\nthe two priors, mu_A or mu_B, represents the least information\nwhen the basic set is finite, say {0,...,n-1}, and no other\ninformation is given (since these are *prior* to any such). Both\nmeasures are then normalizable, mu_A being a uniform probability\ndistribution and mu_B being a particular Poisson probability\ndistribution, both restricted to the finite space.\n\nBut in each of the finite cases, any nonuniform distribution cannot\npossibly be the least informative one; rather, a uniform\ndistribution is seen to be inescapable for each of the finite sets\n{0,...,n-1}. Consequently, the constrained maximizations [1] on\nthese finite sets produce a sequence of geometric distributions\nconverging to the geometric distribution on the infinite set.\n\nI admit, though, that in the statistical mechanics of many-particle\nsystems, there might be physical justifications for a Poisson prior\nto represent some kind of information before the mean value is\nspecified -- e.g. to get the \'correct Boltzmann counting\' that you\nmention. But there is no such physical context in the present\nproblem. (I had cited the Boltzmann distribution only to give a\nfamiliar example that formally can reduce to a geometric.)\n\n\n> Here is a more careful setting that should explain our differences:\n>\n> For a probability distribution on a finite set of alternatives,\n> given by probabilities p_n summing to 1, the Shannon entropy is\n> defined by\n> S = - sum p_n log_2 p_n.\n> The main use of the entropy concept is the maximum entropy principle,\n> used to define various interesting ensembles by maximizing the entropy\n> subject to constraints defined by known expectation values\n> <f> = sum P_n f(n)\n> for certain key observables f.\n>\n> If the number of alternatives is infinite, this formula must be\n> appropriately generalized. In the literature, one finds various\n> possibilities, the most common being, for random vectors with\n> probability density p(x), the absolute entropy\n> S = - k_B integral dx p(x) log p(x)\n> with the Boltzmann constant k_B and Lebesgue measure dx.\n> The value of the Boltzmann constant k_B is conventional and has no\n> effect on the use of entropy in applications.\n> There is also the relative entropy\n> S = - k_B integral dx p(x) log (p(x)/p_0(x)),\n> which involves an arbitrary positive function p_0(x). If p_0(x)\n> is a probability density then the relative entropy is nonnegative.\n^^^^^^^^^^^\nNo, it won\'t be nonnegative. Relative entropy by your definition\nis non*positive*. Suppose P, Q are probability measures with\nP << m, Q << m for positive measure m. Then the Radon-Nikodym\nderivatives satisfy\n\nintegral dm (dP/dm) log( (dP/dm)/(dQ/dm) ) >= 0\n\nwhich provides the more usual definition of relative entropy.\nOf course, maximizing your nonpositive quantity leads to the\nsame results as minimizing the nonnegative one.\n\n\n> For a probability distribution over an _arbitrary_ sigma algebra,\n> the absolute entropy makes no sense since there is no distinguished\n> measure and hence no probability density. Thus one needs to assume a\n> measure to be able to define a probability density (namely as the\n> Radon-Nikodym derivative, assuming it exists). This measure is\n> called the prior (it is often improper = not normalizable).\n> Once one has specified a prior dmu,\n> <f(x)> = integral dmu(x) rho(x) f(x)\n> defines the density rho(x), and then\n> S(rho)= <-k_B log(rho(x))>\n> defines the entropy with respect to this prior. Note that the\n> condition for rho to define a probability density is\n> integral dmu(x) rho(x) = <1> = 1.\n>\n> In many cases, symmetry considerations suggest a unique natural prior.\n> For random variables on a homogeneous space, the conventional measure\n> is the invariant Haar measure. In particular, for probability theory\n> of finitely many alternatives, it is conventional to consider the\n> symmetric group on the set of alternatives and take as the prior the\n> uniform measure, giving\n> <f(x)> = sum_x rho(x) f(x).\n> The density rho(x) agrees with the probability p_x, and the\n> corresponding entropy is the Shannon entropy is one takes k_B=1/log2.\n\nOf course the above argument acquires special significance\nif a comprehensive finite-sets policy can be implemented.\n\n\n> For random variables whose support is R or R^n, the conventional\n> symmetry group is the translation group, and the corresponding\n> (improper) prior is the Lebesgue measure. In this case one obtains\n> the absolute entropy given above. But one could also take as prior\n> a noninvariant measure\n> dmu(x) = dx p_0(x);\n> then the density becomes rho(x)=p(x)/p_0(x), and one arrives at the\n> relative entropy.\n\nOK, but as mentioned above, this \'relative entropy\' is nonpositive.\n\n\n> If there is no natural transitive symmetry group, there is no natural\n> prior, and one has to make other useful choices. In particular, this\n> is the case for random natural numbers.\n\nIf we adopt a finite-sets policy, together with your view of the\nmeasure mu as a prior, the above is shown to be a non sequitur.\nThe reason you don\'t see the natural prior is that you\'re not\nexamining the priors in terms of their representative sequences\non *finite* spaces.\n\n\n> Choice A. Treating the natural numbers as a limiting situation of\n> finite interval [0:n] suggests to use the measure with\n> integral dmu(x) phi(x) = sum_n phi(n)\n> as (improper) prior, making\n> <f(x)> = sum_n rho(n) f(n)\n> the definition of the density; in this case, p_n=rho(n) is the\n> probability of getting n.\n>\n> Choice B. Statistical mechanics suggests to use instead the measure\n> with\n> integral dmu(x) phi(x) = sum_n phi(n)/n!\n> as prior,\n\nSo mu_B({n}) = 1/n! (n in {0,1,2,...}),\nwhich normalizes to a Poisson distribution with mean 1.\n\n\n> making\n> <f(x)> = sum_n rho(n) f(n)/n!\n> the definition of the density; in this case, p_n=rho(n)/n! is the\n> probability of getting n.\n>\n> The maximum entropy ensemble defined by given expectations depends on\n> the prior chosen. In particular, if the mean of a random natural number\n> is given, choice A leads to a geometric distribution, while\n> choice B leads to a Poisson distribution. The latter is the one\n> relevant for statistical mechanics. Indeed, choice B is the prior\n> needed in statistical mechanics of systems with an indefinite\n> number n of particles to get the correct Boltzmann counting in the\n> grand canonical ensemble. With choice A, the maximum entropy\n> solution is unrelated to the distributions arising in statistical\n> mechanics.\n\nThe present problem is not about the statistical mechanics of a\nmany-particle system, so it seems hard to justify forcing the\nprior to be Poisson by reference to \'correct Boltzmann counting\'.\n\n--r.e.s.\n\n</UL></PRE></font></td></tr></table></BODY><HTML>');"> <IMG SRC=/images/buttons/ip.gif BORDER=0 ALIGN=CENTER ALT="View this Usenet post in original ASCII form"> View this Usenet post in original ASCII form </a></div><P></jabberwocky>"Arnold Neumaier" <Arnold.Neumaier@univie.ac.at> wrote ...
> r.e.s. wrote:
> > "Arnold Neumaier" <Arnold.Neumaier@univie.ac.at> wrote ...
> >>r.e.s. wrote:
> >>>>"Arnold Neumaier" <Arnold.Neumaier@univie.ac.at> wrote ...
> >>>>
> >>>>>The natural least informative distribution on natural numbers
> >>>>>is a Poisson distribution, but here one has to be at least informed
> >>>>>about the mean.
> >
> >>>The maximum-entropy distribution on the natural numbers, constrained
> >>>only to have a given mean, is geometric, not Poisson. (It's a shifted
> >>>version of the usual geometric distribution if is included in the
> >>>naturals. This is just the Boltzmann distribution for energy-levels
> >>>that are the natural numbers scaled by an appropriate constant.)
> >>
> >>The maximum entropy solution
> S(\rho)= <-log(\rho(x))> = max!
> >>for a distribution with density \rho(x) depends on whether
> >>we define densities \rho of a random natural number x by
> >> <f(x)> = sum_n \rho(n) f(n) [A]
> >>or
> >> <f(x)> = sum_n \rho(n) f(n)/n! [B]
> >>corresponding to different choices of priors.
<snip>
Without the measure-theoretic framework, it was difficult to see
what was meant, so thanks for the detailed explanation below. Yes,
of course constrained maximization of
- integral dmu(x) \rho(x) log(\rho(x)) [1]
leads to different densities \rho with respect to the two different
measures \mu, which in the present problem can be characterized by
their behavior on the singleton sets {n} (n in {0,1,2,...}):
(A) \mu_A({n}) = 1(B) \mu_B({n}) = 1/n!
Jaynes' "finite sets policy" (see below) might help to see why
(A) is the correct choice here.
> > However, the original point remains independent of priors ...
> > If a Poisson distribution and a geometric distribution on the same
> > set have the same mean, then the geometric distribution necessarily
> > has greater Shannon entropy than does the Poisson distribution.
>
> Yes.
>
> > A reasonable interpretation of this is that the geometric distribution
> > represents a state of knowledge that incorporates less information
> > than does a state corresponding to the Poisson, *regardless* of how
> > those states come about.
>
> You only proved that it incorporates less 'Shannon entropy'.
> But the identification of 'information' and 'Shannon entropy' is
> dubious for situations with infinitely many alternatives.
> Shannon assumes in his analysis that in the absence of knowledge,
> all alternatives are equally likely, which makes no sense
> in the infinite case (and may even be debated in the finite case).
It's wise to distrust pat answers in the infinite case, but in the
**finite** case, the question has been effectively settled, IMO.
In the simplest version, if one must assign "state of knowledge"
probabilities concerning just two alternatives, any nonuniform
assignment clearly represents information that favors one of the
two alternatives. The same reasoning extends to other finite sets,
or you could appeal to an invariance argument such as you mention
below.
For distributions on *infinite* spaces, it's useful to consider
Jaynes' "finite sets policy". His last book was published
posthumously, and the following is from a pre-publication version:
"Our conclusion -- based on some forty years of mathematical
efforts and experience with real problems -- is that, at least
in probability theory, an infinite set should be thought of
only as the limit of a specific (i.e. unambiguously specified)
sequence of finite sets. Likewise, an improper pdf has meaning
only as the limit of a well-defined sequence of proper pdf's.
... Indeed, experience to date shows that almost any attempt
to depart from our recommended `finite sets' policy has the
potentiality for generating a paradox, in which two equally
valid methods of reasoning lead us to contradictory results."
(He devoted an entire chapter to illustrating how errors easily
arise when this policy is not followed.)
If we apply a finite-sets policy to the present problem, together
with your view of \mu as a prior, then we should choose which of
the two priors, \mu_A or \mu_B, represents the least information
when the basic set is finite, say {0,...,n-1}, and no other
information is given (since these are *prior* to any such). Both
measures are then normalizable, \mu_A being a uniform probability
distribution and \mu_B being a particular Poisson probability
distribution, both restricted to the finite space.
But in each of the finite cases, any nonuniform distribution cannot
possibly be the least informative one; rather, a uniform
distribution is seen to be inescapable for each of the finite sets
{0,...,n-1}. Consequently, the constrained maximizations [1] on
these finite sets produce a sequence of geometric distributions
converging to the geometric distribution on the infinite set.
I admit, though, that in the statistical mechanics of many-particle
systems, there might be physical justifications for a Poisson prior
to represent some kind of information before the mean value is
specified -- e.g. to get the 'correct Boltzmann counting' that you
mention. But there is no such physical context in the present
problem. (I had cited the Boltzmann distribution only to give a
familiar example that formally can reduce to a geometric.)
> Here is a more careful setting that should explain our differences:
>
> For a probability distribution on a finite set of alternatives,
> given by probabilities p_n summing to 1, the Shannon entropy is
> defined by
> S = - sum p_n log_2 p_n.
> The main use of the entropy concept is the maximum entropy principle,
> used to define various interesting ensembles by maximizing the entropy
> subject to constraints defined by known expectation values
> <f> = sum P_n f(n)
> for certain key observables f.
>
> If the number of alternatives is infinite, this formula must be
> appropriately generalized. In the literature, one finds various
> possibilities, the most common being, for random vectors with
> probability density p(x), the absolute entropy
> S = - k_B integral dx p(x) log p(x)
> with the Boltzmann constant k_B and Lebesgue measure dx.
> The value of the Boltzmann constant k_B is conventional and has no
> effect on the use of entropy in applications.
> There is also the relative entropy
> S = - k_B integral dx p(x) log (p(x)/p_0(x)),
> which involves an arbitrary positive function p_0(x). If p_0(x)
> is a probability density then the relative entropy is nonnegative.
^^^^^^^^^^^
No, it won't be nonnegative. Relative entropy by your definition
is non*positive*. Suppose P, Q are probability measures with
P << m, Q << m for positive measure m. Then the Radon-Nikodym
derivatives satisfy
integral dm (dP/dm) log( (dP/dm)/(dQ/dm) ) >=
which provides the more usual definition of relative entropy.
Of course, maximizing your nonpositive quantity leads to the
same results as minimizing the nonnegative one.
> For a probability distribution over an _arbitrary_ \sigma algebra,
> the absolute entropy makes no sense since there is no distinguished
> measure and hence no probability density. Thus one needs to assume a
> measure to be able to define a probability density (namely as the
> Radon-Nikodym derivative, assuming it exists). This measure is
> called the prior (it is often improper = not normalizable).
> Once one has specified a prior dmu,
> <f(x)> = integral dmu(x) \rho(x) f(x)
> defines the density \rho(x), and then
> S(\rho)= <-k_B log(\rho(x))>
> defines the entropy with respect to this prior. Note that the
> condition for \rho to define a probability density is
> integral dmu(x) \rho(x) = <1> = 1.
>
> In many cases, symmetry considerations suggest a unique natural prior.
> For random variables on a homogeneous space, the conventional measure
> is the invariant Haar measure. In particular, for probability theory
> of finitely many alternatives, it is conventional to consider the
> symmetric group on the set of alternatives and take as the prior the
> uniform measure, giving
> <f(x)> = sum_x \rho(x) f(x).
> The density \rho(x) agrees with the probability p_x, and the
> corresponding entropy is the Shannon entropy is one takes k_B=1/log2.
Of course the above argument acquires special significance
if a comprehensive finite-sets policy can be implemented.
> For random variables whose support is R or R^n, the conventional
> symmetry group is the translation group, and the corresponding
> (improper) prior is the Lebesgue measure. In this case one obtains
> the absolute entropy given above. But one could also take as prior
> a noninvariant measure
> dmu(x) = dx p_0(x);
> then the density becomes \rho(x)=p(x)/p_0(x), and one arrives at the
> relative entropy.
OK, but as mentioned above, this 'relative entropy' is nonpositive.
> If there is no natural transitive symmetry group, there is no natural
> prior, and one has to make other useful choices. In particular, this
> is the case for random natural numbers.
If we adopt a finite-sets policy, together with your view of the
measure \mu as a prior, the above is shown to be a non sequitur.
The reason you don't see the natural prior is that you're not
examining the priors in terms of their representative sequences
on *finite* spaces.
> Choice A. Treating the natural numbers as a limiting situation of
> finite interval [0:n] suggests to use the measure with
> integral dmu(x) \phi(x) = sum_n \phi(n)
> as (improper) prior, making
> <f(x)> = sum_n \rho(n) f(n)
> the definition of the density; in this case, p_n=\rho(n) is the
> probability of getting n.
>
> Choice B. Statistical mechanics suggests to use instead the measure
> with
> integral dmu(x) \phi(x) = sum_n \phi(n)/n!
> as prior,
So \mu_B({n}) = 1/n! (n in {0,1,2,...}),
which normalizes to a Poisson distribution with mean 1.
> making
> <f(x)> = sum_n \rho(n) f(n)/n!
> the definition of the density; in this case, p_n=\rho(n)/n! is the
> probability of getting n.
>
> The maximum entropy ensemble defined by given expectations depends on
> the prior chosen. In particular, if the mean of a random natural number
> is given, choice A leads to a geometric distribution, while
> choice B leads to a Poisson distribution. The latter is the one
> relevant for statistical mechanics. Indeed, choice B is the prior
> needed in statistical mechanics of systems with an indefinite
> number n of particles to get the correct Boltzmann counting in the
> grand canonical ensemble. With choice A, the maximum entropy
> solution is unrelated to the distributions arising in statistical
> mechanics.
The present problem is not about the statistical mechanics of a
many-particle system, so it seems hard to justify forcing the
prior to be Poisson by reference to 'correct Boltzmann counting'.
--r.e.s.