1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Concavity of entropy

  1. Jul 23, 2009 #1
    1. The problem statement, all variables and given/known data
    Shannon entropy is a concave function defined as follows:
    [tex]H(X)=-\sum_{x}p(x)\log p(x)[/tex]

    Conditional Shannon entropy is defined as follows:
    [tex]H(X|Y)=\sum_{y} p(y) H(X|Y=y)=-\sum_{y} p(y)\sum_{x}p(x|Y=y)\log p(x|Y=y)[/tex]

    Can we deduce that:
    [tex]\sum_{y} p(y)H(X|Y=y)\geq H(X|Y=y)[/tex]

    2. Relevant equations

    3. The attempt at a solution
    I would say yes because of the concavity but I am confused with the 2 random variables.
  2. jcsd
  3. Jul 24, 2009 #2


    User Avatar

    I'm assuming the RHS of your final expression is also meant to be summed over [tex]y[/tex], otherwise it doesn't make much sense...

    The way to go about this is to think about the nature of [tex]p(y)[/tex]. What constraints do you know about the values that [tex]p(y)[/tex] can take?

    PS Do you really mean [tex]\geq[/tex] in the last line, or do you mean [tex]\leq[/tex]?
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Similar Discussions: Concavity of entropy
  1. Concave Function (Replies: 1)

  2. Concave Functions (Replies: 3)