1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Homework Help: Concavity of entropy

  1. Jul 23, 2009 #1
    1. The problem statement, all variables and given/known data
    Shannon entropy is a concave function defined as follows:
    [tex]H(X)=-\sum_{x}p(x)\log p(x)[/tex]

    Conditional Shannon entropy is defined as follows:
    [tex]H(X|Y)=\sum_{y} p(y) H(X|Y=y)=-\sum_{y} p(y)\sum_{x}p(x|Y=y)\log p(x|Y=y)[/tex]

    Can we deduce that:
    [tex]\sum_{y} p(y)H(X|Y=y)\geq H(X|Y=y)[/tex]

    2. Relevant equations

    3. The attempt at a solution
    I would say yes because of the concavity but I am confused with the 2 random variables.
  2. jcsd
  3. Jul 24, 2009 #2


    User Avatar

    I'm assuming the RHS of your final expression is also meant to be summed over [tex]y[/tex], otherwise it doesn't make much sense...

    The way to go about this is to think about the nature of [tex]p(y)[/tex]. What constraints do you know about the values that [tex]p(y)[/tex] can take?

    PS Do you really mean [tex]\geq[/tex] in the last line, or do you mean [tex]\leq[/tex]?
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook