Formula for entropy conditional on >1 variables?

Click For Summary
SUMMARY

The correct formula for computing conditional entropy with multiple variables is H(A|B,C) = H(A,B,C) - H(B,C). This conclusion was reached after initial uncertainty regarding the validity of the formula H(A,B,C) - H(B,C). The discussion emphasizes the importance of understanding the relationships between the variables involved in conditional entropy calculations, particularly when visualized through a Venn diagram.

PREREQUISITES
  • Understanding of entropy and conditional entropy concepts
  • Familiarity with probability distributions
  • Knowledge of Venn diagrams for visualizing relationships
  • Basic skills in mathematical notation and manipulation
NEXT STEPS
  • Study the derivation of the conditional entropy formula in detail
  • Explore applications of conditional entropy in information theory
  • Learn about joint entropy and its relationship to conditional entropy
  • Investigate Venn diagram representations of entropy concepts
USEFUL FOR

Data scientists, statisticians, and anyone involved in information theory who seeks to deepen their understanding of entropy and its applications in probability distributions.

Inquirer
Messages
4
Reaction score
0
Hello,

I want to compute the conditional entropy H(A | B,C), given probability distributions for each of the variables.

It would be nice to have a right-hand side not involving conditionals. H(A|B) = H(A,B) - H(B) but how does it work out if there are more than one conditional variable?

I tried, as a dumb guess, H(A,B,C) - H(B,C), but from using it further down the line, I doubt that's correct.

Thanks!
 
Physics news on Phys.org
I worked through it again and discovered an error or two in my calculations. In the end it looks like

H(A|B,C) = H(A,B,C) - H(B,C)

is correct after all.

This is quite clear when illustrated with a Venn diagram similar to the one https://secure.wikimedia.org/wikipedia/en/wiki/Conditional_entropy" .
 
Last edited by a moderator:

Similar threads

  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 1 ·
Replies
1
Views
3K
  • · Replies 2 ·
Replies
2
Views
2K
Replies
2
Views
2K
  • · Replies 4 ·
Replies
4
Views
2K
  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 30 ·
2
Replies
30
Views
5K
  • · Replies 2 ·
Replies
2
Views
2K
  • · Replies 2 ·
Replies
2
Views
2K