Conditional Entropy, H(g(X)|X)

In summary, the conditional entropy of a function of random variable X given X is 0 because for any particular value of X, the output of the function is fixed, resulting in no uncertainty about the function's output once X is known. This is represented as H(g(X) | X) = 0 in symbols. The underlying concept is that if we know X, we can precisely infer the output of g(X) with probability 1. This explanation was provided by another person in response to someone seeking clarification on the concept of conditional entropy.
  • #1
PhillipKP
65
0
Hi

I'm trying to convince myself that the conditional entropy of a function of random variable X given X is 0.

H(g(X)|X)=0

The book I'm reading (Elements of Information Theory) says that since for any particular value of X, g(X) is fixed, thus the statement is true. But I don't understand why this makes the conditional entropy 0.

Obviously I don't understand conceptually what conditional entropy really is...

Can anyone please provide some "gentle" insight into this?
 
Physics news on Phys.org
  • #2
If we know X=x, then we can precisely infer g(X) = g(x) with probability 1, and anything else with probability 0. Hence there is no uncertainty about g(X) once we know X. Written in symbols, the previous sentence is H(g(X) | X) = 0.
 
  • #3
Ah that makes very good conceptual sense. Thank you for the short but insightful explanation.

Cheers

Phillip
 

1. What is conditional entropy?

Conditional entropy is a measure of the uncertainty or randomness in a system, given that certain conditions or information are known. It is calculated using the conditional probability of a random variable. In simple terms, it represents the amount of uncertainty that remains in a system even after we have some knowledge about it.

2. How is conditional entropy different from regular entropy?

Regular entropy, also known as Shannon entropy, measures the amount of uncertainty in a system without any prior knowledge or conditions. On the other hand, conditional entropy takes into account additional information or conditions that may affect the system. It is always equal to or less than regular entropy, as knowing more information can only decrease uncertainty.

3. What is the formula for calculating conditional entropy?

The formula for conditional entropy is H(Y|X) = ∑ P(x) * H(Y|x), where H(Y|x) is the conditional entropy of Y given X, and P(x) is the probability of X. This formula can be extended to multiple variables as well.

4. How is conditional entropy used in data analysis?

Conditional entropy is a useful tool in data analysis, particularly in machine learning and information theory. It can be used to measure the amount of information gained or lost when a particular condition is known. It is also commonly used in feature selection and model evaluation to understand the relationships between different variables.

5. Can conditional entropy be negative?

Yes, conditional entropy can be negative in certain cases. This occurs when the amount of information gained from knowing the condition is greater than the uncertainty in the system. It is often seen in systems with strong correlations between variables. However, it is more common for conditional entropy to be positive, indicating that the condition does not provide much additional information about the system.

Similar threads

MHB How to calculate conditional expectation E[g(x) | x>= Q] for x ~ exp(1)
    • Set Theory, Logic, Probability, Statistics
Replies
2
Views
2K
I Entangled, mixed state with conditional entropy zero
    • Quantum Physics
Replies
1
Views
639
A Papapetrou transformation: Conditions to be satisfied to achieve transformation
    • Special and General Relativity
Replies
1
Views
789
I The truth value of ##P(x)## → ##Q(x)##
    • Set Theory, Logic, Probability, Statistics
Replies
16
Views
2K
How to prove the chain rule for mutual information
    • Calculus and Beyond Homework Help
Replies
1
Views
701
I Understanding Physical Processes: EM, Gravitational, Mechanical & Entropic
    • Classical Physics
Replies
1
Views
627
A Why the triangle inequality is greater than the 2 max{f(x),g(x)}
    • General Math
Replies
1
Views
925
I Holographic Principle and/or implications of the entropy bound
    • Beyond the Standard Models
Replies
1
Views
818
A Understanding Free Expansion of Gas: Velocity Distribution Function Evolution
    • Classical Physics
Replies
14
Views
1K
I Enumerating a Large Ordinal: Can We Find a Limit to the Continuum?
    • Set Theory, Logic, Probability, Statistics
Replies
27
Views
3K

Hot Threads

Recent Insights

Back
Top