Entropy of Sets: What It Is & How to Calculate

  • Context: Graduate 
  • Thread starter Thread starter phoenixthoth
  • Start date Start date
  • Tags Tags
    Entropy Set
Click For Summary
SUMMARY

The discussion centers on calculating the entropy of sets, specifically focusing on the mutual information between two sets A and B. The user references the formula for mutual information, I(A;B) = Log_2 (P(A&B) / (P(A)P(B)), and seeks to establish a nontrivial probability measure on the smallest sigma algebra G containing A and B. The conversation highlights the challenge of defining a probability measure that satisfies the conditions P(A), P(B) ∈ [0,1] and P(G) = 1, indicating a need for a systematic approach to this problem.

PREREQUISITES
  • Understanding of information theory concepts, particularly mutual information.
  • Familiarity with sigma algebras and their properties.
  • Knowledge of probability measures and their construction.
  • Basic grasp of logarithmic functions and their applications in entropy calculations.
NEXT STEPS
  • Research the construction of nontrivial probability measures in the context of sigma algebras.
  • Explore advanced topics in information theory, focusing on mutual information and its applications.
  • Study the relationship between entropy and randomness in closed systems.
  • Examine examples of topological entropy and its implications for set theory.
USEFUL FOR

Mathematicians, data scientists, and information theorists interested in the theoretical foundations of entropy and mutual information in set theory.

phoenixthoth
Messages
1,600
Reaction score
2
What is the entropy of a set?

One of the two should be a general guidline:

# A measure of the disorder or randomness in a closed system.
# A measure of the loss of information in a transmitted message.

I've seen topological entropy (bowen) and entropy of random variables, but what about of sets?
 
Physics news on Phys.org
mutual information

What I'm really getting at is the so called "mutual information" one set A has of another set B.

This is defined in information theory if A and B are random variables.

I want it if they are general sets.

I had a 'thought.' Maybe I can look at the smallest sigma algebra G containing A and B (I don't mean the intersection), and invent a nontrivial probability measure on this G. This turns A and B into events. Then the formula I've seen for mutual information is this:
I(A;B)=Log_2 (P(A&B) / (P(A)P(B))).

But what would be a nontrivial probability measure to put so that P(A), P(B) ∈ [0,1]. Also, P(G)=1. Is there some canonical nontrival P() that I can construct? How would I do this?
 

Similar threads

Graduate Feature Selection, Information Gain and Information Entropy
  • · Replies 5 ·
Replies
5
Views
2K
Graduate How Does Unequal Probability Affect Entropy Calculation in Information Theory?
  • · Replies 3 ·
Replies
3
Views
2K
Graduate Reconciling Determinism, Entropy, and Quantum Statistics
  • · Replies 6 ·
Replies
6
Views
3K
Graduate Could the Laws of Physics Be Proportional to Entropy?
  • · Replies 5 ·
Replies
5
Views
2K
High School Why Does Black Hole Entropy and Information Loss Matter?
  • · Replies 7 ·
Replies
7
Views
2K
Graduate As stars contract, why does total entropy rise?
  • · Replies 4 ·
Replies
4
Views
2K
High School Is Entropy the Key to Understanding the Big Bang and the Fate of the Universe?
  • · Replies 27 ·
Replies
27
Views
6K
Undergrad How can the maximum entropy and minimum energy principles be physical?
  • · Replies 13 ·
Replies
13
Views
4K
High School Question about the Entropy of a Black Hole Singularity
  • · Replies 17 ·
Replies
17
Views
4K
Undergrad Problem regarding understanding entropy
  • · Replies 3 ·
Replies
3
Views
2K