Information Theory - Shannon's Self-Information units

Click For Summary
SUMMARY

The discussion centers on Shannon's self-information units, specifically the definition of "bits" as the unit of measurement for information content, represented mathematically as -log2(P(A)). The term "bit" refers to a binary digit, which signifies a storage device with two stable states. The conversation also highlights that while "bit" is commonly used for binary information, other units like "nat" can be employed for natural information using base e. The need for rigorous proof of these definitions is questioned, suggesting that coding examples may serve as justification.

PREREQUISITES
  • Understanding of Shannon's Information Theory
  • Familiarity with logarithmic functions and their bases
  • Basic knowledge of binary systems and digital storage
  • Experience with coding theory and examples
NEXT STEPS
  • Research the mathematical foundations of Shannon's Information Theory
  • Explore the differences between "bits" and "nats" in information measurement
  • Study coding examples that illustrate the application of self-information units
  • Investigate the implications of using different logarithmic bases in information theory
USEFUL FOR

Students and professionals in computer science, data science, and telecommunications, particularly those interested in information theory and coding practices.

Mihel
Messages
1
Reaction score
0
Information Theory - Shannon's "Self-Information" units

Hi,

I'm familiar with information and coding theory, and do know that the units of Shannon information content (-log_2(P(A))) are "bits". Where "bit" is a "binary digit", or a "storage device that has two stable states".

But, can someone rigorously prove that the units are actually "bits"? Or we should only accept it as a definition and then justify it with coding examples.

Thanks!
 
Engineering news on Phys.org


No, you can use what ever logarithmic base. For natural information you could use unit "nat" with base e and for binary information unit "bit" with base 2.
 

Similar threads

Is the Theory of Conservation of Information Faulty?
  • · Replies 14 ·
Replies
14
Views
2K
High School Is information the 5th state of matter?
  • · Replies 18 ·
Replies
18
Views
3K
Help understanding Shannon entropy
  • · Replies 9 ·
Replies
9
Views
4K
Graduate Proof that all information can be coded in binary?
  • · Replies 3 ·
Replies
3
Views
2K
High School Transferring information faster than the speed of light
  • · Replies 24 ·
Replies
24
Views
3K
Graduate Algorithmic information theory's QM view of reality
  • · Replies 2 ·
Replies
2
Views
2K
Undergrad A Bold New Take on Quantum Theory
  • · Replies 25 ·
Replies
25
Views
5K
How is information sent to and interpreted by computer monitors?
  • · Replies 8 ·
Replies
8
Views
2K
Graduate How Does Unequal Probability Affect Entropy Calculation in Information Theory?
  • · Replies 3 ·
Replies
3
Views
2K
Graduate What is the minimum observer energy?
  • · Replies 3 ·
Replies
3
Views
2K