Kolmogorov Entropy: Intuitive Understanding & Relation to Shannon Entropy

[paralleltransport]

Dear all,

I'm trying to have an intuition of what Kolmogorov Entropy for a dynamical system means. In particular,

1. what is Kolmogorov Entropy trying to quantify? (what are the units of KS Entropy, is it bits or bits/seconds).
2. What is the relation of KS entropy to Shannon Entropy?
3. In what way does Kolmogorov entropy successfully quantify the question (1) tries to answer? Why is the definition intuitively the "right" one?

I've been trying to read on it, but it seems dominated by math jargon that makes a physics person a little bit intimidated. My background is that I know statistical mechanics (graduate level).

For example Shannon entropy would be:

1. Tries to quantify the minimum # of bits (or symbols of some language) to specify to uniquely specify a particular state of a system. This also tells us how "unpredictable" the system is. If more bits is needed, that means you need more information to really "pin" down the microstate of the system.

2. Well shannon-entropy is shannon.

3. Expection[\ln p] reduces to very simple cases: for example, let's say I have N balls, one of which is heavier, equally likely. What is the minimum # of bits? p = {1 \over N} and if you compute it makes sense log(N) should be the # of symbols to code for a microstate.

Thank you!

 

[gtoguy97]

Kolmogorov entropy (also known as algorithmic complexity or Kolmogorov-Sinai entropy) is a measure of the unpredictability of a dynamical system. It is defined as the rate of growth of the number of distinct patterns seen in the system. Unlike Shannon entropy, which quantifies the average information content in a system, Kolmogorov entropy measures the average amount of information needed to uniquely identify a given state of the system. The units of KS entropy are typically bits/second.Kolmogorov entropy is related to Shannon entropy in that they both quantify the complexity of a system, but in different ways. Shannon entropy measures the amount of information that can be extracted from a system based on its probability distribution. Kolmogorov entropy, on the other hand, measures the amount of information required to uniquely identify a given state of the system.The definition of Kolmogorov entropy is intuitively the "right" one because it captures the idea that more information is needed to specify the exact state of a system. For example, let's say you have two balls, one of which is heavier than the other. If you know that one ball is heavier, you need log(2) bits of information to uniquely identify which ball is heavier. If instead you had N balls, then you would need log(N) bits of information. This is essentially what Kolmogorov entropy tries to capture: the amount of information needed to uniquely identify a state of a system.