What are some good resources for learning about information theory?

[AndreasC]

As I've been studying statistical mechanics as well as some other things, I keep hearing about "information theory". For instance, I've heard about information theory as it relates to entropy, regarding some theorems of statistical mechanics, and I even heard about it in a Carl Bender lecture, where he said that you could show using information theory techniques that you could find series that approximate functions or solutions better than Pade or Taylor series etc. I find these things very interesting but I have no idea what to even look for to learn more about the subject, it seems to be one of these categories that different people from different disciplines may use in much different ways. Does anyone have any good sources to gain some understanding?

 

[Frabjous]

At the gentle introduction level, I enjoyed
Pierce An Introduction to Information Theory: Symbols, Signals and Noise
https://www.amazon.com/dp/0486240614/?tag=pfamazon01-20
Stone Information Theory: A Tutorial Introduction
https://www.amazon.com/dp/0956372856/?tag=pfamazon01-20

I never tried learning it as physics.

 

[AndreasC]

At the gentle introduction level, I enjoyed
Pierce An Introduction to Information Theory: Symbols, Signals and Noise
https://www.amazon.com/dp/0486240614/?tag=pfamazon01-20
Stone Information Theory: A Tutorial Introduction
https://www.amazon.com/dp/0956372856/?tag=pfamazon01-20

I never tried learning it as physics.

Interesting. However, I'm slightly more interested in the physics related approach.

 

[Frabjous]

Interesting. However, I'm slightly more interested in the physics related approach.

There’s Shannon’s paper which started it all
http://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
but it might not be the best place to start.

There is also this thread
https://www.physicsforums.com/threads/a-good-book-to-introduce-information-theory.830307/

 

[vanhees71]

For me the best book for physicists is

A. Katz, Principles of Statistical Mechanics, W. H. Freeman
and Company, San Francisco and London (1967).

 

[atyy]

http://www.inference.org.uk/mackay/itprnn/book.html
Information Theory, Inference, and Learning Algorithms
David MacKay

There are two major ideas in information theory, one is the quantification of information, which is closely related to the entropy of statistical physics. In physics, we usually make use of Stirling's approximation. In Mackay's book, you can find the material under the discussion of asymptotic equipartition.

The second major idea is about how well information can be transmitted in the presence of noise. In Mackay's book, it is in the chapter on the noisy channel coding theorem. As far as I understand, this idea is not much used in physics.

Two physics books that discuss basic information theory are

https://www.amazon.com/dp/0521873428/?tag=pfamazon01-20
https://ocw.mit.edu/courses/physics...-fall-2013/lecture-notes/MIT8_333F13_Lec6.pdf (free notes on which the book is based)
Statistical Physics of Particles
Mehran Kardar

https://www.amazon.com/dp/0486497550/?tag=pfamazon01-20
Science and Information Theory
Leon Brillouin

An example of information-like quantities in the research literature is:

https://arxiv.org/abs/1007.4825
Renyi entropy, mutual information, and fluctuation properties of Fermi liquids
Brian Swingle

 

[hutchphd]

I love The Information by James Gleick. One of my favorite reads...not technical but fascinating

 

[Frabjous]

W. T. Grandy, ‘‘Resource letter ITP-1: Information theory in physics,’’
Am. J. Phys. 65, 466–476 􏰐1997

is a guide to the literature

 

[radium]

Elements of information theory by Cover and Thomas is one of the best general texts. Information, physics, and computation by Mezard and Montanari applies ideas from information theory and the physics of disordered systems to computation.

I also second David MacKay’s book.

 

[AndreasC]

W. T. Grandy, ‘‘Resource letter ITP-1: Information theory in physics,’’
Am. J. Phys. 65, 466–476 􏰐1997

is a guide to the literature

Very interesting, I'll check this out first and then look at the rest.

 

[marcusl]

I also highly recommend E. T. Jaynes's Phys Rev article showing the link between information theory and statistical mechanics
Jaynes's papers at Wash U