Introductory Probability/Information Theory textbook reccomendations

[Niznar]

Hey folks,

I've been interested lately in reteaching myself probability as a foundation for information theory. I took an introductory course about 8 months ago and while I did fairly well, I don't feel like I took much intuition away from the course. Are there any recommendations on books or learning materials? Thanks!

 

[Constantinos]

A good book for the course in probability which is taught at my University is that of Bertsekas and Tsitsiklis ( Introduction to Probability). It's good in training the intuition and provides all the basic material one needs for more advanced courses.

 

[chrisaldrich]

For information theoretic specific textbooks on probability you might try the ones by Paul Pfeiffer or one of the two by Alfred Renyi. All three are excellent and have the added benefit of being published by Dover so they're relatively inexpensive. Pfeiffer's is more conversational and thus likely more accessible.

The most commonly used texts by university electrical engineering departments for preparation of information theory are Probability, Statistics, and Random Processes For Electrical Engineering (3rd Edition) by Alberto Leon-Garcia and Probability, Random Variables and Stochastic Processes by Athanasios Papoulis, which I'd generally recommend in that order though both are excellent. (Leon-Garcia has also written other books on communications which you might read afterwards.) These two also have the added benefit that they have material on Stochastic Processes which you're likely to want for your studies later on as well.

I have the text by Bertsekas and Tsitsiklis that Constantinos mentions and found it to be interesting but not great -- it's also a very cheaply manufactured text. MIT uses it for an intro course on probability, but it's likely that they do so because of the authors' affiliation.

Also inexpensively you'll probably find some of the texts from Schaum's Outline Series useful for additional examples/supplementation:
Schaum's Outline of Probability, Random Variables, and Random Processes, Second Edition (Schaum's Outline Series) by Hwei P. Hsu
Schaums Outline of Probability by Seymour Lipschutz
Schaum's Outline of Probability and Statistics by Murray R. Spiegel, John J. Schiller and R. Alu Srinivasan

You might also consider:
Probability and Random Processes With Applications to Signal Processing and Communications by Scott Miller and Donald Childers (I haven't browsed through it, but it's obviously application specific to your needs and might be a great text)

Probability and Statistics for Engineers and Scientists (9th Edition) by Ronald E. Walpole, Raymond H. Myers, Sharon L. Myers and Keying E. Ye (It's a bit easier presentation than some of the others while being very logical - take a peek at it if Leon-Garcia isn't quite your speed.)

If you want to go overboard on the heavier math side, take at look at William Feller's magnum opus.

You might also find my recent post http://chrisaldrich.posterous.com/on-choosing-your-own-textbooks" useful.

 

[jasonRF]

A standard intro book that I really like is "a first course in probability" by Ross. Old editions are cheap - I took a class from the 3rd edition that I still have on my shelf and refer to. Whether or not is much different from the way you were taught I cannot know ...

One non-standard book that I really like (and will buy sometime!) is "the art of probability" by Hamming. I think it does a great job with intuition, and will likely be a different perspective from the way you were taught. It isn't cheap, though, so try to find it in your library to see if you like it.

As mentioned by chrisaldrich, the books by Leon-Garcia and Papoulis are standard EE books. Leon-Garcia would be okay for a first intro to probability and has quite reasonable chapters on random processes - his topic selection is certainly very good for EEs. The only basic probabiliy topics that it does better than Ross is multivariate distributions (especially Gaussian random vectors), and convergence of sequences of random variables (Ross doesn't really discuss this at all). I love Papoulis but it would likely not be what you are looking for if it is intuition you are after. He does do a nice job covering topics like IQ sampling of random processes that are useful in communications, though. I actually took three courses on these subjects, one out of Ross, one from leon-Garcia (2nd edition), and one from Papoulis (3rd edition), so I know those books pretty well; for basic probability I would stick to Ross. For intuition and something different I like Hamming. Random processes - I would go with leon-garcia for an easy to understand and apply introduction.

If your library is well stocked, you may want to check out "intuitive probability and random processes" by Kay, which I have not read. His other books are really good, so perhaps that one is too!

good luck.

jason

EDIT: forgot about "fifty challenging problems in probability" by Mosteller. Has problems and detailed solutions. Emphasizes probabilistic reasoning, as opposed to emphasizing math. This is just the problems and solutions, but working problems is the best way to learn!

 

[rezeew]

I highly recommend the textbook "Introduction to Probability" by Joseph K. Blitzstein and Jessica Hwang. This book provides a clear and intuitive introduction to the fundamentals of probability, with a focus on real-world applications. It also includes exercises and examples to help solidify your understanding of the concepts. Another great resource is "Information Theory, Inference, and Learning Algorithms" by David MacKay, which covers both probability and information theory in a comprehensive and accessible manner. Both of these textbooks are highly recommended for self-study and will provide a strong foundation for understanding information theory. Good luck on your learning journey!