Need Help calculating Entropy & probabilities of a simple Markov Process

Click For Summary
SUMMARY

This discussion focuses on calculating the probabilities and entropy of a binary Markov process where the transition probabilities are defined as follows: 0 follows a 0 with a probability of 7/8, and 1 follows a 1 with a probability of 1/2. The user seeks to determine the probabilities of symbols 0 and 1, concluding that p(0) = 4p(1) and aiming for p(0) = 4/5 and p(1) = 1/5. Additionally, two methods for calculating entropy are proposed: H(S) = p(0) H0 + p(1) H1 and H(S) = -p(0)log(p(0)) - p(1)log(p(1)), with a request for clarification on the correct approach.

PREREQUISITES
  • Understanding of Markov processes and transition probabilities
  • Familiarity with entropy concepts in information theory
  • Knowledge of logarithmic functions and their properties
  • Ability to solve linear equations
NEXT STEPS
  • Calculate the probabilities p(0) and p(1) using the provided transition probabilities
  • Explore the application of the formula H(S) = -p(0)log(p(0)) - p(1)log(p(1)) for entropy calculation
  • Study the implications of Markov processes in real-world applications
  • Review additional examples of entropy calculations in binary systems
USEFUL FOR

Students studying probability theory, data scientists working with Markov models, and anyone interested in information theory and entropy calculations.

degs2k4
Messages
72
Reaction score
0

Homework Statement



I am in a hurry with the following problem:

We have a source that produces binary symbols, 0 and 1.
0 follows a 0 at probability 7/8
1 follows a 1 at probability 1/2

A) Calculate probability of the symbols 0 and 1 to appear.
B) Calculate entropy of source.

The Attempt at a Solution



A) We can say that:

0 follows a 0 at probability p(0,0) = 7/8 => 1 follows a 0 at probability p(0,1) = 1/8
1 follows a 1 at probability p(1,1) = 1/2 => 0 follows a 1 at probability p(1,0) = 1/2

p(0) = p(0) p(0,0) + p(1) p(1,0) = p(0) 7/8 + p(1) 1/2
p(1) = p(0) p(0,1) + p(1) p(1,1) = p(1) 1/2 + p(0) 1/8

Now, we have to get p(0) and p(1) but how ? I only come up with p(0) = 4 p(1)
(according to "a solution", p(0) should be 4/5 and p(1) 1/5)

B) I think it can be calculated in 2 ways:

Way 1:
H(S) = p(0) H0 + p(1) H1
(sum of the probabilities of getting a 0 and 1 multiplied by its respective variabilities, where
H0 = -7/8log(7/8) - 1/8log(1/8)
H1 = -1/2log(1/2) - 1/2log(1/2)

Way 2:
H(S) = -p(0)log(p(0)) -p(1)log(p(1))

Which way is the correct one?

Thanks in advance!
 
Physics news on Phys.org
Anyone? I am a bit in a hurry, need to have this clear for tomorrow...
 

Similar threads

  • · Replies 5 ·
Replies
5
Views
2K
Replies
8
Views
2K
Replies
6
Views
2K
Replies
1
Views
1K
Replies
28
Views
2K
  • · Replies 5 ·
Replies
5
Views
3K
  • · Replies 8 ·
Replies
8
Views
2K
  • · Replies 6 ·
Replies
6
Views
2K
  • · Replies 4 ·
Replies
4
Views
2K
  • · Replies 7 ·
Replies
7
Views
2K