Errors in Probability: Continuous and Discrete Distributions

1. Classifying as discrete, continuous, or mixed

These statements (or equivalents) can be found in authoritative-seeming websites:

X “A random variable is either discrete or continuous.”

X “A random variable is continuous if it can take infinitely many values.”

A random variable may be discrete, continuous, or mixed.  The same adjective is applied to its distribution.

✓	discrete	Can only take countably many values. (That is, finitely many, or a countable infinity.)Necessarily, each such value has a nonzero probability.
✓	continuous	Can take an uncountable set of values, and any specific value has a zero probability. (A full definition requires measure theory.)
✓	mixed	Can take an uncountable set of values, but there are some values it can take with nonzero probability.

X “Every random variable can be written as the sum of a discrete random variable and a continuous random variable.”

Not so. Example:

Darts are thrown at a disc target radius 1.  If a dart hits the target it scores its distance from the edge of the disc; if the dart misses the disc the score is 0.

The score will have a nonzero probability of being 0, then a continuous distribution from 0 to 1.  This cannot be represented as the sum, X+Y, of a discrete r.v. X and a continuous r.v. Y.

What the author meant was:

✓A mixed random variable can always be represented by compounding a continuous random variable and a discrete random variable via a Boolean random variable. 

Let the set of values of X with nonzero probability be {$D_i$} with probabilities $p_1, p_2, …$, writing $p_D=\Sigma p_i$, and the rest of the values be the set C, $P(X\in C)=1-p_D$, with density function f(x).  We construct the discrete r.v. $X_D$ such that $P(X_D=D_i) = \frac{p_i}{p_D}$ and the r.v. $X_C$ with pdf $f_C(x)=\frac{f(x)}{1-p_D}$.  Finally, the Boolean r.v. $X_B$ takes the values True with probability $p_D$ and False otherwise.  Then

$X=$ if $(X_B)$ then $X_D$ else $X_C$.

2. Random versus Equally Likely

In common usage, these terms are treated as equivalent, but in mathematics they are distinct.

X “A number from 1 to 10 is chosen at random.  What is the probability that it is divisible by 3?”

There is no way to answer this question because we are not told the distribution.  “At random” merely means the choice was not deterministic, i.e. there was more than one number that might have been chosen.  Presumably, a uniform distribution is intended, but it should be stated.

3. Countably infinite distributions

Infinite discrete distributions can lead to some counter-intuitive results.

Example 1: “If two natural numbers are chosen at random, what is the probability that they are coprime?

Strictly speaking, this problem statement is invalid.  As noted above, no distribution is specified, but this time we cannot presume a uniform distribution.

There is no such thing as a uniform countably infinite distribution since every individual value would have zero probability.

That said, this problem can be solved, and is a mathematical gem.  We can in principle restrict to the range 1 to N, solve it approximately, and take the limit as N tends to infinity.  Glossing over those details…

The probability that both numbers are divisible by some smaller prime p = $p^2$.

The probability that not both numbers are divisible by prime p = $1-p^{-2}$.

The probability that no prime divides both is $P = \Pi_p(1-p^{-2})$, where the product is over all primes.

Consider $\frac 1P = \Pi_p(1-p^{-2})^{-1} = \Pi_p\Sigma_{n=0}^{\infty}p^{-2n}$.

If we multiply out the product, every possible combination of a product of distinct primes (including the case of no primes), each raised to an even power, occurs exactly once in the resulting sum.  Thus

$\frac 1P = \Sigma_{n=1}^{\infty}\frac 1{n^2} = \frac{{\pi}^2}6$

(For those who do not know how to show $\Sigma_{n=1}^{\infty}\frac 1{n^2} = \frac{{\pi}^2}6$, that’s another gem.  Hint: factorise $\frac 1{x^2}\sin^2(x)$ based on its known roots.)

So $P=\frac 6{{\pi}^2}$.

Example 2: “Gambler’s Ruin”

Every now and then, someone rediscovers this foolproof gambling strategy for, say, roulette.

“Ignoring the complication of the 0 and 00 slots, we bet $1 on red.  If it loses we bet $2 on red.  If that loses we bet $4 on red, doubling up each time.  When eventually red comes up, we take our profit of $1.”

(Solution below.)

Example 3: “If you pay me $1, I will toss a coin repeatedly until it comes up heads.  If it is heads the first time, I will pay you 1⊄.  If the head comes up on the second toss I will pay you 2⊄.  If on the third, 4⊄, and so on, doubling for each additional tail.”

Should you play the game?

The probability that you win 1⊄ = $\frac 12$

The probability that you win 2⊄ = $\frac 14$

The probability that you win 4⊄ = $\frac 18$

The probability that you win $2^n$⊄ = $2^{-n-1}$

So the expected winnings should be

X $\Sigma_{n=0}^{\infty} 2^n.2^{-n-1}$⊄=$\Sigma _{n=0}^{\infty}2^{-1}$⊄=$\infty$⊄

In both example 2 and example 3, the error is taking the sum to infinity, ignoring practical constraints.

The solution to example 2:

You have limited funds.  At some point, you may run out of money.  Although the probability is small, the loss is enormous.  This exactly balances your expected win of $1.

The solution to example 3:

Again, the mistake is in assuming I have infinite wealth.  Still somewhat unrealistically, suppose I am ‘only’ worth $1bn, or about $2^{37}$⊄.

$\Sigma _{n=0}^{37}2^{-1}$⊄=19⊄.

4. Impossible versus zero probability

X “If an event has zero probability then it is impossible”

In a continuous distribution, each individual value has zero probability, yet can happen.  A radioactive atom may decay at any time T.  Any exact value of T has zero probability, yet can in principle occur.