B What motivates Bayes' Theorem?

[Agent Smith]

TL;DR

What motivates Bayes' Theorem

As far as I know, Bayes' theorem is $P(A|B) = \frac{P(A) \times P(B|A)}{P(A) \times P(B|A) + P(\neg A) \times P(B|\neg A)}$.

I recall someone saying Bayes' theorem revolutionized probability. Bayes himself and Laplace are supposedly key figures in this revolution. I know how to apply the theorem, but in what sense is the theorem a "revolution"?

 

[jedishrfu]

Bayes theorem allows for probabilities to be updated based on new evidence making it a better way to estimate probabilities.

https://en.wikipedia.org/wiki/Bayes'_theorem

 

[PeroK]

Bayes theorem allows for probabilities to be updated based on new evidence making it a better way to estimate probabilities.

https://en.wikipedia.org/wiki/Bayes'_theorem

You may be confusing Bayesian Statistics, which is what you describe, and Bayes' Theorem, which is the essential and elementary component of probability theory quoted in the OP. The former is the Bayesian Interpretation, as given in that Wikipedia page.

Perhaps Bayes' Theorem was revolutionary, in that it allowed probabilities to be calculated in reverse. Bayesian statistics allows probabilities to be calculated in a wider range of circumstances. Most of the time, Bayesian and frequentist calculations agree.

That said, a frequentist obviously updates probabilities based on new evidence. What the Bayesian can do is start with very limited evidence or no evidence at all! The frequentist can't do that.

 

[PeroK]

TL;DR Summary: What motivates Bayes' Theorem

As far as I know, Bayes' theorem is $P(A|B) = \frac{P(A) \times P(B|A)}{P(A) \times P(B|A) + P(\neg A) \times P(B|\neg A)}$.

I've never seen it like that before. The denominator on the right is just an expansion of $P(B)$. The simplest form of Bayes' Theorem, IMO, is:
$$P(B)P(A|B) = P(A \cap B) = P(A)P(B|A)$$You can draw a Venn diagram to see this.

 

[vela]

What the Bayesian can do is start with very limited evidence or no evidence at all! The frequentist can't do that.

Which one are you, a Bayesian or frequentist?

 

[PeroK]

Which one are you, a Bayesian or frequentist?

I'm a frequentist myself. I'm not convinced that the probabilities that can be calculated from the Bayesian Interpretation are, in all circumstances, meaningful.

 

[PeroK]

For example, using Bayesian methods to calculate the probability that the universe is flat (or whatever). Yes, you get a number and can call it a probability, but what does it mean? As a frequentist, you have to be honest and say that there is no answer (from frequentist probability theory). And sometimes "no answer" is better and more honest than an answer that may or may not mean anything.

 

[pines-demon]

Bayes theorem has been extrapolated to create the Bayesian interpretation of probability which has been extrapolated to a whole Bayesian epistemology. This has had some influence in physics with some ideas of Bayesian thermodynamics and Bayesian interpretations of quantum mechanics, but none of these are very popular.

 

[haushofer]

Bayes' formula allows you to convert P(A|B) into P(B|A). E.g., in tossing a coin you can calculate the probability of an outcome given a (null)hypothesis. But often you want to know the reverse: what's e.g. the probability the coin is not fair given the outcome?

Or consider a suspect. You can calculate with some model the probability he was at the crime scene given he's innocent. But a prosecutor wants to know the reverse probability. Confusing these is known as the prosecutor's fallacy.

 

[haushofer]

For example, using Bayesian methods to calculate the probability that the universe is flat (or whatever). Yes, you get a number and can call it a probability, but what does it mean? As a frequentist, you have to be honest and say that there is no answer (from frequentist probability theory). And sometimes "no answer" is better and more honest than an answer that may or may not mean anything.

It means how much confidence you have given certain background information. Subjectivity doesn't deprive things of meaning.

 

[PeroK]

It means how much confidence you have given certain background information. Subjectivity doesn't deprive things of meaning.

One could argue that science must be objective.

 

[pines-demon]

Bayes' formula allows you to convert P(A|B) into P(B|A). E.g., in tossing a coin you can calculate the probability of an outcome given a (null)hypothesis. But often you want to know the reverse: what's e.g. the probability the coin is not fair given the outcome?

Or consider a suspect. You can calculate with some model the probability he was at the crime scene given he's innocent. But a prosecutor wants to know the reverse probability. Confusing these is known as the prosecutor's fallacy.

In mathematics one has to distinguish between probabilities and likelihood.

 

[Agent Smith]

I've never seen it like that before. The denominator on the right is just an expansion of $P(B)$. The simplest form of Bayes' Theorem, IMO, is:
$$P(B)P(A|B) = P(A \cap B) = P(A)P(B|A)$$You can draw a Venn diagram to see this.

Si, I don't know why, if it all, one form is preferred over the other. The derivation is also difficult to follow.

 

[Agent Smith]

Bayes' formula allows you to convert P(A|B) into P(B|A). E.g., in tossing a coin you can calculate the probability of an outcome given a (null)hypothesis. But often you want to know the reverse: what's e.g. the probability the coin is not fair given the outcome?

Or consider a suspect. You can calculate with some model the probability he was at the crime scene given he's innocent. But a prosecutor wants to know the reverse probability. Confusing these is known as the prosecutor's fallacy.

Is that it? We can compute "reverse probabilities"?

 

[PeroK]

Si, I don't know why, if it all, one form is preferred over the other.

I prefer simplicity and memorability.

The derivation is also difficult to follow.

What derivation?

 

[Agent Smith]

Bayes theorem allows for probabilities to be updated based on new evidence making it a better way to estimate probabilities.

https://en.wikipedia.org/wiki/Bayes'_theorem

This "new evidence" is a sample, but we can use this even in standard (frequentist) statistics.

 

[Agent Smith]

I prefer simplicity and memorability.

What derivation?

The derivation from your simpler formula to the more complex one I posted. I drew a Venn Diagram and it meshes with what I know. Gracias

 

[PeroK]

The derivation from your simpler formula to the more complex one I posted. I drew a Venn Diagram and it meshes with what I know. Gracias

If $A_1, \dots A_n$ are mutually exclusive events that exhaust the sample space, then for any event $B$:
$$P(B) = P(A_1)P(B|A_1) + \dots +P(A_n)P(B|A_n)$$A special case of this is where the events are $A$ and $\neg A$, in which case:
$$P(B) = P(A)P(B|A) + P(\neg A)P(B|\neg A)$$That's a general expansion of $P(B)$, so there is no need to include that in Bayes' theorem. The simple $P(B)$ should do just as well.

 

[Dale]

TL;DR Summary: What motivates Bayes' Theorem

in what sense is the theorem a "revolution"?

The main thing is that it allows you to reverse conditional probabilities. So if you know the unconditional probabilities and also the conditional probability $P(A|B)$ then you can calculate the reversed conditional probability $P(B|A)$.

This is important in science. Usually my theory tells me $P(observation|theory)$. But then I can use Bayes theorem to determine $P(theory|observation)$. So I can use experiment to test my theory.

 

[Agent Smith]

Is there anything I should note other than that probabilities can be reversed with Bayes' theorem? The Wikipedia page says that Bayes' "real intention" was to "prove the existence of God."

For $P(A|B) = \frac{P(A) \times P(B|A)}{P(B)}$, what happens when $P(B) = 0$?
Gracias. The Wikipedia article on Bayes' theorem mentions the caveat that $P(B) \ne 0$.

 

[Vanadium 50]

TL;DR Summary: What motivates Bayes' Theorem

I recall someone saying

Reference please? Otherwise we are chasing ghosts.

 

[Agent Smith]

@Vanadium 50 , Wikipedia?

 

[Vanadium 50]

So we're going to play that game, eh? "It's somewhere in those zillion pages."

Count me out.

 

[Agent Smith]

So we're going to play that game, eh? "It's somewhere in those zillion pages."

Count me out.

Wikipedia has a good article on Bayes' theorem. Most of what I wrote here cometh from there.

 

[Agent Smith]

What the Bayesian can do is start with very limited evidence or no evidence at all! The frequentist can't do that.

A frequentist takes a sample (the least). A Bayesian ___?

 

[Agent Smith]

For example, using Bayesian methods to calculate the probability that the universe is flat (or whatever). Yes, you get a number and can call it a probability, but what does it mean? As a frequentist, you have to be honest and say that there is no answer (from frequentist probability theory). And sometimes "no answer" is better and more honest than an answer that may or may not mean anything.

Throwing darts here, but does the process work like this:
Fix a significance level $\alpha = 0.05$
$H_0$ = The universe is curved i.e. $\mu_0 = v$ and $\sigma_0 = u$ where $v$ is a specific value. Here $\mu_0$ is the mean of some physical measurement and $\sigma_0$ its standard deviation
$H_a$ = The universe is flat i.e. $\mu_0 > v$

We then proceed to make some measurements (sample, of size $n$). Compute the mean from the sample $\mu_s = x$. We find the $\text{z score} = \frac{\mu_s - \mu_0}{\frac{\sigma}{\sqrt n}}$. We can read off our $\text{P-value}$ from a z-table. If $\text{P-value} < \alpha$ we can reject $H_0$ in favor of $H_a$ and conclude that the universe is flat.

?

 

[PeroK]

A frequentist takes a sample (the least). A Bayesian ___?

You can read about it. The difference is more about what fundamentally is a probability. At this stage you should focus on probability theory and your course material.

It's unfortunate that many people associate Bayes Theorem with Bayesian statistics. This leads to these esoteric debates, which are of limited value at this stage. Bayes Theorem is elementary and fundamental and shouldn't lead to a debate about Bayesian statistics.

 

[PeroK]

Throwing darts here, but does the process work like this:
Fix a significance level $\alpha = 0.05$
$H_0$ = The universe is curved i.e. $\mu_0 = v$ and $\sigma_0 = u$ where $v$ is a specific value. Here $\mu_0$ is the mean of some physical measurement and $\sigma_0$ its standard deviation
$H_a$ = The universe is flat i.e. $\mu_0 > v$

We then proceed to make some measurements (sample, of size $n$). Compute the mean from the sample $\mu_s = x$. We find the $\text{z score} = \frac{\mu_s - \mu_0}{\frac{\sigma}{\sqrt n}}$. We can read off our $\text{P-value}$ from a z-table. If $\text{P-value} < \alpha$ we can reject $H_0$ in favor of $H_a$ and conclude that the universe is flat.

?

There is only one universe, so the sample size is 1.

 

[Agent Smith]

There is only one universe, so the sample size is 1.

Yes that's correct, I forgot. What about the measurements I "used" to test the hypotheses that the universe is flat?

 

[PeroK]

Yes that's correct, I forgot. What about the measurements I "used" to test the hypotheses that the universe is flat?

The universe is either flat or it isn't. Probability doesn't apply. One could say that the distribution is everything and in this case there is either no distribution or the distribution is entirely unknown.