How do I express that a 100% occurrence in a small sample is low "confidence"?

• I
• Archosaur
Archosaur
TL;DR Summary
How do I express that a 100% frequency occurrence in a small sample is low "confidence", when, strictly speaking, its 95% confidence interval is (1,1)?
In experiment A: I observe an event 2 times in 2 trials.
In experiment B: I observe an event 100 times in 100 trials.

In both cases, I calculate a frequency of 100%
In both cases, I calculate a 95% confidence interval of (1, 1).

But intuitively the result of experiment B is "stronger" than that of A. How can I express this as a number?

Archosaur said:
TL;DR Summary: How do I express that a 100% frequency occurrence in a small sample is low "confidence", when, strictly speaking, its 95% confidence interval is (1,1)?

In experiment A: I observe an event 2 times in 2 trials.
In experiment B: I observe an event 100 times in 100 trials.

In both cases, I calculate a frequency of 100%
In both cases, I calculate a 95% confidence interval of (1, 1).

But intuitively the result of experiment B is "stronger" than that of A. How can I express this as a number?

Assume a null hypothesis of whatever frequency you think is appropriate. 50% maybe. Then calculate the probability that such an experimental result is due to chance, ie. that your null hypothesis is true. This will usually be very close to zero in the second case.

Or you could do a Bayesian analysis and the 95% credible interval would not be (1,1) in either case, but it would be quite broad in the low data case and quite narrow in the high data case

In the Bayesian case a beta distribution is the conjugate prior for a binomial random variable. The posterior is ##\beta(a+1,b+1)## where ##a## is the number of successes observed and ##b## is the number of failures observed.

From that you can calculate the credible interval. For ##(a=2,b=0)## we find that the 95% credible interval for ##\beta(3,1)## is 0.368 to 1.000. In contrast, for ##(a=100,b=0)## we find that the 95% credible interval for ##\beta(101,1)## is 0.971 to 1.000

This is awesome. Thanks very much for pointing me to the Beta distribution - this is exactly what I was looking for. I made a python function that calculated frequency and "credibility" (1 - width of 95% credible interval) for O observations in N trials up to 100, because I was curious what a credibility heatmap would look like in this space.

berkeman and Dale
Excellent! Personally, I think that this behaves as a reasonable person would expect. With just 2 observations it seems reasonable to say “I am pretty sure the probability is greater than 30%”. And with 100 observations it also seems reasonable to say “I am pretty sure the probability is greater than 96%”.

• Set Theory, Logic, Probability, Statistics
Replies
3
Views
851
• Set Theory, Logic, Probability, Statistics
Replies
1
Views
751
• Set Theory, Logic, Probability, Statistics
Replies
1
Views
815
• Set Theory, Logic, Probability, Statistics
Replies
14
Views
1K
• Set Theory, Logic, Probability, Statistics
Replies
6
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
6
Views
3K
• Set Theory, Logic, Probability, Statistics
Replies
4
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
1
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
18
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
24
Views
4K