# Signal signficancies low expected events

• A
• ChrisVer
In summary, the conversation discusses the calculation of the probability of discovering new physics based on observed events and expected background events. There is a debate about whether to use frequentist or Bayesian statistics, with the conclusion that for low numbers of events, the Poisson distribution should be used. It is cautioned that there is no meaningful way to interpret the probability of seeing a signal and the only meaningful number is the probability of a background fluctuation.
ChrisVer
Gold Member
Hi, I am a little "confused" of how to treat a problem stating: suppose you expect 1.25 background (SM) events from one measurement. During one particular measurement, you observe 5 events. What's the probability that you discovered new physics?
The standard/basic way to go about it is to calculate the significance of your "hypothetized new signal", that is $s = \frac{S}{\sqrt{B}}= \frac{5-1.25}{\sqrt{1.25}}=3.35$, which tells us that the new signal should be 3.35 sigma (standard deviations) away from the expected value to explain the observed events. In probabilities this translates to 99.958% signal exists/ 0.042% it's a statistical fluctuation, right?

My question is that the probabilities in those cases are taken from the standard normal distribution. Wouldn't (for so small expected events) the random variable (observed events) be taken out a Poisson distribution? In explicit wouldn't I have to calculate what's the probability of : $P( x \ge 5 | \lambda =1.25 )\approx 0.9\%$ (being statistical fluctuation) and the complementary for a signal? This would be relatively bad for the significance, as the standard deviation of a Poisson distributed variable is not equal to the square root of the events (for low expected events)...

Thanks.
Maybe this deserves to be moved in the statistics, but please do whatever you find fit.

Last edited:
ChrisVer said:
What's the probability that you discovered new physics?
Ooops! Be careful here! Are you using frequentist or Bayesian statistics? In Bayesian statistics it depends on your prior. Frequentist statistics does not answer this question. It answers the question "what is the probability this (or something more extreme) would happen by chance?" which is not the same question.

And yes, for a low number of events you need to use the Poisson distribution.

mfb and ChrisVer
Orodruin said:
In Bayesian statistics it depends on your prior.
I don't think I can use a prior for this? Except for saying that I expect (given the model) $B+ \mu S$ where $\mu$ is taken from a uniform prior, and calculating stuff like the 95-quantile for mu to claim a discovery or not (via credibility levels)... seeing eg how the likelihood works:
$L( x_{obs} | B + \mu S ) = Poi(x_{obs} | B+\mu S) Uni(\mu)$
again I don't think it's easy to determine by simple calculations the ranges for mu or the results of its quantiles (would have to input those in a statistical program).

Orodruin said:
Frequentist statistics does not answer this question. It answers the question "what is the probability this (or something more extreme) would happen by chance?" which is not the same question.
I think this is done easier via calculations? I.e. it results in the way I calculated the Poisson probability (and also how partially I interpreted the result of it, being a stat fluctuation/happened by chance). Maybe I was wrong to move one step further and say that the complementary (99.1%) is the probability of having seen a signal.

Orodruin said:
And yes, for a low number of events you need to use the Poisson distribution.
yup, so the significance (as I defined it) for such cases wouldn't be a good indicator?

I have these two comics posted on my office door:
Xkcd:

SMBC: http://www.smbc-comics.com/index.php?id=4127

mfb, ChrisVer, Lord Crc and 1 other person
ChrisVer said:
Maybe I was wrong to move one step further and say that the complementary (99.1%) is the probability of having seen a signal.
Correct. Stay away from such an interpretation, there is no meaningful way to define "the probability that you saw a signal". "The probability of a background fluctuation that large" is the only meaningful number.

ChrisVer

## 1. What are "signal significances" in scientific research?

"Signal significances" in scientific research refer to the statistical measure of the likelihood that a particular signal or phenomenon is present in a dataset, rather than being due to random chance. It is often used in studies that involve low expected events, where the signal may be small and difficult to detect.

## 2. How are signal significances calculated?

Signal significances are typically calculated using statistical methods such as the standard deviation or the p-value. These methods compare the observed data to what would be expected under the null hypothesis (i.e. no signal present). The resulting value represents the confidence level of the signal's presence.

## 3. Why are low expected events important in scientific research?

Low expected events are important in scientific research because they can provide valuable insights into rare or unexpected phenomena. These events may hold the key to understanding complex systems or uncovering new discoveries that can advance our knowledge and understanding of the world.

## 4. What are some common challenges when studying low expected events?

Some common challenges when studying low expected events include the need for large datasets to achieve statistical significance, the potential for false positives due to multiple testing, and the difficulty in distinguishing between a true signal and random noise. Additionally, low expected events may require specialized techniques and equipment to detect and measure accurately.

## 5. How can scientists improve the detection of low expected events?

Scientists can improve the detection of low expected events by using advanced statistical methods, increasing the size and quality of datasets, and collaborating with other researchers to combine data and resources. They can also utilize new technologies and techniques to improve the sensitivity and accuracy of their measurements.

Replies
4
Views
1K
Replies
1
Views
1K
Replies
17
Views
5K
Replies
26
Views
5K
Replies
12
Views
2K
Replies
49
Views
10K
Replies
10
Views
1K
Replies
4
Views
2K
Replies
0
Views
615
Replies
12
Views
2K