Signal signficancies low expected events

  • A
  • Thread starter ChrisVer
  • Start date
  • #1
ChrisVer
Gold Member
3,337
440

Main Question or Discussion Point

Hi, I am a little "confused" of how to treat a problem stating: suppose you expect 1.25 background (SM) events from one measurement. During one particular measurement, you observe 5 events. What's the probability that you discovered new physics?
The standard/basic way to go about it is to calculate the significance of your "hypothetized new signal", that is [itex] s = \frac{S}{\sqrt{B}}= \frac{5-1.25}{\sqrt{1.25}}=3.35[/itex], which tells us that the new signal should be 3.35 sigma (standard deviations) away from the expected value to explain the observed events. In probabilities this translates to 99.958% signal exists/ 0.042% it's a statistical fluctuation, right?

My question is that the probabilities in those cases are taken from the standard normal distribution. Wouldn't (for so small expected events) the random variable (observed events) be taken out a Poisson distribution? In explicit wouldn't I have to calculate what's the probability of : [itex]P( x \ge 5 | \lambda =1.25 )\approx 0.9\%[/itex] (being statistical fluctuation) and the complementary for a signal? This would be relatively bad for the significance, as the standard deviation of a Poisson distributed variable is not equal to the square root of the events (for low expected events)...

Thanks.
Maybe this deserves to be moved in the statistics, but please do whatever you find fit.
 
Last edited:

Answers and Replies

  • #2
Orodruin
Staff Emeritus
Science Advisor
Homework Helper
Insights Author
Gold Member
16,817
6,626
What's the probability that you discovered new physics?
Ooops! Be careful here! Are you using frequentist or Bayesian statistics? In Bayesian statistics it depends on your prior. Frequentist statistics does not answer this question. It answers the question "what is the probability this (or something more extreme) would happen by chance?" which is not the same question.

And yes, for a low number of events you need to use the Poisson distribution.
 
  • Like
Likes mfb and ChrisVer
  • #3
ChrisVer
Gold Member
3,337
440
In Bayesian statistics it depends on your prior.
I don't think I can use a prior for this? Except for saying that I expect (given the model) [itex]B+ \mu S[/itex] where [itex]\mu[/itex] is taken from a uniform prior, and calculating stuff like the 95-quantile for mu to claim a discovery or not (via credibility levels)... seeing eg how the likelihood works:
[itex]L( x_{obs} | B + \mu S ) = Poi(x_{obs} | B+\mu S) Uni(\mu) [/itex]
again I don't think it's easy to determine by simple calculations the ranges for mu or the results of its quantiles (would have to input those in a statistical program).

Frequentist statistics does not answer this question. It answers the question "what is the probability this (or something more extreme) would happen by chance?" which is not the same question.
I think this is done easier via calculations? I.e. it results in the way I calculated the Poisson probability (and also how partially I interpreted the result of it, being a stat fluctuation/happened by chance). Maybe I was wrong to move one step further and say that the complementary (99.1%) is the probability of having seen a signal.

And yes, for a low number of events you need to use the Poisson distribution.
yup, so the significance (as I defined it) for such cases wouldn't be a good indicator?
 
  • #5
34,449
10,563
Maybe I was wrong to move one step further and say that the complementary (99.1%) is the probability of having seen a signal.
Correct. Stay away from such an interpretation, there is no meaningful way to define "the probability that you saw a signal". "The probability of a background fluctuation that large" is the only meaningful number.
 
  • Like
Likes ChrisVer

Related Threads on Signal signficancies low expected events

  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
3
Views
3K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
5
Views
2K
Replies
1
Views
1K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
3
Views
793
  • Last Post
Replies
3
Views
2K
Replies
4
Views
2K
Top