Calculating noise in a data sample - what region to use

  • Context: Undergrad 
  • Thread starter Thread starter tappy
  • Start date Start date
  • Tags Tags
    Data Noise
Click For Summary

Discussion Overview

The discussion revolves around calculating noise in a data sample represented by counts versus position. Participants explore the appropriate method for determining standard error (SE) in the presence of varying signal levels across the data set, including considerations for different regions of the data where noise and signal characteristics differ.

Discussion Character

  • Exploratory
  • Technical explanation
  • Debate/contested
  • Mathematical reasoning

Main Points Raised

  • One participant suggests that noise may be defined as the standard error (SE) calculated as stdev/sqrt(N), questioning whether SE should be calculated over the entire data set or separately for regions with and without signal.
  • Another participant requests more context and clarification on the definition of noise, indicating that interpretations may vary among individuals.
  • A participant presents calculated values for standard deviation and SE across different regions, expressing concern about the appropriateness of using a single SE for the entire data set when values differ significantly between regions.
  • One participant proposes excluding a transition region in the analysis and suggests averaging values from distinct regions, raising questions about how to present error in the results.
  • Another participant acknowledges the suggestion to exclude certain data and agrees with the approach of using average values, while expressing concern about how to present the error associated with those averages.
  • A participant discusses the characterization of noise as a random variable, indicating that if the noise is Gaussian, it can be represented by a standard deviation, but notes that the noise in this case may not be simple and could be Poisson distributed.

Areas of Agreement / Disagreement

Participants express differing views on how to define and calculate noise and standard error in the data set. There is no consensus on whether to use a single SE for the entire data or separate SEs for different regions, and the discussion remains unresolved regarding the best approach to present error in the results.

Contextual Notes

Participants highlight the complexity of noise characterization, noting that the noise may not conform to simple distributions like Gaussian, and that assumptions about the data's behavior in different regions may affect calculations.

tappy
Messages
6
Reaction score
0
I have a data set of number of counts vs position where counts were detected. I want to find the noise in the sample. Am I right to think that by 'noise' the requestor wants to know the standard error (SE = stdev/sqrt(N)) where N is the sum of x-axis points.
Also if the above it true then is the SE calculated over the whole length of the data. The data acquired has almost zero (noisy) counts along the first half of the line profile (as expected) and then the signal kicks in later, so should SE be calculated separately on the noisy signal at the start and the real signal near the end or should SE be calculated over the whole data.
Thanks in advance
 
Physics news on Phys.org
Hi tappy, :welcome:

A bit more context, please: what's this about and can you post a picture of the measurements.
For some, noise is the background level, for others it's the sigma in that level, etc..
 
Last edited:
al at% vs wt %.png

The graph shows counts along a line profile.
I have calculated stdev & SE over the whole profile, over the part with signal and the part without signal so what bothers me is that if I use the SE over the whole profile as error in the data does it make sense as the SE in the part with signal and without signal is so much smaller in comparison.
For N=192, stddev is 17.7, SE = 1.3 but in the area with signal, stdev is 4.0, SE is 0.3 and in the area without signal stdev is 0.8, SE is <0.1.
Should I use the same SE over the whole data range or instead use the separate SE for each region.
Thanks.
 
Apparently there is some transition between 90 and 120, so I'd leave out that part in analyzing. If you expect the signal to be constant below 90 and above 120, you have four average values, each with a stdev and an SE.
tappy said:
I want to find the noise in the sample
The noise looks pretty Poisson like, so it seems reasonable to average a number of channels. Seems to me the noise in the sample as such isn't all that interesting. Don't you process the result and come with one answer like Al mass% minus Al at % for > 120 = soandso ? And you want the SE in that number ?

Or is it two answers, like: blue > 120 minus blue < 90 +/- ... and Orange > 120 minus orange < 90 +/- ...
 
Great thanks for that, leaving out the data between 90 and 120 makes sense but good to hear it from someone else. Indeed the noise isn't that interesting in this case however I suppose what got me worried was how should the error be presented, your suggestion of 4 average values is a good one and is how I will proceed with this.

Thanks for the help.
 
So generally if you have a measurement even if you measure the exact same thing repeatedly you don't get the same result. So the measured value is considered a random variable, X.

Generally, you like to consider the measurement to be some true value plus some noise, ##X=x+\epsilon##, where x is not a random variable, it is the true value being measured, and ##\epsilon## is a random variable called the noise. So finding the noise means to characterize that random variable, ##\epsilon##.

If your noise is unbiased and Gaussian then ##\epsilon=N(0,\sigma)## so the noise can be characterized with a single number, the standard deviation. Your noise does not look so simple, so it may take more effort to characterize it. It may be Poisson distributed.
 

Similar threads

  • · Replies 4 ·
Replies
4
Views
3K
  • · Replies 12 ·
Replies
12
Views
2K
Replies
28
Views
4K
  • · Replies 3 ·
Replies
3
Views
3K
  • · Replies 40 ·
2
Replies
40
Views
5K
  • · Replies 18 ·
Replies
18
Views
3K
  • · Replies 6 ·
Replies
6
Views
2K
  • · Replies 19 ·
Replies
19
Views
3K
Replies
17
Views
6K
  • · Replies 49 ·
2
Replies
49
Views
12K