I Calculating noise in a data sample - what region to use

  • I
  • Thread starter Thread starter tappy
  • Start date Start date
  • Tags Tags
    Data Noise
AI Thread Summary
To calculate noise in a data sample, it is essential to determine whether to use the standard error (SE) over the entire dataset or separate regions, particularly when one region has significant signal and another is predominantly noise. The discussion highlights that the SE calculated over the whole profile may not accurately represent the noise characteristics, as the standard deviations differ significantly between the noisy and signal regions. It suggests averaging values from distinct segments, excluding transitional data, to provide a clearer representation of the noise. The noise may exhibit characteristics of a Poisson distribution, complicating its characterization. Ultimately, the approach to presenting error should reflect the nature of the data and the regions analyzed.
tappy
Messages
6
Reaction score
0
I have a data set of number of counts vs position where counts were detected. I want to find the noise in the sample. Am I right to think that by 'noise' the requestor wants to know the standard error (SE = stdev/sqrt(N)) where N is the sum of x-axis points.
Also if the above it true then is the SE calculated over the whole length of the data. The data acquired has almost zero (noisy) counts along the first half of the line profile (as expected) and then the signal kicks in later, so should SE be calculated separately on the noisy signal at the start and the real signal near the end or should SE be calculated over the whole data.
Thanks in advance
 
Physics news on Phys.org
Hi tappy, :welcome:

A bit more context, please: what's this about and can you post a picture of the measurements.
For some, noise is the background level, for others it's the sigma in that level, etc..
 
Last edited:
al at% vs wt %.png

The graph shows counts along a line profile.
I have calculated stdev & SE over the whole profile, over the part with signal and the part without signal so what bothers me is that if I use the SE over the whole profile as error in the data does it make sense as the SE in the part with signal and without signal is so much smaller in comparison.
For N=192, stddev is 17.7, SE = 1.3 but in the area with signal, stdev is 4.0, SE is 0.3 and in the area without signal stdev is 0.8, SE is <0.1.
Should I use the same SE over the whole data range or instead use the separate SE for each region.
Thanks.
 
Apparently there is some transition between 90 and 120, so I'd leave out that part in analyzing. If you expect the signal to be constant below 90 and above 120, you have four average values, each with a stdev and an SE.
tappy said:
I want to find the noise in the sample
The noise looks pretty Poisson like, so it seems reasonable to average a number of channels. Seems to me the noise in the sample as such isn't all that interesting. Don't you process the result and come with one answer like Al mass% minus Al at % for > 120 = soandso ? And you want the SE in that number ?

Or is it two answers, like: blue > 120 minus blue < 90 +/- ... and Orange > 120 minus orange < 90 +/- ...
 
Great thanks for that, leaving out the data between 90 and 120 makes sense but good to hear it from someone else. Indeed the noise isn't that interesting in this case however I suppose what got me worried was how should the error be presented, your suggestion of 4 average values is a good one and is how I will proceed with this.

Thanks for the help.
 
So generally if you have a measurement even if you measure the exact same thing repeatedly you don't get the same result. So the measured value is considered a random variable, X.

Generally, you like to consider the measurement to be some true value plus some noise, ##X=x+\epsilon##, where x is not a random variable, it is the true value being measured, and ##\epsilon## is a random variable called the noise. So finding the noise means to characterize that random variable, ##\epsilon##.

If your noise is unbiased and Gaussian then ##\epsilon=N(0,\sigma)## so the noise can be characterized with a single number, the standard deviation. Your noise does not look so simple, so it may take more effort to characterize it. It may be Poisson distributed.
 
I was reading documentation about the soundness and completeness of logic formal systems. Consider the following $$\vdash_S \phi$$ where ##S## is the proof-system making part the formal system and ##\phi## is a wff (well formed formula) of the formal language. Note the blank on left of the turnstile symbol ##\vdash_S##, as far as I can tell it actually represents the empty set. So what does it mean ? I guess it actually means ##\phi## is a theorem of the formal system, i.e. there is a...
Back
Top