Estimate of noise statistics within some error

[Squatchmichae]

This question concerns estimating the PDF of noise, based upon observations of a data stream consisting of noise embedded with transient signals. I would like to know if my Proposed Solution is a correct approach.

Suppose I have "long" stream of seismic data, consisting of noise, and with some occasional pulses of transient signals; we can make assumptions about the time-bandwidth product. I DO assume the that the time-width of each signal pulse is very small relative to the length of the data stream.

Objective: Estimate the noise in the data.

Question:
How much of my data stream can be signal in order for my estimate of the noise variance to be correct within Δσ?

Proposed Solution:
(1) Perform a hypothesis test H₀: pure noise, H₁: is a signal plus noise, with unknown variance.
(2) Form a maximum likelihood ratio.
(3) If the probability of a missed detection--the probability of choosing H₀ over H₁ when H₁ is true, is sufficiently large--then that means my data is mostly noise.
(4) How to estimate the error, or uncertainty Δσ?

Thanks! I am sure this is easy for you DSP types?

 

[marcusl]

A likelihood ratio test (LRT) is usually derived for known noise statistics, most commonly for additive white Gaussian noise (AWGN). Setting the threshold between H0 and H1 is done by evaluating what is essentially the Boltzmann distribution, that is, the probability that the observed energy is consistent with being generated by a Gaussian process. If you don't know the form of the noise, then you have a problem in trying to form/evaluate the LRT.

Can you provide more information and context? Is this a homework problem or a research topic?

 

[Squatchmichae]

Hi Marcusl,
Thanks for the response.

To answer your last question, my question is associated with research--I would like to ensure that I am getting a good estimate of noise of my seismic data, but I don't want to have to pre-filter microseismic events out first. That would require making a noise model of the data first.

I suppose I should have said the maximum likelihood ratio--so I would maximize the unknown pdf with respect to the unknown standard deviation. I DO assume the data is normal.

My primary objective is see estimate the noise variance, and determine how many seismic events (with some mean time-bandwidth product) can be in the data stream, before my variance estimate is incorrect by an amount delta-sigma.
Thanks!

 

[marcusl]

I still don't follow. Have you worked through the details of Bayesian detection of a signal in noise? (I can recommend textbooks if you'd like.) You assume the noise is normal but you don't know the variance. How do you evaluate the likelihood?

A different approach might be better. Do you know the spectrum of the seismic events? If so, then measure the noise power spectral density (PSD) in a region where that spectrum is zero. Since the noise is white, its PSD is constant so you've characterized it everywhere.

 

[alphysicist]

I appreciate your proposed solution and can see that it is a valid approach to estimating the noise in your data. However, I would suggest considering additional methods and techniques to ensure the accuracy of your estimate. Here are a few suggestions:

1. Use multiple hypothesis tests: Instead of just testing for the presence of a signal, consider performing multiple hypothesis tests to determine the specific characteristics of the signal (such as its amplitude or frequency). This can provide a more accurate estimate of the noise variance.

2. Use statistical techniques: Instead of relying solely on a hypothesis test, consider using statistical techniques such as regression analysis or time-series analysis to better understand the noise characteristics in your data.

3. Consider the impact of transient signals: While your assumption about the time-width of each signal pulse being small relative to the length of the data stream may be valid, it is important to also consider the impact of these signals on your noise estimate. For example, if the transient signals are particularly strong or occur frequently, they may significantly affect your estimate of the noise variance.

4. Validate your results: It is always important to validate your results using independent data or simulations. This can help ensure the accuracy of your estimate and identify any potential biases or errors in your approach.

In terms of estimating the error or uncertainty (Δσ) in your noise estimate, this can be done using statistical techniques such as confidence intervals or standard error calculations. Additionally, performing sensitivity analyses and testing the robustness of your results can also provide insights into the potential error in your estimate. Overall, I would recommend considering a combination of methods and techniques to ensure the accuracy of your noise estimate.