Engineering Fourier transform when the data is lacking datapoints

[arcTomato]

Homework Statement

I would like to know the equation of Fourier transform when the data has lack.

Relevant Equations

Fourier transform

I would like to know the equation of Fourier transform when the data has lack. like this sine wave.

 

[berkeman]

The answer to your question is pretty trivial, IMO. What do you think the answer is?

 

[arcTomato]

Hi @berkeman! Thank you.( you always help me)
I really don't know, but normal discrete Fourier transform works well?
\begin{eqnarray} a_j &=& \sum_{k}^{}x_k e^{ 2 \pi i j k/N} \end{eqnarray}

 

[Dr. Courtney]

https://arxiv.org/pdf/1507.01832.pdf

The method described in the above paper will work because it computes an explicit numerical integral from the data rather than an FFT (fast Fourier transform). FFTs require a number of equally spaced data points. What you need is an integration method that works in the case where the available data is not equally spaced. Lots of explicit numerical integration methods will do that.

 

[arcTomato]

Thank you! @Dr. Courtney !
Tha paper looks so useful for me. I will try it.

 

[arcTomato]

Hi @berkeman !
If you don't mind, I would like to know the continuation of our talk.

 

[berkeman]

Hi @berkeman !
If you don't mind, I would like to know the continuation of our talk.

Sure arcT. My point was that in this problem you clearly have high resolution sample data for a sine wave, and can fill in any missing datapoints with a simple curve fit. That will give you the continuous data that you need to do the straightforward FT on the data.

Now, if you had datapoints that did not so clearly indicate a simple waveform, then you would have a much more difficult problem...

 

[gneill]

Another approach is to use periodogram analysis. It searches for sinusoidal curves that fit the data which may have missing points or uneven sampling times, so essentially a least-squares spectral analysis. See, for example the Scargle Periodogram. Look for that as a launch point to other periodogram methods.

Periodograms are often used in analyzing astrophysical data where the sampling is interfered with by day/night or viewing conditions (weather).

 

[scottdave]

Do you have access to software (like Matlab) to calculate DFT from time series? Here is a link to Matlab's documentation for the FFT function. If you have the actual samples, and not just a picture, then you could actually do some calculations. https://www.mathworks.com/help/signal/ug/discrete-fourier-transform.html

 

[arcTomato]

Hi guys @berkeman , @scottdave , @gneill
Your helps are so useful.
I will study about Periodograms and use Matlab.
Thank you!

 

[scottdave]

This is a nice article, which explains a simple FFT analysis of a stream of samples in Scilab (a free Matlab alternative).
https://linuxhint.com/scilab_fft/

 

[arcTomato]

Hi @scottdave .
I didn't know this.
Thank you!

 

[scottdave]

Hi @scottdave .
I didn't know this.
Thank you!

How did it go? One thing about using the FFT in Matlab or Scilab (or Octave, another free alternative): you will need some sort of placeholder value where the missing data points are. If you just skip them, then everything will get "squished" in time domain. There are different methods for handling missing information. You could think of them as noise and so replace with a random number, or simply replace with the average of all points. There are other methods. Each has some pros and cons.

 

[boneh3ad]

Sure arcT. My point was that in this problem you clearly have high resolution sample data for a sine wave, and can fill in any missing datapoints with a simple curve fit. That will give you the continuous data that you need to do the straightforward FT on the data.

Now, if you had datapoints that did not so clearly indicate a simple waveform, then you would have a much more difficult problem...

The problem is that this only works if you have a time series that so very clearly has a form that is amenable to filling in the blanks. You can look at this one and say "gee whiz, that looks like a sine wave and I can probably do a curve fit to determine the missing points." Now what happens if your signal is two superposed sine waves? Three? Forty? Five thousand?

If you want a more general procedure, you need to use an integral method instead of an FFT as @Dr. Courtney suggested or else look into the nonuniform discrete Fourier transform.

 

[arcTomato]

Hi @scottdave ,@boneh3ad
I am trying to use Matlab now. (I have not used it ever)

so I understand that I should study about the nonuniform discrete Fourier transform( or integral method).
(My goal is making the program of those methods in C.)
Thanks a lot.