Eliminating noise from Excel Data

[Gmoney2487]

I am observing the Sun with a small radio telescope for a research project. I am measuring the intensity of the flux and comparing it to X-ray flux data from the NOAA. I am plotting radio flux intensity against time and i get a big "hump" in the middle of the day due to less atmospheric interference, I want to eliminate the "hump" but haven't been able to figure out how, I have tried linear, logarithmic, and exponential trendlines to try to get it out and have a nice smooth curve with no luck. The big problem is that the x-axis is time so means nothing in line equations!

I need to get this "hump out so I will have a nice smooth curve and be able to see spikes and be able to compare it with the X-ray data and start doing the real work!


Any tips or ideas on what to do?

 

[berkeman]

I am observing the Sun with a small radio telescope for a research project. I am measuring the intensity of the flux and comparing it to X-ray flux data from the NOAA. I am plotting radio flux intensity against time and i get a big "hump" in the middle of the day due to less atmospheric interference, I want to eliminate the "hump" but haven't been able to figure out how, I have tried linear, logarithmic, and exponential trendlines to try to get it out and have a nice smooth curve with no luck. The big problem is that the x-axis is time so means nothing in line equations!

I need to get this "hump out so I will have a nice smooth curve and be able to see spikes and be able to compare it with the X-ray data and start doing the real work!


Any tips or ideas on what to do?

Welcome to the PF. Can you just create a colunm of data that just has the hump in it, and subtract that out of the real data?

Can you post an example spreadsheet? (Disable any macros that you are using, though).

 

[fleem]

Yeah, what berkeman said. Use two dishes, one pointed slightly away from the sun and one pointed at the sun. Subtract the flux of the control dish from the dish pointing at the sun (but compensate accordingly for what solar flux does make it into the control dish).

 

[Gmoney2487]

The excel files are really big so I can't upload one but i can email it to you, I have tried just subtracting the hump but its not due to noise, the hump is there because the radio intensity of Sun increases throughout the day because has a higher altitude and therefore the telescope is looking though less atmosphere. I was thinking that maybe there is a an equation could describe the atmospheric interference and I could use that to get rid of the hump, but no luck with that yet.

I cannot use two telescopes because we don't have another one. although we are trying to collaborate with a neighboring University that has an SRT program so we could do interferometry

 

[berkeman]

the hump is there because the radio intensity of Sun increases throughout the day because has a higher altitude and therefore the telescope is looking though less atmosphere. I was thinking that maybe there is a an equation could describe the atmospheric interference and I could use that to get rid of the hump, but no luck with that yet.

Ah, that makes some sense. It's not noise, it's varying attenuation due to the thickness of the atmosphere that the signal has to travel through.

Why don't you post your attempt at modelling that thickness and the resulting attenuation -- maybe we can help guide your calculations...

 

[Gmoney2487]

The farthest I have gotten is getting the Air Mass ~ secx where x is your angle from the zenith but I don't know how I would incorporate that into a line equation, and also haven't been able to find any references on this anywhere.Which leads me to believe I am missing something simple because this experiment has been done before. if you got the email i sent you can see what i mean

 

[fleem]

the hump is there because the radio intensity of Sun increases throughout the day because has a higher altitude and therefore the telescope is looking though less atmosphere.

Oh. I thought you meant it had something to do with man-made interference. The problem here though, is that the composition of the atmosphere is different at different altitudes and different times of day. And a lot of it may have to do with the ionosphere.

 

[Lambduh]

I'm a complete astronomy noob so if some of this is completely obvious just tell me to go sit in the corner:) Sooo...

Why you want to get rid of the hump in the first place? If your signal is almost the exact same everyday then why not just make comparisons between the day to day radio intensities? How long are these spikes of noise that you're looking for? Seconds, hours, days? Because that in itself could make a difference in what you're looking for from the data.

I don't know enough about propegation of EM waves with those wavelengths but will environmental conditions such as cloud cover, humidity, etc affect how much they are attenuated by the time they reach you? Because if you're looking for a correlation between X-rays and radio waves but have multiple things affecting the intensity of you're radio waves it will be hard to compare them. What I'm thinking is that if the attenuation constants of the atmosphere change at all you won't know what's causing the change in your signal level although it will probably be more of an offset than a spike. /shrug

 

[Gmoney2487]

I want to get rid of the hump because then I will be able to see smaller radio bursts and it will obviously make clear between interfrence and actual solar emission

 

[berkeman]

I want to get rid of the hump because then I will be able to see smaller radio bursts and it will obviously make clear between interfrence and actual solar emission

No, you need to understand data filtering and data collection better before you assume that there is a simple way to process your data. After all, that's part of your project, right?

Please post your thoughts on noise reduction in your thesis...

 

[maverick_starstrider]

If you can mathematically determine the increased flux due to time of day (by considering your position on the Earth and the projection of the sun's rays on the current orientation of the earth, i.e. season) then you'll have a template to subtract from your signal. However, subtracting a hump just because it's a hump without being able to identify a priori what it is and what its form will be is really just bad science.

 

[Gmoney2487]

I haven't done anything on noise reduction, the SRT software calibrates and filters out most of the noise so you get good data. My problem is not from terrestrial noise but the from the Sun's position in the sky. When the Sun is lower on the horizon I am looking through more atmosphere therefore less radio waves get through to the telescope. In the middle of the day when the Sun is at its' highest altitude the telescope is looking through less atmosphere and therefore more radio flux is getting through to the telescope. I want to account for the change in intensity due the altitude of the Sun so I can see just the radio intensity due to the Sun's brightness temperature.

The best way I have come up with is pick some flat parts of the graph and subtract get the linear equation and subtract them from the data points but that hasn't gotten rid of the "hump".

As far as actual filtering I can't do anything about it with the equipment I have.

 

[Redbelly98]

Is the radio signal at essentially one frequency? If so, you could model the absorption as a simple exponential decay function of atmosphere thickness:

s_detect = s_actual e^-γL​

Where L=L₀ secθ is the amount of atmosphere the waves travel through, and γ is a decay rate which you could get by doing a least-squares regression fit to your data.

Note, if the radio spectrum is broad, this may not work as γ would be different for different parts of the spectrum.

 

[mordechai9]

As I understand it, you want a curve which is more or less flat, and just shows spikes when the sun's intensity is momentarily flaring up. You don't care about the overall radio intensity, just the little deviations which occur over the course of the day. If this is the case, here's what you want to do.

1.) Compute the average intensity that you have over the entire day. If you plotted this as a straight line on your graph, it should sit somewhere in the middle, slicing the bell into top and bottom halves.

2.) Use your program or manually get a curve fit formula for your data. I'm guessing your program package has a feature which generates a smooth curve fit to the data. You can usually generate a polynomial curve fit which will nicely approximate the behavior, and increase the accuracy by using a higher order polynomial, which has more terms. For example, a polynomial a + bx + cx^2 will give you a better approximation than a polynomial a +bx, and so on. With sufficiently many terms this should look like a line on your graph which closely mimics the overall shape of your intensity vs. time. Call this plot A(t), for the approximate intensity versus time.

3.) Now, for every original data point (intensity and time), subtract the corresponding value of A(t). You should be able to get a program or a spreadsheet to do this automatically. Save this data, and then once you obtain this new curve, add the average intensity value calculated in step (1.) to every data point. By doing this, you rescale the curve so that it sits at a an average intensity value.

Now, you should wind up with a curve that just shows the deviation of the intensity from average. Is this what you were looking for?

 

[TooFastTim]

You could try using a sliding window filter. SWF are essentially finite impulse response filters and are pretty good at removing unwanted noise. Alternatively you could try a Kalmann filter.