What to plot as the errors for a moving average plot of temperature?

[sian130]

I have been monitoring the temperature of a mirror surface placed outside at night. The temperature is measured at 10s intervals and so, as you can imagine, a plot of the data for one period is quite noisy. I have therefore decided to "smooth" the data by plotting a moving average over a ten-minute time period.

This is where I need some help. I need to plot some error bars. I've tried plotting the standard deviation of the data in the ten-minute time periods used for the moving averages, but it doesn't look right. Because the temperature has, in some cases, remained constant for up to two hours, I have ended up with almost no error bars except for little ovals of errors where the temperature changes significantly.

Would it be better practice to simply plot the uncertainty in the measurement caused by the apparatus instead? The temperature probe I am using has an uncertainty of +/-0.5 degrees. Or is there a way of combining the uncertainty of the measurement with the standard deviation? I just don't think it's correct to have large proportions of my graphs with "zero" errors, especially as I know that there is an uncertainty from the temperature probe.

Any advice, or suggestions for potentially useful books on the subject would be much appreciated.

 

[Stephen Tashi]

Unless you formulate a clear probability model for how the data is generated, it is meaningless to talk about "error bars". (It's also meaningless to talk about "error bars" unless you define what an "error bar" is supposed to indicate. For example, if the temperature at time 5200 secs is 21.2 C and I draw error bars at plus or minus 1.4 C around that point, what fact is that supposed to convey? )

Don't let the use of statistics lobotomize your knowledge of the physics of the situation. You didn't explain any physical reason that you decided to use a moving average. Is the moving average supposed to be a better indication of the actual temperature? Is there some physical reason that the mirror can't change temperature as fast as the raw data shows? Can you use the physical data of the situation to establish any bound on how fast the the actual temperature of the mirror can change?

Yes, it is possible to formulate a probability model that would incorporate the known statistics of the sensor errors. To make a good model of those errors can get complicated if the errors are stated as percentages and the magnitude of the measurements varies by an order of magnitude. It is also hard to make a good model of this if the error is almost always in the last significant digit of the sensor reading - meaning the last significant digit of the data to whatever base the raw sensor output gives, not necessarily in the last significant digit of the data once it has been processed to a base 10 number.

 

[sian130]

Thank you for your response.

Well I'd like to convey the uncertainty in the measurement at each point in the time period, which will be a combination of the uncertainty from the moving average calculation (i.e. the standard deviation) and also the uncertainty associated with the apparatus, which is quoted as being +/-0.5 deg C. This is what I want to convey by using "error bars".

Ideally I wouldn't have plotted a moving average, but the purpose of my graph is compare several different mirror surface temperatures on one graph, so in order to make a clearer comparison between surfaces, I decided to smooth the data to make the graph less noisy. The precise values of temperature every 10s or so are not important to me, rather I'm more interested in showing that, for example, one of the mirrors consistently hovers 3degC above all the others. So the moving average has no physical significance, but as the purpose of the graphs is to portray comparisons in the behaviour of several surfaces it is important that the graph is visually clear.

As for establishing how fast the temperature of the surface can change, I don't know, but I know that it can change by about half a degree or so in 10s! I have started to model the situation in a computer program but I haven't modeled heat transfer in anything smaller than 1 second intervals, as I'm more concerned with changes in temperature on a longer timescales.

How would I go about formulating a probability model? A typical temperature range of the surface is about 5degC (say, from 0degC to -5degC over the course of 12 hours). The temperature probes have an uncertainty as quoted by the manufacturer of +/-0.5degC, valid in the range -60degC to +60degC.

Thank you!

 

[Stephen Tashi]

Thank you for your response.

Well I'd like to convey the uncertainty in the measurement at each point in the time period, which will be a combination of the uncertainty from the moving average calculation (i.e. the standard deviation) and also the uncertainty associated with the apparatus, which is quoted as being +/-0.5 deg C. This is what I want to convey by using "error bars".

Let's see if I got that.

T_m(t_i) = measured temperature at time t_i
A_m(t_i) = \frac{ \sum_{k=i-n}^{i+n} T_m(t_i)}{2n+1} = moving average of measured temperatures at t_i
T(t_i) = actual temperature at time t_i
A(t_i) = \frac{\sum_{k=i-n}^{i+n} T(t_i)} {2n+1} = actual moving average temperature at time t_i.

We assume
T_m(t_i) = T(t_i) + E(t_i) where the "error" E(t_i) is a random variable (for each t_i).

We assume that E(t_i) = E_{moth}(t_i) + E_{meas}(t_i) where E_{moth}(t_i) is the "error" made by mother nature and E_{meas}(t_i) is the error made by our measurement process.

Let D(t_i) = A(t_i) - A_m(t_i). You want to estimate the standard deviation of D(t_i) ? You also want to express this standard deviation as a function of two other standard deviations - one due to mother nature and one due to measurement error?

so in order to make a clearer comparison between surfaces

How would I go about formulating a probability model?

Generally speaking you would make a model for the physical process using deterministic equations from physics plus some random variables representing "uncontrolled" physical inputs to these equations. The equations would have various unknown parameters. Then you would investigate whether it is possible to estimate the values of these parameters from the measured data and other known facts.

It is not clear to me whether the goal of the comparison "between surfaces" is intended to be precisely how their temperatures would change with the same inputs or whether the thing being compared is some property (like reflectivity, specific heat etc.) that is not identical to temperature but merely related to measured temperatures. A physical model might clarify what property of the mirror surface concerns you. For example, suppose I am interested in the position of an object and I have data on a varying Force that is applied to it. If I measure the statistics of the force (such as standard deviation) , I can't conclude that the statistics of the position are directly proportional to it. (F = ma, not F = mx). There is some "inertia" to heat transfer isn't there? So if the real quantity of interest is not temperature itself, the equations relating temperature to the quantity would be useful.

 

[sian130]

Let's see if I got that.

T_m(t_i) = measured temperature at time t_i
A_m(t_i) = \frac{ \sum_{k=i-n}^{i+n} T_m(t_i)}{2n+1} = moving average of measured temperatures at t_i
T(t_i) = actual temperature at time t_i
A(t_i) = \frac{\sum_{k=i-n}^{i+n} T(t_i)} {2n+1} = actual moving average temperature at time t_i.

We assume
T_m(t_i) = T(t_i) + E(t_i) where the "error" E(t_i) is a random variable (for each t_i).

We assume that E(t_i) = E_{moth}(t_i) + E_{meas}(t_i) where E_{moth}(t_i) is the "error" made by mother nature and E_{meas}(t_i) is the error made by our measurement process.

Let D(t_i) = A(t_i) - A_m(t_i). You want to estimate the standard deviation of D(t_i) ? You also want to express this standard deviation as a function of two other standard deviations - one due to mother nature and one due to measurement error?

Well, all I want is the value of the error, i.e. E(t_i) = E_{moth}(t_i) + E_{meas}(t_i) . In that case, can I simply add the standard deviation (which I am taking to be the error caused by mother nature, i.e. E_{moth}(t_i)) to the measurement error, E_{meas}(t_i) ?

It is not clear to me whether the goal of the comparison "between surfaces" is intended to be precisely how their temperatures would change with the same inputs or whether the thing being compared is some property (like reflectivity, specific heat etc.) that is not identical to temperature but merely related to measured temperatures.

I'm interested in how their temperatures would change with the same inputs, i.e. with the same weather conditions (e.g. ambient temperature and wind speed). My aim is to see which mirror surfaces are more likely to experience condensation on cold, clear nights. Therefore, if I know how the temperatures are likely to change, then I can predict condensation if the temperature falls below the dew point temperature on a given day. That's why I'm doing the experiment - I'm plotting the temperatures throughout the night and then trying to tweak my model until it simulates the observations. So far I have a model which allows me to vary emissivity, thermal conductivity, mass and area of the mirrors. But my original question just refers to the actual experimental data - I want to present my graphs with error bars which portray the uncertainties in the measurement.

 

[Stephen Tashi]

Well, all I want is the value of the error, i.e. E(t_i) = E_{moth}(t_i) + E_{meas}(t_i) . In that case, can I simply add the standard deviation (which I am taking to be the error caused by mother nature, i.e. E_{moth}(t_i)) to the measurement error, E_{meas}(t_i) ?

OK, but the standard deviation of the error E(t_i) isn't a direct indicator of how the moving average curve itself would vary if you could run the experiment over and over again under the same conditions.

You have a situation where it's almost irresistible to do certain arithmetic and this urge would drive most people to make the following assumptions.

1. Assume the E_{moth}(t_i) each have mean zero and are independent and identically distributed random variables for all t_i.

2. Assume E_{meas}(t_i) each have mean zero and are independent and identically distributed random variables for all t_i.

and we may have to make more assumptions , but the general idea is to use the deviations we can measure D_m(t_i) = T_m(t_i) - A_m(t_i) and argue that the variance of these deviations is approximately the variance of D(t_i) = T_m(t_i) - A(t_i).

Compute D_m for all t_i and consider this one big set of data for the same random variable D_m. Find the variance of D_m.

It is the variances of sums of independent random variables that can be computed as sums (not their standard deviations). So Var(D_m) = Var(D) = Var(E_{moth}) + Var(E_{meas}).

Then you have the problem of translating the measurement device specifications into a numerical value for Var(E_{meas}).

If you can do that, you can solve for Var(E_{moth}).


A person with very serious intentions would check things like whether D_m really appears to have mean zero and whether D_m(t_i) really is identically distributed for all times t_i. For example, do the devations tend to be positive when temperatures are falling?

My aim is to see which mirror surfaces are more likely to experience condensation on cold, clear nights.

That's a clear goal. It focuses attention on a certain range of temperatures. I assume your data is for temperatures in that range. But is the question of interest a yes-or-no question? Or is there a more-or-less aspect to the question of condensation?

trying to tweak my model until it simulates the observations. So far I have a model which allows me to vary emissivity, thermal conductivity, mass and area of the mirrors

According to physical laws, should the temperature-vs-environment behavior of the mirror change once some condensation does form on it? Is the formation of "some" condensation something that promotes even more condensation at the same temperature?

 

[sian130]

Thank you for your response again. I am working through this at the moment...please bear with me!