- #1

CRGreathouse

Science Advisor

Homework Helper

- 2,820

- 0

## Main Question or Discussion Point

Warning: I've only taken one stats class, back as an undergrad (though it was a very fast-paced class designed for mathematicians). My understanding of all things statistical is consequently weak.

I'm trying to design a program to accurately time functions. The functions themselves are of no importance here, only the timing code.

At the moment my program runs the test suite (10 million runs) with an empty function, to measure overhead. It stores a fixed number of runs, 7 at the moment, then computes the average and standard deviation of the overhead. This lets me construct a 95% confidence interval for the overhead:

[tex][\mu-1.96\sigma,\mu+1.96\sigma][/tex]

Simple enough so far, yes? So then I time each actual function once. (I don't want to ruin them multiple times because the real functions, as expected, take a fair bit longer than the empty function.) At this point I make the assumption that the distribution of the timing errors of the functions is the same as that of the overhead function (which seems reasonable to me). This gives me a 95% confidence interval (under my assumption) as such:

[tex][t(1-1.96\sigma/\mu), t(1+1.96\sigma/\mu)][/tex]

Here's the part I want help on. I combine the intervals by taking the low-end estimate for the function's speed and subtracting the high-end estimate for the overhead, to the high-end estimate for the function minus the low-end estimate for the overhead. How do I describe my confidence that this is correct? Less than 95% (errors can accumulate), more than 95% (errors are likely to cancel, maybe like sqrt(2) rather than 2?), or just 95%? Is there a better way to calculate this? Have I made mistakes or bad assumptions in my analysis?

I'm trying to design a program to accurately time functions. The functions themselves are of no importance here, only the timing code.

At the moment my program runs the test suite (10 million runs) with an empty function, to measure overhead. It stores a fixed number of runs, 7 at the moment, then computes the average and standard deviation of the overhead. This lets me construct a 95% confidence interval for the overhead:

[tex][\mu-1.96\sigma,\mu+1.96\sigma][/tex]

Simple enough so far, yes? So then I time each actual function once. (I don't want to ruin them multiple times because the real functions, as expected, take a fair bit longer than the empty function.) At this point I make the assumption that the distribution of the timing errors of the functions is the same as that of the overhead function (which seems reasonable to me). This gives me a 95% confidence interval (under my assumption) as such:

[tex][t(1-1.96\sigma/\mu), t(1+1.96\sigma/\mu)][/tex]

Here's the part I want help on. I combine the intervals by taking the low-end estimate for the function's speed and subtracting the high-end estimate for the overhead, to the high-end estimate for the function minus the low-end estimate for the overhead. How do I describe my confidence that this is correct? Less than 95% (errors can accumulate), more than 95% (errors are likely to cancel, maybe like sqrt(2) rather than 2?), or just 95%? Is there a better way to calculate this? Have I made mistakes or bad assumptions in my analysis?