Optimizing Round-Off Accuracy in Source Spectrum Calculations

[DEvens]

TL;DR

Quirk of calculation produced a consistent round-off value over man data entries

So the sequences was the following.

Data was provided for a parameter in a standardized situation. This was a source spectrum for a total volume of material.

We needed the source spectrum per cubic meter. There are 100,016 cubic meters in the total volume. Simple division.

So then I come along and compare what was calculated to what the other analyst put in the file. And, since we only keep 3 digits (the stats will erase anything more) I am expecting there to be small differences. So I copy-paste the data into a spreadsheet, take the difference, and divide by the original value.

And every single fractional difference is 1.6E-4. Forty values in a row. Hmm... Hmmm... Oh yes. The original data for the total volume is 3 digits. So the terminal digit is always zero. So the way roundoff works, that divide-by 100,016, then the round-off bumps it up by 1.6E-4 every time.

 

[jedishrfu]

Round off leads to chaos as detailed in the story of its discovery. In particular, listen to the story of Ed Lorenz and how round-off changed his weather model predictions (at 4:45 mark)