- #1
- 1,207
- 464
- TL;DR Summary
- Quirk of calculation produced a consistent round-off value over man data entries
So the sequences was the following.
Data was provided for a parameter in a standardized situation. This was a source spectrum for a total volume of material.
We needed the source spectrum per cubic meter. There are 100,016 cubic meters in the total volume. Simple division.
So then I come along and compare what was calculated to what the other analyst put in the file. And, since we only keep 3 digits (the stats will erase anything more) I am expecting there to be small differences. So I copy-paste the data into a spreadsheet, take the difference, and divide by the original value.
And every single fractional difference is 1.6E-4. Forty values in a row. Hmm... Hmmm... Oh yes. The original data for the total volume is 3 digits. So the terminal digit is always zero. So the way roundoff works, that divide-by 100,016, then the round-off bumps it up by 1.6E-4 every time.
Data was provided for a parameter in a standardized situation. This was a source spectrum for a total volume of material.
We needed the source spectrum per cubic meter. There are 100,016 cubic meters in the total volume. Simple division.
So then I come along and compare what was calculated to what the other analyst put in the file. And, since we only keep 3 digits (the stats will erase anything more) I am expecting there to be small differences. So I copy-paste the data into a spreadsheet, take the difference, and divide by the original value.
And every single fractional difference is 1.6E-4. Forty values in a row. Hmm... Hmmm... Oh yes. The original data for the total volume is 3 digits. So the terminal digit is always zero. So the way roundoff works, that divide-by 100,016, then the round-off bumps it up by 1.6E-4 every time.