Calculator automatic rounding is screwing up my calculation.

Join the discussion
Ask a follow-up here, or get your own question answered by working scientists, mathematicians and engineers — people, not an autocomplete.
Real named experts · corrections over time · the nuance an AI answer skips
2 replies · 2K views
Lebombo
Messages
144
Reaction score
0
Hi,

When I type into my calculator 13.6/(1.000000001)
the calculator returns the answer 13.59999999

Then, when I type 13.6/(1.0000000001)
the calculator returns the answer 13.6.

Obviously, both 13.6/(1.0000000001) and 13.6/1 cannot equal 13.6,

so what's the deal with the calculator returning the wrong answer?
 
Mathematics news on Phys.org
The calculator has a finite precision for displaying digits, it rounds 13.6/(1.0000000001) = 13.59999999864 (approximately) to 13.6. Try subtracting 13.6 from the result, often calculators have internally one or two more digits than they show.
 
  • Like
Likes   Reactions: Lebombo