Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

B Calculator automatic rounding is screwing up my calculation.

  1. Aug 7, 2016 #1
    Hi,

    When I type into my calculator 13.6/(1.000000001)
    the calculator returns the answer 13.59999999

    Then, when I type 13.6/(1.0000000001)
    the calculator returns the answer 13.6.

    Obviously, both 13.6/(1.0000000001) and 13.6/1 cannot equal 13.6,

    so what's the deal with the calculator returning the wrong answer?
     
  2. jcsd
  3. Aug 7, 2016 #2

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    The calculator has a finite precision for displaying digits, it rounds 13.6/(1.0000000001) = 13.59999999864 (approximately) to 13.6. Try subtracting 13.6 from the result, often calculators have internally one or two more digits than they show.
     
  4. Aug 8, 2016 #3
    Thank you
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted



Similar Discussions: Calculator automatic rounding is screwing up my calculation.
Loading...