1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

B Calculator automatic rounding is screwing up my calculation.

  1. Aug 7, 2016 #1
    Hi,

    When I type into my calculator 13.6/(1.000000001)
    the calculator returns the answer 13.59999999

    Then, when I type 13.6/(1.0000000001)
    the calculator returns the answer 13.6.

    Obviously, both 13.6/(1.0000000001) and 13.6/1 cannot equal 13.6,

    so what's the deal with the calculator returning the wrong answer?
     
  2. jcsd
  3. Aug 7, 2016 #2

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    The calculator has a finite precision for displaying digits, it rounds 13.6/(1.0000000001) = 13.59999999864 (approximately) to 13.6. Try subtracting 13.6 from the result, often calculators have internally one or two more digits than they show.
     
  4. Aug 8, 2016 #3
    Thank you
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted



Similar Discussions: Calculator automatic rounding is screwing up my calculation.
Loading...