# Why is Excel telling me the GCD of 1.13*100 and 100 is 4?

#### SamRoss

Gold Member
In Microsoft Excel, if I type in the formula =GCD(113,100) then it gives me the correct answer of 1. However, if I type in =GCD(1.13*100,100), which means the same thing, it tells me 4. What's going on and how can I fix it? Thanks

Related Programming and Computer Science News on Phys.org

#### jedishrfu

Mentor
I think it’s due to the 1.13*100 being treated as a float ie it’s not the same as the integer value 113.

You could test this by using the int() function on 1.13*100 to see if that fixes it.

• WWGD and sysprog

#### SamRoss

Gold Member
I think it’s due to the 1.13*100 being treated as a float ie it’s not the same as the integer value 113.

You could test this by using the int() function on 1.13*100 to see if that fixes it.
That fixed it. Thanks!

#### Mark44

Mentor
In Microsoft Excel, if I type in the formula =GCD(113,100) then it gives me the correct answer of 1. However, if I type in =GCD(1.13*100,100), which means the same thing, it tells me 4. What's going on and how can I fix it? Thanks
Here's the documentation for the gcd function in Excel:
GCD(number1, [number2], ...)
The GCD function syntax has the following arguments:
• Number1, number2, ... Number1 is required, subsequent numbers are optional. 1 to 255 values. If any value is not an integer, it is truncated.
Due to the way that floating point numbers are stored, the value 1.13*100 turns out to be a little less than 113. This is true of computers in general, and isn't some bug in Excel. You can verify this by the formula
Code:
-(1.13*100 - 113)*10^6
The actual difference between 1.13*100 and 113 is so small that it is displayed as 0, but if you multiply this difference by, say, 10^6, you will get -1.42109*10^(-8).
Since 1.13*100 < 113, according to the docs above, it is truncated, leaving the integer value 112. Certainly gcd(112, 100) = 4

Last edited:

#### Tom.G

Science Advisor
For the messy details see:

above found with: https://www.google.com/search?&q=convert+0.13+to+binary

### Want to reply to this thread?

"Why is Excel telling me the GCD of 1.13*100 and 100 is 4?"

### Physics Forums Values

We Value Quality
• Topics based on mainstream science
• Proper English grammar and spelling
We Value Civility
• Positive and compassionate attitudes
• Patience while debating
We Value Productivity
• Disciplined to remain on-topic
• Recognition of own weaknesses
• Solo and co-op problem solving