Why is Excel telling me the GCD of 1.13*100 and 100 is 4?

  • Thread starter Thread starter SamRoss
  • Start date Start date
  • Tags Tags
    Excel Gcd
Click For Summary

Discussion Overview

The discussion revolves around the behavior of the GCD function in Microsoft Excel when applied to floating point numbers versus integers. Participants explore why the formula =GCD(1.13*100,100) yields a different result than =GCD(113,100) and seek to understand the underlying reasons for this discrepancy.

Discussion Character

  • Technical explanation, Conceptual clarification, Debate/contested

Main Points Raised

  • One participant notes that the GCD function returns 1 for integers but 4 for the floating point calculation, suggesting a difference in how Excel treats these values.
  • Another participant proposes that the issue arises because 1.13*100 is treated as a float, which is not equivalent to the integer 113.
  • A later reply confirms that using the int() function on 1.13*100 resolves the issue, indicating that the truncation of the floating point number affects the GCD calculation.
  • One participant explains that due to floating point representation, 1.13*100 is slightly less than 113, leading to it being truncated to 112, which results in GCD(112, 100) = 4.
  • Additional details are provided regarding the binary representation of floating point numbers, linking to external resources for further exploration.

Areas of Agreement / Disagreement

Participants generally agree on the explanation involving floating point representation and truncation, but there is no consensus on the implications or broader understanding of the GCD function's behavior in this context.

Contextual Notes

Limitations include the dependence on the specifics of floating point arithmetic and how Excel implements the GCD function, which may not be universally applicable across different contexts or software.

SamRoss
Gold Member
Messages
256
Reaction score
36
In Microsoft Excel, if I type in the formula =GCD(113,100) then it gives me the correct answer of 1. However, if I type in =GCD(1.13*100,100), which means the same thing, it tells me 4. What's going on and how can I fix it? Thanks
 
Technology news on Phys.org
I think it’s due to the 1.13*100 being treated as a float ie it’s not the same as the integer value 113.

You could test this by using the int() function on 1.13*100 to see if that fixes it.
 
  • Like
Likes   Reactions: WWGD and sysprog
jedishrfu said:
I think it’s due to the 1.13*100 being treated as a float ie it’s not the same as the integer value 113.

You could test this by using the int() function on 1.13*100 to see if that fixes it.
That fixed it. Thanks!
 
SamRoss said:
In Microsoft Excel, if I type in the formula =GCD(113,100) then it gives me the correct answer of 1. However, if I type in =GCD(1.13*100,100), which means the same thing, it tells me 4. What's going on and how can I fix it? Thanks
Here's the documentation for the gcd function in Excel:
GCD(number1, [number2], ...)
The GCD function syntax has the following arguments:
  • Number1, number2, ... Number1 is required, subsequent numbers are optional. 1 to 255 values. If any value is not an integer, it is truncated.
Due to the way that floating point numbers are stored, the value 1.13*100 turns out to be a little less than 113. This is true of computers in general, and isn't some bug in Excel. You can verify this by the formula
Code:
-(1.13*100 - 113)*10^6
The actual difference between 1.13*100 and 113 is so small that it is displayed as 0, but if you multiply this difference by, say, 10^6, you will get -1.42109*10^(-8).
Since 1.13*100 < 113, according to the docs above, it is truncated, leaving the integer value 112. Certainly gcd(112, 100) = 4
 
Last edited:
  • Like
Likes   Reactions: WWGD, SamRoss, Ibix and 1 other person

Similar threads

Replies
27
Views
4K
  • · Replies 27 ·
Replies
27
Views
5K
  • · Replies 3 ·
Replies
3
Views
3K
  • · Replies 7 ·
Replies
7
Views
2K
  • · Replies 7 ·
Replies
7
Views
4K
  • · Replies 5 ·
Replies
5
Views
3K
  • · Replies 20 ·
Replies
20
Views
3K
  • · Replies 22 ·
Replies
22
Views
4K
  • · Replies 5 ·
Replies
5
Views
6K
  • · Replies 3 ·
Replies
3
Views
4K