- #1

- 12

- 0

any ideas?

- Thread starter SolStis
- Start date

- #1

- 12

- 0

any ideas?

- #2

HallsofIvy

Science Advisor

Homework Helper

- 41,833

- 956

The problem was that since 0.1= 1/10 so the denominator is NOT a power of 2, in binary that is a repeating "decimal". It cannot be stored exactly in memory so the arithmetic is not exact.

- #3

- 12

- 0

Thanks for the reply though

- #4

- 90

- 0

http://docs.sun.com/source/806-3568/ncg_goldberg.html" [Broken]

Last edited by a moderator:

- #5

- 299

- 1

Say you want to work to three decimal places, then multiply all the numbers by 1000 and store them as integers. That way you will get exact results.

- Last Post

- Replies
- 1

- Views
- 2K

- Replies
- 1

- Views
- 3K

- Last Post

- Replies
- 1

- Views
- 1K

- Replies
- 7

- Views
- 7K

- Last Post

- Replies
- 2

- Views
- 4K

- Last Post

- Replies
- 1

- Views
- 609

- Replies
- 1

- Views
- 1K

- Replies
- 5

- Views
- 47K

- Last Post

- Replies
- 14

- Views
- 4K