18 min read
Floating-Point Numbers
Computers and calculators store numbers in a fixed, limited format, so they cannot represent every decimal exactly.
Why Computers Round Too
A calculator screen has a fixed number of digits. A computer program stores numbers in a fixed amount of memory. Neither can hold infinitely many digits, so any number that would need infinite digits gets cut off or rounded somewhere.
This does not only affect "awkward" numbers like . Computers store numbers in binary (base ), and just like never ends exactly in base , some ordinary decimals — including — never end exactly in base . This can cause tiny, surprising errors.
Example 1 — a well-known floating-point surprise
In many computer programs, typing does not display as exactly — it may show something like .
- and cannot be stored exactly in the computer's binary format
- Each is stored as an extremely close, but not perfectly exact, approximation
- Adding the two tiny approximation errors together produces a visible, tiny extra digit
This is not a bug in the program — it is a direct result of how computers store decimal numbers.
Practice
More lessons in Numerical Precision, Accuracy and Errors · Next: Error in Real Problems · Previous: Reporting Appropriate Precision