Round off error

[vnorberg]

Hello,

I have a question whether there is a limit for numbers before Umberto is rounding?
After implementing a similar function, I all of the sudden receive negative round off error instead of 0:

E.g.: (the expression is based on different functions and variables and is therefore not able to just delete the factor, since it is being used at other places)

Exp.1 = (70/1.2-70/1.2*10000000/10000000)*10000000 = 0

Exp.2 = (70/1.2-70/1.2*10000001/10000001)*10000000 = -7.105427357601E-08

How come and how can I avoid this problem?


Thank you in advance

[mmaeusbacher]

Hello vnorberg,

this is a well known problem called floating point inaccuracy and affects a lot of software products. This is due to the fact that Umberto is calculating with binary numbers: To calculate the formulas in your example the decimal numbers have to be converted to binary numbers and after the calculation the result has to be converted back to a decimal number. This may result in fragments like in your second example.

There is a way to circumvent this problem programmatically, but it takes a lot of memory and is much slower, especially when dealing with millions of numbers like in Umberto. Excel, for example, works the same way and you get the same results when calculating your formulas in Excel.

I´m afraid there´s no general solution for this problem. Maybe you can rearrange your formulas so the fragments do not occur, e.g. if you change the second formula to:

70/1.2*10000000-70/1.2/10000001*10000001*10000000

you will get the correct result (= 0)