Author Topic: Round off error  (Read 16951 times)

Offline vnorberg

  • New Member
  • *
  • Posts: 5
    • View Profile
Round off error
« on: August 05, 2015, 14:01 »

I have a question whether there is a limit for numbers before Umberto is rounding?
After implementing a similar function, I all of the sudden receive negative round off error instead of 0:

E.g.: (the expression is based on different functions and variables and is therefore not able to just delete the factor, since it is being used at other places)

Exp.1 = (70/1.2-70/1.2*10000000/10000000)*10000000 = 0

Exp.2 = (70/1.2-70/1.2*10000001/10000001)*10000000 = -7.105427357601E-08

How come and how can I avoid this problem?

Thank you in advance

Offline mmaeusbacher

  • New Member
  • *
  • Posts: 4
    • View Profile
Re: Round off error
« Reply #1 on: August 06, 2015, 10:13 »
Hello vnorberg,

this is a well known problem called floating point inaccuracy and affects a lot of software products. This is due to the fact that Umberto is calculating with binary numbers: To calculate the formulas in your example the decimal numbers have to be converted to binary numbers and after the calculation the result has to be converted back to a decimal number. This may result in fragments like in your second example.

There is a way to circumvent this problem programmatically, but it takes a lot of memory and is much slower, especially when dealing with millions of numbers like in Umberto. Excel, for example, works the same way and you get the same results when calculating your formulas in Excel.

I´m afraid there´s no general solution for this problem. Maybe you can rearrange your formulas so the fragments do not occur, e.g. if you change the second formula to:


you will get the correct result (= 0)