rounding stream and the magic NR!?

[Nubeat7]

hi, as already worked out with troga little bit in my ppq stepposition thread, there are some questions about rounding in stream, spezially to round down to int, here i have 3 methods of how to do it and their behaviors,

the first one in both examples always works right but is cpu intensive, the most common method to subtract 0.5 works with 0.4999995! this is the real magic number trog told me while ago that it is 0.49999997!? so can there be a difference in different cpus? or did something changed in FS and why does 0.4999996 works with the number 8 but not with 9?

the second often used method is:
rounded = rndint(x)
rounded = rounded -1(rounded > x)

here you have to do a variable that it works right, like:
rnd = rndint(x)
rounded = rnd - 1 (rnd > x)

[MyCo]

This "magic number" isn't really magic, it's the highest possible 32bit floating point number below 0.5. Or in programmer terms it's the predecessor of 0.5

The exact value is:
dec.: 0.49999997
hex: 0x3EFFFFFF

The hex value of 0.5 is:
0x3F000000

As you can see:
0x3F000000 - 0x3EFFFFFF = 1


But this only works for very low values, for higher values, this type of rounding fails. Because the 32bit accuracy isn't enough.

BTW: If you want to use a really crazy way of rounding, try this code (it's my own invention ):

Code: Select all

streamin in;
streamout rounded;

rounded = in - 0.499999 + 8388611 - 8388611;

[Nubeat7]

nice one it works with integers till 32 when using bigger numbers it rounds one number down on odd numbers? this is also the case in my example before and 9 is not a big number and the 0.49999997 is also not working on 9.0 but it does with 8.0 (it works under 9 but from there it always rounds down on full odd numbers), why this is happening?

[MyCo]

It's very hard to explain. Basically the representation of the real result isn't possible in 32bit floating point number. So there is a rounding to the next possible number.

Some roundings are completely wrong, caused by the processor itself. eg:

Code: Select all

rounded = rndint(8.5);  // returns 8
rounded = rndint(6.5);  // returns 6