About value inaccuracy

[martinvicanek]

Oh, I see, the manual says it can only store floats, float arrays or strings. Hm, you could use strings to encode integers. Or use floats that have an exact machine representation so you can convert them to integers. If you can get away with integers in a range 0...127 (as your example seems to indicate) it should be doable.

[Tronic]

example:
@in_float.round(8)

output 100.10212/127.0
(@ins[0].round(8)*127.0).round(5) # >> 100.10212

[martinvicanek]

Divide by 128 instead?

[tulamide]

Hey Martin,

yes, 128 was one of the "low_bit_count"-dividers I used and it worked like a charme. But, as I said, for 128 values that results in a range of 0.0078125 - 1.0.
These normalized float values are used later on, and they need to be in the range 0.0-1.0, and using another normalizing brings up the "too-many-bits-for-single-precision" issue again.


Hey tronic,

I'm not sure if it will work, but I will give it a try!

EDIT: No, it doesn't work. It just leads to certain integer values not recognized at all. They are just left out. For example 32, 47, 49, 58, etc.

[KG_is_back]

This method allows you to store up to 29bit unsigned integer as a float value in 0-1 range. Values lower than 2^24 will be stored as denormals, so it is a thing to consider. It may not be optimal solution, but it works...
In case you need to store more tan 29bits you have to split them into 29bit segments.

Code: Select all

def i_to_f01(n)
   a=(n&(2**29-1)).to_s(2)
   a.insert(-24,"0")
   watch "bin",a+" "+a.length.to_s
   a=a.to_i(2)
   [a].pack("L").unpack("f")[0]
end

def f01_to_i(n)
   a=[n].pack("f").unpack("L")[0]
   a=a.to_s(2)
   a[-24]=""
   a.to_i(2)
end

EDIT: the highest value actually isn't 1 but 0.9999999403953552

[Tronic]

Hey tronic,

I'm not sure if it will work, but I will give it a try!

EDIT: No, it doesn't work. It just leads to certain integer values not recognized at all. They are just left out. For example 32, 47, 49, 58, etc.

my solution not work for you?

[tulamide]

@Tronic
Unfortunately it didn't work. As I described, some integers are simply not calculated.

@KG
That's a very interesting bitpacker. I will see if I can implement it.

@all
Many thanks for all your ideas so far! Don't hesitate to post if you came up with another!

[Tronic]

@Tronic
Unfortunately it didn't work. As I described, some integers are simply not calculated.

mmm
Please, can you show me an not working example?

EDIT: OK, got it, not need anymore, sorry.

[TheOm]

You can just directly store an unsigned integer in the range [0, 1065353216] as a float and it will be between 0.0 and 1.0.
Conveniently, this is also the maximum possible number of values that you can store in a single precision float from 0.0 to 1.0 ((2^7 - 1) * 2^23 + 1).
Here is a ruby implementation for signed(range [-532676608, 532676608]) and unsigned ints:

ruby float int.fsm

(630 Bytes) Downloaded 924 times

I have also tested the correctness of this approach for all possible values.

[tulamide]

I decided to go with the solution provided by KG and TheOm. It does not fulfill all criteria (the range 0.0-1.0 is fixed to 0-1065353216, but I need a smaller integer range), but at least the integers will keep intact and can be saved via preset, and calculating a new float range from integers is not too cpu heavy and can be done on request.

Thanks for all the help! I hope this also showed the issue and that it is more complex than thought.