About value inaccuracy

For general discussion related FlowStone
KG_is_back
Posts: 1196
Joined: Tue Oct 22, 2013 5:43 pm
Location: Slovakia

Re: About value inaccuracy

Post by KG_is_back »

Tulamide, I have one thing to mention. When you save preset with FS preset manager, float values are actually stored as 8digit decimals. This effectively means floats will get quantized to 0.0000001 steps (rounding anything lower than that to zero). It popped up when I tried saving filter coefficients (which are very sensitive to rounding)... It is probably the most serious issue with inaccuracy in flowstone.
tulamide
Posts: 2714
Joined: Sat Jun 21, 2014 2:48 pm
Location: Germany

Re: About value inaccuracy

Post by tulamide »

KG_is_back wrote:Tulamide, I have one thing to mention. When you save preset with FS preset manager, float values are actually stored as 8digit decimals. This effectively means floats will get quantized to 0.0000001 steps (rounding anything lower than that to zero). It popped up when I tried saving filter coefficients (which are very sensitive to rounding)... It is probably the most serious issue with inaccuracy in flowstone.

KG many thanks for mentioning that! We discovered such a fine difference as well, as the one I'm working for did the hard works of going through each seperate value and discovered that some of them work correctly when 0.0000001 added (within green). I then made the Ruby test and came to the result that I could simply add 0.0000001 to ALL values to get them to work.

EDIT: I'm talking of 8-bit-integers converted to floats in the range 0-1

Since I thought of a coincidence I didn't want to rely on it and decided to go with your proposals, but if this is a fix behaviour of the preset manager I might save all the time-wasting work of re-coding and just add 0.0000001!
"There lies the dog buried" (German saying translated literally)
Post Reply