Support

If you have a problem or need to report a bug please email : support@dsprobotics.com

There are 3 sections to this support area:

DOWNLOADS: access to product manuals, support files and drivers

HELP & INFORMATION: tutorials and example files for learning or finding pre-made modules for your projects

USER FORUMS: meet with other users and exchange ideas, you can also get help and assistance here

NEW REGISTRATIONS - please contact us if you wish to register on the forum

best way to obtain global presets?

DSP related issues, mathematics, processing and techniques

best way to obtain global presets?

Postby wlangfor@uoguelph.ca » Sun Jan 12, 2020 4:37 pm

I was curious if there were any tried and tested methods better than just CSV using .txt files for global preset data. In My own usage it covers size of plugin, help options, ad popup toggle and so on.

Let Me know please if anyone has made or found something that I could use for this please :).
My youtube channel: DSPplug
My Websites: www.dspplug.com KVRaudio flowstone products
User avatar
wlangfor@uoguelph.ca
 
Posts: 912
Joined: Tue Apr 03, 2018 5:50 pm
Location: North Bay, Ontario, Canada

Re: best way to obtain global presets?

Postby deraudrl » Sun Jan 12, 2020 5:30 pm

Some subset of XML maybe? I've seen some tiny parsers in C/C++, have to believe there's one in Ruby floating around somewhere... here's a place to start:
http://www.ohler.com/dev/xml_with_ruby/xml_with_ruby.html
I keep a pair of oven mitts next to my computer so I don't get a concussion from slapping my forehead while I'm reading the responses to my questions.
deraudrl
 
Posts: 236
Joined: Thu Nov 28, 2019 9:12 pm
Location: SoCal

Re: best way to obtain global presets?

Postby trogluddite » Sun Jan 12, 2020 6:58 pm

The main problem with anything other than simple CSV is that it's not easy to write a fast parser in FS. The .ini file format isn't too bad (very much like the VST preset text format), but anything much more complex (e.g. needing regular expressions etc.) can get greedy very quickly.

I had a look at the Ruby XML parser libraries in the above link and, as far as I can tell, they all rely on compiled C++ extensions or .dlls. This makes them fast, but very likely to be incompatible with FS, as it insists on binaries compiled using Microsoft Visual Studio, which most Ruby extensions aren't (in the wild, Ruby is used mostly on servers running on UNIX based systems). They also couldn't be embedded within the schematic, so you'd have dependencies on external files which users would have to be careful to install correctly.

I have made an XML parser in native Ruby for FlowStone (link below), but the problem is that doing it that way is very slow for all but the smallest files, and it still involves a lot of Ruby code (though the example could be trimmed down to a much smaller sub-set of XML and have the search features removed).
XML Class V100 Beta001.fsm
(20.27 KiB) Downloaded 1286 times
All schematics/modules I post are free for all to use - but a credit is always polite!
Don't stagnate, mutate to create!
User avatar
trogluddite
 
Posts: 1727
Joined: Fri Oct 22, 2010 12:46 am
Location: Yorkshire, UK

Re: best way to obtain global presets?

Postby tulamide » Mon Jan 13, 2020 11:15 am

@trog

Could you maybe be more specific? What are we talking about if you say "very slow"? A second for a preset? A minute? And which part makes it slow, the parsing, the file access, etc? And at which point would you consider it being fast?

As you can see, I'm trying to find out on what the relatives slow/fast are based under these specific circumstances.

Also, presets are in fxp/fxb format in FS4, which is the standard for vst. So why introducing another format besides preset texts?
"There lies the dog buried" (German saying translated literally)
tulamide
 
Posts: 2686
Joined: Sat Jun 21, 2014 2:48 pm
Location: Germany

Re: best way to obtain global presets?

Postby trogluddite » Mon Jan 13, 2020 2:07 pm

@tulamide
As rough guide to speed, I'd say it's similar to the Ruby parser in RubyEdits. So, for a 'config' type file of only a few dozen tags, probably not too noticeable. When I tried it on something longer, like some SVG files I was trying to extract data from for GraphicsPaths, with maybe a few hundred elements in several nesting levels, it was slow enough that I even had to increase the Ruby timeout sometimes (though that was on a rather slower machine than I have now!) Of course, it may not be the most efficient code possible, or even the best I could write now, after many years more experience writing Ruby and Regexps.

The slow part is the regular expression matching for elements with arbitrary names which could occur at any arbitrary position in the input string - i.e. lots of any-length, any-place wild-cards. So it could be speeded up quite a bit for a format where the tag names are known in advance and the source is guaranteed to be well formed XML, maybe with limited nesting levels and a specific layout (e.g. every tag on a new line, etc.) But, as you say, once you insist on a more specific format, you might as well choose one that's the simplest possible to parse or is already an accepted standard, like .fxp for presets or .ini for config files. So I agree that there's not much point in re-inventing the wheel!
All schematics/modules I post are free for all to use - but a credit is always polite!
Don't stagnate, mutate to create!
User avatar
trogluddite
 
Posts: 1727
Joined: Fri Oct 22, 2010 12:46 am
Location: Yorkshire, UK

Re: best way to obtain global presets?

Postby tulamide » Mon Jan 13, 2020 4:07 pm

trogluddite wrote:@tulamide
As rough guide to speed, I'd say it's similar to the Ruby parser in RubyEdits. So, for a 'config' type file of only a few dozen tags, probably not too noticeable. When I tried it on something longer, like some SVG files I was trying to extract data from for GraphicsPaths, with maybe a few hundred elements in several nesting levels, it was slow enough that I even had to increase the Ruby timeout sometimes (though that was on a rather slower machine than I have now!) Of course, it may not be the most efficient code possible, or even the best I could write now, after many years more experience writing Ruby and Regexps.

The slow part is the regular expression matching for elements with arbitrary names which could occur at any arbitrary position in the input string - i.e. lots of any-length, any-place wild-cards. So it could be speeded up quite a bit for a format where the tag names are known in advance and the source is guaranteed to be well formed XML, maybe with limited nesting levels and a specific layout (e.g. every tag on a new line, etc.) But, as you say, once you insist on a more specific format, you might as well choose one that's the simplest possible to parse or is already an accepted standard, like .fxp for presets or .ini for config files. So I agree that there's not much point in re-inventing the wheel!

Thanks!
I wonder why nowadays everything has to be human-readable, as it seems? My own fileformats often are binary, inspired by RIFF, and therefore don't need to save all the hr overhead, can rely on a defined structure (parser = minimum effort) and are quite fast in the meaning of this discussion.

As we both said, it doesn't really need another format, but I'm curious why people made the shift to hr files? I never saw any human actually trying to read an extensive vector graphic "manually".
"There lies the dog buried" (German saying translated literally)
tulamide
 
Posts: 2686
Joined: Sat Jun 21, 2014 2:48 pm
Location: Germany

Re: best way to obtain global presets?

Postby trogluddite » Mon Jan 13, 2020 6:39 pm

@tulamide
Likewise; my attitude is that human-readable is only necessary if you also intend human-writeable. In many cases, I think it's a bit risky to reveal so much "implementation detail" in a support file - most users would be much less tempted to tinker with a binary, so they're less likely to break anything. If I need config files for an FS schematic, I usually take the lazy option and put data into a Ruby container that I can Marshal - no parser to write at all!

Now that I mentioned it, Ruby Marshal might be a good answer to LeAttol's original question for 'global' settings. So long as all the data can be sent to a RubyEdit and collected into an Array or Hash, using Marshal is a very easy way to handle it, and allows data to be nested in categories very easily:
Code: Select all
# To Save
File.write(file_path, Marshal.dump(data_container))

# To Load
data_container = Marshal.load(File.read(file_path))
All schematics/modules I post are free for all to use - but a credit is always polite!
Don't stagnate, mutate to create!
User avatar
trogluddite
 
Posts: 1727
Joined: Fri Oct 22, 2010 12:46 am
Location: Yorkshire, UK

Re: best way to obtain global presets?

Postby wlangfor@uoguelph.ca » Tue Feb 04, 2020 4:19 pm

Mm, yes; I see the dillema. This subject is by all rights my forte as I used to work with php quite a bit and sql data furthermore.

Writing algorithms that do this sort of thing is pretty common.

I can't help to wonder though; the c parse object, it seems fast; I remember finding it in the mvdlee missing links and it screamed old gem when I first saw it. But with the lack of the old synthmaker forum it really comes down to a lot of guesswork when it comes down to second guessing it.

Has anyone had experience with the c parse primitive? It seems very useful dependent on its success.
My youtube channel: DSPplug
My Websites: www.dspplug.com KVRaudio flowstone products
User avatar
wlangfor@uoguelph.ca
 
Posts: 912
Joined: Tue Apr 03, 2018 5:50 pm
Location: North Bay, Ontario, Canada

Re: best way to obtain global presets?

Postby trogluddite » Tue Feb 04, 2020 6:04 pm

I must admit, I haven't used the custom parser ever since we got Ruby - Regexes are just so convenient! And AFAIK, no-one has ever tested them against each other for efficiency. My guess is that Regexes would probably be faster, as the Ruby Regex implementation "compiles" expressions into a kind of byte-code, so that effectively, they always run in C++ (so long as you're re-using the same ones, of course - it would be a lot slower if you're creating them on-the-fly). However, that may not be the case - just my hunch.

One advantage that Ruby does have is that you can read a file line-by-line instead of loading the whole file, which would be useful for very big files. But for something "small" like a preset/ini file, that's probably not so much of a factor.
All schematics/modules I post are free for all to use - but a credit is always polite!
Don't stagnate, mutate to create!
User avatar
trogluddite
 
Posts: 1727
Joined: Fri Oct 22, 2010 12:46 am
Location: Yorkshire, UK

Re: best way to obtain global presets?

Postby wlangfor@uoguelph.ca » Tue Feb 04, 2020 7:41 pm

trogluddite wrote:I must admit, I haven't used the custom parser ever since we got Ruby - Regexes are just so convenient! And AFAIK, no-one has ever tested them against each other for efficiency. My guess is that Regexes would probably be faster, as the Ruby Regex implementation "compiles" expressions into a kind of byte-code, so that effectively, they always run in C++ (so long as you're re-using the same ones, of course - it would be a lot slower if you're creating them on-the-fly). However, that may not be the case - just my hunch.

One advantage that Ruby does have is that you can read a file line-by-line instead of loading the whole file, which would be useful for very big files. But for something "small" like a preset/ini file, that's probably not so much of a factor.


Yeah, lately lol.. I have been literally making like three versions of a project just so I can test which one is more efficient than the other.

It's very un-efficient lol. Maybe I'll try and figure out the c parser, hmm. I think I can so I will try and post here regarding results.
My youtube channel: DSPplug
My Websites: www.dspplug.com KVRaudio flowstone products
User avatar
wlangfor@uoguelph.ca
 
Posts: 912
Joined: Tue Apr 03, 2018 5:50 pm
Location: North Bay, Ontario, Canada

Next

Return to DSP

Who is online

Users browsing this forum: Google [Bot] and 19 guests