Shared Mem Wavetable Oscillator

[martinvicanek]

Hi guys,

I have been trying to build a wavetable oscillator based on Trog's shared mem concept. One problem is how to create and populate the wavetable mem so it can be used in a poly section. Tulamide has provided a nice Ruby solution, where memory is not only allocated for 16 byte aligned access. The same module also actually writes data from a green array into the mem. Only after the writing is completed is a valid address passed to an ASM module for mem access. That way crashes are eliminated (well, less likely ).

Below is a demo which uses linear interpolation and some SSE trickery for speed optimization. It turns out that there is actually room for two waveforms stored in one array. I am posting it eventhough it is half-baked, as there is similar activity in another thread.

[KG_is_back]

Very, very nice! I do not understand the "lock" hack

Perhaps it is not necessary to put 4 wavetables in one array. with clever bitwise logic it is possible to shuffle the correct value to all channels even when mem is not 4channel. However, the code is rather lengthy and I've not written it down (a lot of shuffling and I don't have the nerves for it right now xD ).

[tulamide]

I do not understand the "lock" hack

We are dealing with two parts that are independant from each other: Ruby and Assembler. Ruby doesn't know what Assembler does and vice-versa. Now, memory allocated by Ruby (whenever we use something, we allocate memory) will be controlled by Ruby. Therefore it is critical if we pass access to memory created with Ruby to Assembler.

Older languages used a pretty simple mechanism: The programmer allocates memory, uses it, and frees it after use. This can get pretty complicated for the programmer, should he/she forget to free memory. And in fact, it was often the reason for so called memory leaks.
Ruby goes another path. Pretty much every memory access is automated, so that the programmer doesn't have to care about it. For example, you just call "class.new" and Ruby takes care of allocating memory for the class structure. Since you don't have direct access to memory, another system is also needed to free memory. Ruby uses a system called "garbage collection".
Basically, whenever memory is allocated, information about it is put on a queue. Everything on the queue is checked regularly if still valid. If not, the memory is freed and the information removed from the queue. For example, if we did something like a = Array.new(100) and then somewhere later in the code do a = 2, then the array and its associated memory is no longer valid. Ruby then will free that memory. But memory that is freed will be used by the os or other applications, processes, and so on. But Assembler still works with that memory, so it would lead to issues and crashes.
By accessing the frame every once in a while from within Ruby the garbage collector will never get a chance to free the memory associated with the frame - which is what is needed to let Assembler work with that memory. Ruby acts as a memory protector for the asm module, so to say.

[KG_is_back]

Oh... so because the frame is regularly used by event method (in this case by the "watch" method inside it) it is protected from getting "expired" like a milk on a summer sun.

I thought objects in instance variables are protected from GC. Does this happen, because when rubyedit instance is not active for some time its instance variables may get scrapped as unnecessary?

[Tronic]

I do not understand the "lock" hack

We are dealing with two parts that are independant from each other: Ruby and Assembler. Ruby doesn't know what Assembler does and vice-versa. Now, memory allocated by Ruby (whenever we use something, we allocate memory) will be controlled by Ruby. Therefore it is critical if we pass access to memory created with Ruby to Assembler.

Older languages used a pretty simple mechanism: The programmer allocates memory, uses it, and frees it after use. This can get pretty complicated for the programmer, should he/she forget to free memory. And in fact, it was often the reason for so called memory leaks.
Ruby goes another path. Pretty much every memory access is automated, so that the programmer doesn't have to care about it. For example, you just call "class.new" and Ruby takes care of allocating memory for the class structure. Since you don't have direct access to memory, another system is also needed to free memory. Ruby uses a system called "garbage collection".
Basically, whenever memory is allocated, information about it is put on a queue. Everything on the queue is checked regularly if still valid. If not, the memory is freed and the information removed from the queue. For example, if we did something like a = Array.new(100) and then somewhere later in the code do a = 2, then the array and its associated memory is no longer valid. Ruby then will free that memory. But memory that is freed will be used by the os or other applications, processes, and so on. But Assembler still works with that memory, so it would lead to issues and crashes.
By accessing the frame every once in a while from within Ruby the garbage collector will never get a chance to free the memory associated with the frame - which is what is needed to let Assembler work with that memory. Ruby acts as a memory protector for the asm module, so to say.

not all true,
the problem of the crash is mainly due to how Flowstone manages memory and especially the case, here less discussed and considered, which is the relocation after the memory is freed.
to test it just to copy and paste some ruby modules that create frame, and you'll see that the memory is not managed by ruby, but by FlowStone, ruby only uses it.

[tulamide]

not all true,
the problem of the crash is mainly due to how Flowstone manages memory and especially the case, here less discussed and considered, which is the relocation after the memory is freed.
to test it just to copy and paste some ruby modules that create frame, and you'll see that the memory is not managed by ruby, but by FlowStone, ruby only uses it.

I know what you mean, because the behaviour you describe is exactly what I prevent with this hack

[Exo]

Very nice, thanks Martin and tulamide.

This is a solution when using frames.

I prefer to use the mem create with the mem to address, this is more stable. Check out my free running poly oscs this doesn't have any problem with garbage collection because we can create and delete the memory ourselves.

I do have one crashing issue regarding using the "mem to address" directly with a wavetable create and a wave file. I have a crash on load sometimes but I don't think it is related to this issue because I am not using Ruby. But I think I may have an invalid address while the wavetable or wavefile is loaded.

I like the fact that you are outputting zero while the frame is created, I think I will try that because it could solve my stability problem.

But I think the stability problem is also there when using the float array to mem, because the float array is copied to a mem so every time the float array updates the address changes.

[Exo]

This is a good solution for most cases though.

Anywhere I would use a "float array to mem" > "mem address" I could replace with this.

But this solution wouldn't be suitable in the case of the wavetable create primitive, because passing it to the Ruby component as a float array would be way to heavy.

The case I am using is a bandlimited wavetable (4096 sample points) that is 32MB in size, Ruby cannot handle that.

So very large arrays are not suitable. But smaller arrays this is perfect

[tulamide]

The case I am using is a bandlimited wavetable (4096 sample points) that is 32MB in size, Ruby cannot handle that.

Well, actually it can. But the time needed is just inacceptable (and the reason for Flowstone to interfear). You encounter the downside of the garbage collector. Before writing a wall of text, have a look at this blog for an explanation: http://www.platanus.cz/blog/working-wit ... es-in-ruby