Sharing real-time data between app's

[trogluddite]

Hi All,
As I've hinted at in a few other recent posts, I've been working on a way to share data in real time between different VST instances and different applications - and finally I have a little something to show you.
Be warned, it takes a little bit of setting up with these prototypes - I've just rebuilt from scratch following the bad news that the Ruby Win32API is rather broken at the moment. But that's been kind of cool - I just built my first ever C++ dll, so I'm rather pleased with myself.

First, here are the files that you sill need...

MappedFile.fsm

Contains the Flowstone export modules.

(267.01 KiB) Downloaded 1612 times

Mapped File Source.zip

All the other gubbins.

(10.02 KiB) Downloaded 1504 times

Unfortunately because of the way this works, there's no way to embed the .dll file within the schematic or exports - the apps that want to share data have to share the same copy of the .dll. So here's how to go about trying it out...

1) Unzip the zip file.
2) Put the "FS_MappedFile.dll" in a safe location, and make a note of where. (Ignore the other files - they're just the original C++ source code so that the 'C++ gurus' can have a bit of a laugh!)
3) Now open the Flowstone file.
4) At the top is a 'DLL location' string box - you need to put the full path to the .dll file into here. This then gets memorised by all the other nodules (would be easier with the 'Folder' primitives, but I really need to go and get some sleep!).
5) On the module "Shared Memory" click the 'open' button - a very large number should appear in the 'address' box.
6) Near the bottom are "Test RX" and "Test TX" - export both of these as EXEs.
7) Now open the two EXE's that you just exported and have a play (keep FS open too for even more, erm, "fun").
8) With any luck, you should see the bar-graphs in the receivers move with the transmitter's knobs - and also inside FlowStone.

Not the most exciting example, and a lot of refinement to do yet. The .dll will always need installing by anyone you share exports with, unfortunately - but if I understand how all this .dll stuff works, there's really no way to avoid that (the same would be true if it had been done the Ruby way).

But lots of scope for exciting things. If you look inside, you'll see that the data is accessed by the .dll using Ruby audio frames. Even my previous Win32API version was able to handle audio, and the C++ .dll should be faster still. Sharing audio would work properly only for plugins inside the same VST host, as there would have to be a shared audio clock - but for asynchronous data, you would be able to share between applications too, as in the example.

WARNING - do not try to put the modules into your toolbox! This crashes Flowstone - not sure why yet, but it's probably just my beginner's level C++ code doing something odd!

Lots to do yet, though.
I'd be glad to hear from anyone who can critique my C++ code, or is savvy about concurrency, mutexes and all that malarky, as I realise that different app's sharing will need a way to avoid tripping over each other - so I'm throwing this open to anyone who fancies contributing.

Happy Flowstoning
Trog.

[RJHollins]

no critique from me at this point ... but a big WOOOOOOOOYEAHHHHHH for this function !!!

I have a project idea that I've always wanted to do [actually there are several] .... but all were out of my programming reach. From only reading your post, Trog, you are tackling the most important, fundamental function to make this a realty. This is HUGE !!

.... have we told you how much we've all missed you !

I don't have the background to help programming wise, but I look forward to this thread as the GURU's huddle in and hope I can learn/understand some of this AND be able to implement. So cool !

Thanks TROG.

oh ... and as MaMaTROG would say .... 'get your rest'.

[Tzarls]

Nice work Trog... really nice comeback I must say!

Now, if you really want some (constructive) criticism:

- Use a hader file (.h) to declare you functions and the .cpp file to define them. Althought I don´t know the specifics of building a .dll file (and I know there must be some gotchas) except from the one used to make VSTs (and those come already made with the SDK) this will make your life easier and your code more reusable. Remember to use guards.

- Get rid of those macros! Most of them can be converted to functions, and since most of them are one-liners you can define them in the header file and have them inlined. That will have the same final effect you get when using macros without the traps that sometimes hide under those #defines.

- Concurrency.... met her some time ago. We had a rough beginning, but eventually learned to live together.... The biggest problem here will be trying to avoid to block the audio thread. Anyway, I recommend you use some existing library (free and opensource and cross platform if possible) that abstracts away the internal workings and gives you a simple interface to mutexes and semaphores and locked queues and the like. My suggestion would be the zthread library - a bit old and no longer maintained (AFAIK) but simple and stable enough (AFAIK), albeit a little bit undocumented.

- Finally....... c++? Are you sure you´re trying to stay away from the rabbit´s hole????

[trogluddite]

Thanks guys - good to hear that it is working OK after the mad scramble to replace the old Ruby way to do it!

And thanks for the tips, Tzarls.
Mostly this was a cut-and-paste job from MSDN examples and Malc's template file (and LOTS of trial and error to get the pointer casting right!). Your tips have given me some good leads where to look in my C++ tutorials etc. for the next step.

- Finally....... c++? Are you sure you´re trying to stay away from the rabbit´s hole????

He he - yes, it was a bit like old times last night. Like RJ, this is something I'm desperate to get working for a couple of projects, so I followed the old "pragmatic programming" principle - whatever the hell I have to do to make it work!
Oddly though, it's much easier than Ruby in some ways for this - translating the (always C) MSDN examples for the Windows API calls into Ruby is a nightmare as it has none of the same types/structures. In C++, just "#include <windows.h>" and every Win32 function just appears like magic (hate Visual Studio though - might have to change that!).

But today, the sun is beautiful outside, so time to finish my coffee and go out into nature and relieve the square eyes!

[digitalwhitebyte]

all nice, but everything will be immune to memory leaks?

[trogluddite]

all nice, but everything will be immune to memory leaks?

If I get my coding right! (and Malc too, I suppose).

The Mapped File that's used as the shared memory looks after itself, thankfully. It's a win32 API object managed by the OS, and keeps track of the handles to it that each process holds. Freeing up the local memory happens automatically as soon as the handle is closed (done by the "About to close" prim') - but the OS is also supposed to free the handle automatically if a process shuts down (normally or with error). The main shared area of memory then gets de-allocated by the OS once there are no longer any processes holding a handle.

Here's the Windows reference if you're interested - note that I'm doing it using the system page file method rather than a real HDD file. For a small enough mapping, this means there shouldn't be any HDD activity involved unless you're very short on physical memory - though I plan to make a 'real File' version too, so that there's the option to make the data persistent between sessions (one step at a time, though!).

Within the .dll, it's a little more tricky - you have to remember to delete any output Strings or Arrays before sending a new value, otherwise the old value becomes a 'zombie' that will eat memory - but that's just normal C++ stuff really.

At least it should be fairly simple to test if it does leak. It was almost impossible to tell in the old Ruby version I made because the Ruby garbage collector is so unpredictable - so using the .dll method should be a bit more robust in this respect..