Custom DSP Code 2 (comunity project)

[KG_is_back]

Hello Flowstone community!
Recently I've started a new Flowstone project that we all will hopefully benefit from, once it's finished. A new custom DSP code component/compiler. It will contain following features:

- all features currently present in stock DSP code component (some replaced with faster versions)
- MartinVicanek's stream math functions
- ability to declare and use your own functions within the code
- ability to declare functions in assembly
- proper IF/ELSE logic, FOR, WHILE and UNTIL loops (these will be useful only in mono and mono4)
- ability to use proper integers & some integer calculations
- ability to shuffle channels (you will not have to split your code into parts and use unpack-pack prims)
- new pointer input and output for processing mems and ruby frames like arrays inside of code (not copies)
- self-optimization

I already have first third of the compiler done and rest of it layered out in head, but I'm not very good at making GUI. If there is someone, who can help me out with it, I'd be very grateful.

Here's how I imagine the GUI. It will be split into 7 individual windows (preferably re-sizable and re-arrangeable when "toggle EditPanel" is on).
In first window inputs and outputs will be declared, including memin and pointer input/output.
In second window variables (and constants) will be declared. Individual SSE channels may be given different values like "float var=[3,5,7,9];" will fill channels 0-3 with respective values automatically. This will also work for arrays.
in third window you may declare functions. Functions will be placed inline when used, just like in stock DSP code. Assembler-declared functions will be added somewhere inside the module through ruby, cos' it's quite an advanced feature.
The rest four windows will serve for writing the actual code one for each stage.

The entire module will have one output for green text, which will give out the generated assembler code. In current version of flowstone you'd have to copy-paste it into a assembler component, but this feature would automate that. Hopefully, some update to assembler will be done in next update, possibly widening the possibilities even more.

Please, feel free to express your ideas, opinions and wishes Once I'll have a working prototype I will post it here and/or on FSguru...

[Exo]

Something like this would be epic!

Great Idea, would definitely need the input and help of the community to make this really great.

I cannot wait to see what you have done already

[Exo]

By the way what are you writing it in, Ruby?

[KG_is_back]

By the way what are you writing it in, Ruby?

Yop, Ruby... compiling/interpreting is basically handling strings and nested arrays and in ruby that is done quite easily. Here is a little insight on how it works in 4 steps:

0. make list of variables, functions and constants used in the code, or at least have "patterns" on how they look like.

1.split the string into individual code lines ( ended with ";" ) and then split each line into tokens (variable names, operators etc. ). Example:

Code: Select all

before:
"a=b*coef+3;
b=coef-3;"
after:
[ [ "a","=","b","*","coef","+","3"] , ["b","=","coef","-","3"] ]


2.build a parse tree. That means to split a line into basic operations and their arguments. Each argument may also be a basic operation with multiple arguments, creating a tree-like structure. In this step, precedence of operators has great weight. example: - first code line of the previous example

Code: Select all

#each operation is an array, starting with the operation name, followed by its arguments. It may be hard to visualize it in this array form:

["=", "a" , ["+", "3", ["*", "coef","b"]]]

parse tree diagram:
    =
   / \
  a   +
      / \
     3   *
         / \
        b   coef



3. replace the basic operations in the tree with their assembler "blueprints". First the most inner (bottom) operations are processed, results stored in registers and passed to higher operations as arguments, until entire line turns into assembler. example step by step:

Code: Select all

1.do the multiplication: ["*", "coef", "b" ]
movaps xmm0,coef;
mulps xmm0,b; //xmm0 contains the result of the multiplication
2. pass it to the addition ["+", "3", xmm0 ]
movaps xmm0,coef;
mulps xmm0,b;
movaps xmm1,xmm0; //xmm0 is now free again
addps xmm1,F3; //now xmm1 contains result of addition
3. now handle the assignment ["=", "a" , xmm0 ]
movaps xmm0,coef;
mulps xmm0,b;
movaps xmm1,xmm0;
addps xmm1,F3;
movaps xmm0,xmm1;
movaps a,xmm0; //now the codeline is completely finished



Operation may have multiple "blueprints" for cases when one or more arguments are constants. For example exponential function may give multiplication chain if exponent is integer... or square root if exponent is a 1/2.

In this step, it is crucial to write an algorithm that manages registers most efficiently. The operation itself may consist of multiple opcodes and may internally use intermediate registers too, but they get recycled once they are not needed. When you run out of registers you have to store intermediate results in temporary variables - stock DSP code can't do that and creates a bug where things like"xmm999" occur - you have to do it manually (or rearrange the code in a way that it doesn't happen).

4.optimization - the "blueprint" of each operation assumes that the input argument may be either variable or register. In cases when it's register, this results in numerous unnecessary movaps. They may be detected and removed (freeing more registers, which means also several temporary variables may be removed) or even completely avoided in previous step. The free registers may be used to store variables and constants that serve as arguments in multiple places. Stock DSP code practically doesn't do any of this. I'm pretty sure I can replicate it's behavior and improve upon it...

[kortezzzz]

Sound great, although I don't really understand the whole featureS list.
Anyway, I can't resist to challenge. I'll happily design it's GUI once I understand much deeply what it is all about.
So KG, PM me.

[Exo]

Very interesting and logical approach KG.

I'm thinking "if/else" could be supported for all channels but it would be slow, but maybe the user wouldn't mind.
For mono user could set a compiler flag such as "~mono" to tell compiler to treat ifs as mono.

I think your idea for 7 windows would be a bit confusing and overkill, is there a technical reason why that is better/ easier than one window?

For the GUI I am envisioning a proper code input in one window with syntax highlighting like Ruby. But this would be quite a lot of work I think.

Also it should be as easy as possible to add new code constructs such as "switch" ect or even invent totally new ones.

[tulamide]

It may sound silly, but after having read how it is organized, I instantly thought of visual dsp programming. What I mean is this:
People like me don't have a clue regarding the language and its use (Assembler). It is so abstract that it is very hard for me to learn. But, with your approach it would just be one more step to visually develop assembler modules. Instead of strings like "a * b", we could have a graphic "mono multiply" with ins and out. Behind the fancy graphic simply lies the "a * b" code, that you then build the assembler code from. The GUI/interface would be doable, and creating a filter would then be just a matter of "connecting" visual representations of underlying code. Just imagine Martin's stream math modules as visual components in a Flowstone-like environment...

[KG_is_back]

I'm thinking "if/else" could be supported for all channels but it would be slow, but maybe the user wouldn't mind.For mono user could set a compiler flag such as "~mono" to tell compiler to treat ifs as mono.

Yes, that is a good idea. But I think I have a better one... things like array index and if/else statements would accept both SSE arguments and normal 32bit arguments ( something like "IF (x==0):2" whould mean to do do the comparison of x and than pass second channel to IF statement). When the argument is a SSE variable, the condition will create two inverse bitmasks. The code is bypassed only if all values are false. When not all are true, the two inverse bitmasks would be used to update or not update value in each assignment accordingly. example:

Code: Select all

what will user write:
IF x==0
{
a=10+b*9
b= 11*a^7
}
what will compiler handle it as:
IF pMask=(x==0); nMask= !(pMask);
{
a=pMask&(10+b*9) + a&nMask
b= pMask&(11*a^7) + b&nMask
}
the ELSE statement will simply be handled as another IF statement and will receive "nMask" as an condition


I think your idea for 7 windows would be a bit confusing and overkill, is there a technical reason why that is better/ easier than one window?

Now that I think about it a second time, you're probably right - it would be very impractical. The reason why I thought it would be a good idea is, that the compiler first makes a list of variables, arrays, pointers and functions as a hash. Each variable/function name is a key to its declaration/assembler blueprint (in case of functions). Similar list exists for operations and assembly-declared functions. Also another reason why I wanted each stage to have separate window, is because they are compiled separately. I also having to write "stage(0){ }" but code outside is considered stage2 is a little bit confusing. In my mind it would be much more practical to have like one window with text editor with 4 tabs.

For the GUI I am envisioning a proper code input in one window with syntax highlighting like Ruby. But this would be quite a lot of work I think.

Well... All we really need is a reliable note-pad styled text editor, with ability to color pats of text. The above mentioned lists of funcs,vars and ops may be used as a keyword list to color-code the text.

[KG_is_back]

OK people, here is a working prototype. It's more like a proof of concept.

declaration and code body must be written separately. The code recognizes most common operators (= + - * / comparison & | (logical or) as well as brackets). Also abs() function is supported and it is possible to add practically any function (I will not describe how, because the convention will change in the near future - now it's just for testing purses). The compiler is also very rough - it doesn't recognize several problems, for example it can't detect if it run out of xmm registers nor take precautions to prevent that.

I now, it's not that amazing nor useful yet, but the rest of it is merely adding new features and fixing the ones that are there. I also tried to #comment the code a little, but it's quite messy. I will clean it up, don't worry - just wanted to show it here...

[Exo]

This is a really good start KG

I think writing functions is the most interesting part of this project (for me at least) .

Once the parser is more mature we could use it to write functions in normal code too.