Custom DSP Code 2 (comunity project)

[KG_is_back]

@exo
Do not add new functions just yet. I will revise, optimize simplify and add new features to ASMfunction class. Most likely the format will change in the near future. I will also write a complete manual with tutorials/examples once it's finished. It you're merely curious how it works, here's an explanation:

Functions and operators are instances of ASMfunction class. The class contains 2 functions that are made public:

test(inputTypeString) - this function tests, whether the function is valid for that number and types of inputs. If it is, it returns an array where at [0] is expected minimal CPU cost and at [1] the amount of registers the function needs for operation (so the compiler may move intermediate results from registers into variables when function needs more registers then are available - in fact it can keep doing so infinately, so no more 8-successive-operations-per-line limit in DSPcode2). It will possibly contain additional informations in the future.
The input is a string where each char represents an input. For example "rvca" means the function has 4 inputs - first is a register, second is a variable, third is a constant and last one is an array (yes, you can supply function with a full array as an argument)

generate(inputTypeString,[inputs],@regs,@uniq) - this function receives the input type string, array, which contains the names of the inputs, @regs array, which contains the real register numbers vs relative ones (whenever you use xmm(0) in your code generator it will be replaced by "xmm"+@regs[0].to_s ) and lastly @uniq, which is a string that changes for each function call, block, etc. basically it is used to make labels and certain types of variables unique for each function call.
It should return an array with 5 entries (in the functions I've written the array is called ret[]) :

ret[0]= an array of variables and constants, that should be added into the code declaration by compiler. Always use BlueprintVar() method to declare these. Simply change the string in the first argument by this scheme:
'var float a=4; local float x=7; const int i=11;'
This would initialize a variable "a" that will be recycled by any function call (will not keep the value to next sample). A local variable "x" which will be made unique for each function call (will keep value to next sample - every myFunction(...) in code will have it's own "x"). And a constant "i", which is "read-only" and will be identical to any integer 11 you use anywhere. Always use this function before you use blueprintCode() method, because it also does some additional stuff in the background, that I have no lust to explain right now.

ret[1]= an initialization code (a string), that will be put into stage(0) (before the actual stage0 code). In all the functions I've added so far, this is empty string, because none of them needs such code.

ret[2]= an iterative code (a string), that will be placed, where you've used the function.

ret[3]= info bar. It returns additional info [expected CPU cost, no. of registers used]. Should return the same as test() method.

ret[4]= the return. This should contain the name of the variable or register, where the return value ends up.

The code may be generated in several different ways, and there are several things you must keep in mind.
1. register numbers are managed by the compiler. Use xmm(i) method when you what to return the real name of the register. Always use registers with the least index. (for example use xmm(0) and xmm(1) when you need two - don't use xmm(0) and xmm(7), because that means the function uses all of the registers). Also don't forget to mention how many registers you've used in ret[3]
2. names of variables and constants you've declared in blueprintVar() are not the real names that will be used by the compiler. To get the real name use @consts and @vars hash. For example @consts["i"] would return the real name of the constant 11 declared in blueprintVar example above.
3. labels for jumps must be made unique for each function call. add @uniq string to their names to do so.
4. input names are in "inputs" array.


Alternatively you may use blueprintCode() method. You simply supply it with your ASM code blueprint and it will automatically change all variable names and labels and return the real code as string. to use input argument in blueprintCode() method use names arg0/arg1/... They serve as placeholders and will be replaced by real input names once function is called.


For example following codes will do the same:

Code: Select all

one way to do it:
ret[2]="movaps "+xmm(0)+","+inputs[0]+";" + "addps "+xmm(0)+","+inputs[1]+";"
second way to do it:
ret[2]=blueprintCode("movaps xmm0,arg0; addps xmm0,arg1;",@name)


Now, How to create function pack/add new functions?

Go to "functions" module and there to "packs" module. Create a text component. This component will contain your function pack and will be installed once you connect it to the string array. (the new 3rd party function packs may be installed so easily)

The function pack is a ruby code, that will be run via eval() function in the module above (you can see it in the "functions" module).
First you create instance of ASM function class and supply the function name:

Code: Select all

func=ASMfunction.new("myFunction")


next you must create a string, which will contain the ruby code for the "generate()" method. In the method the string is run as code using eval().
The code should return an array of 5 values - explained above.

next you must add this ASMblueprint to the ASM function you've created. That is done using addFunction method.

arguments for the method are:
1- an array of input type strings, which specify for what input types this code generator is valid.
2- the code string mentioned above
3- the info array, which will be returned by test() method. [expected CPU cost, no. of xmm registers the code uses]

Code: Select all

#convert integer to float
func=ASMfunction.new("myFunction")
code="ret=[]
ret[0]=blueprintVar('',@name,inputs)
ret[1]=''
ret[2]=blueprintCode(
'cvtdq2ps xmm0,arg0;',@name)
ret[3]=[5,1]
ret[4]=xmm(0)
return ret"
func.addFunction(["v","c","r"],code,[5,1])


You may add multiple different code generators for different input types, so the function may return different code depending on the types of inputs. If you don't supply certain arrangement of input types, the function will not be generated ("unknown function for provided arguments" error will be raised).

Last thing to do is to add the function into the @functions hash, which is used by the compiler:

Code: Select all

@functions["myFunction"]=func


As I've said The current version of ASM function class is just a prototype. I will add new features and overall change the specification very soon.

[Exo]

Thanks for the explanation KG.

Yes I figured the design wouldn't be set just yet but wanted to experiment with it.

I have quite a bit of experience in object oriented design so hopefully this input will be useful...

To make adding functions easier and less error prone I think everything should be inside the ASMfunction class.

For instance you are exposing a lot of details that we don't need to know about ,for example the "code" is build externally to the class and then added to the function. I think a simpler API would be like this...

Code: Select all

#round to int
func=ASMfunction.new("rndint")
func.setVariables('',@name,inputs)
func.setStage1Code("Some stage 1 stuff") #This can bit omitted
func.setCode("Stage 2 stuff, I don't think calling the function "setStage2Code" is necessary")
func.setStage3Code("Stage 3 code") #Also can be omitted
func.setEstimatedCPU(5) #Can be omitted
func.setRegistersUsed(1) #Could also be omitted?
func.setInputs("rvca")
@functions["rndint"]=func


So we are just dealing with function calls that are very descriptive. Also we can omit certain ones, so in the case of not needing stage 1 we can just not call "setStage1Code" instead of setting ret[1]='' ect..

So simple function could be...

Code: Select all

#round to int
func=ASMfunction.new("rndint")
func.setVariables('',@name,inputs)
func.setCode("Stage 2 stuff, I don't think calling the function "setStage2Code" is necessary")
func.setInputs("rvca")
@functions["rndint"]=func


We just call what is relevant, the ASMfunction class can take care of any blank or default values if they are not set.

So that is where I see this heading, I would find an API like that easier to work with and understand.

Cheers

[KG_is_back]

It sounds reasonable... But here's the problem... Some functions (special cases of functions) do not use a simple "copy-paste from assembler to blueprintCode() method" code generators.

For example pow(var,integer constant)=( var^integer) will be implemented by stacking multiplications, rather than CPU heavy power function. Such code is directly dependent on input parameters (the integer input and also number of available registers) which also results in variable CPU cost, variable number of variables and constants and register number requirements.

Also in current version the same code-generator may be used by different combinations of "rvca" (basically every ASMfunction2 instance contains a Hash, where each "rvca" is a key to a codegen). You will basically have to add variables/codegens, etc. per "rvca" basis.

All of that is what potentially makes DSPcode2 an actual improvement over original DSPcode - multiple implementations of the same function, optimized for all types of inputs (and in case of constants also their values).
We have to be smart about the API, so it doesn't remove any features, yet makes the process simple.

perhaps something like this:

Code: Select all

func=ASMfunction.new("name")
rvca=["rvca","rr"] #a list of "rvca" input statuses
func.create(rvca) #this will initialize new codegens per "rvca" with default values

#basic methods (should do the trick 90% of times)

func.setVariables(rvca,"variables") #this will add new variables to each rvca. By default ""
func.setInitCode(rvca,"code") #this will add stage0 code Generator. By default ""
func.setRunCode(rvca,"code") # this will add stage2 code Generator. By default "//name"
func.setAfterCode(rvca,"code") #stage3 codegen...

#In all of above cases the CPU cost and register requirements will be generated automatically.

func.setAdvancedCodegen(rvca,"Ruby code-Generator") #basically will bypass all basic functions
#the "Ruby code-Generator" is a string containing a Ruby code (which will be evaluated using eval() ).
#in it, the user will have to assign correct data to "@variables" "@initCode" etc.
#the user will have to specify reg-count and CPU cost. He will have access to functions, that may estimate the values from the code in "@runCode" (the same ones that estimate it automatically when basic methods are used)


The "one Code gen for each rvca" is obligatory. Not only functions are implemented via that system, but also all operators and statements (namely if-else, loops, hop). Newly I've added also, that for the generate method not only inputs are provided (be it variables, constants or registers), but also code that generated them. For example "abs(5*in);" recevies "xmm0" as input and also "movaps xmm0,F5; mulps xmm0,in;" as code for that input, and lets you place it anywhere in the blueprint (in the beginning by default).

[Exo]

Yes that API looks good to me. Most of it is pretty obvious, I think that is about as simple as it could be without losing anything.

This thing is going to be great, once we have a good set of functions everything from oscillators to filters ect, it would be pretty simple and efficient to do the entire of the audio for a plugin in one code!

[Perfect Human Interface]

Here's a question: Will ifs works like "real" ifs or will it just compile to bitmasking? In other words, can I skip a block of computations if a conditional is not met?

[KG_is_back]

Here's a question: Will ifs works like "real" ifs or will it just compile to bitmasking? In other words, can I skip a block of computations if a conditional is not met?

"Ifs" will be based on real conditional jumps, so yes - they will be true "execute or skip code"-type of thing. It will bring out several problems though - because of SSE 4-channel processing, only one channel( the first one) will actually be used as the condition for all 4 channels. Either all 4 channels be skipped or all 4 will be executed.

There will be a way to mimic per-channel bypass in following way:

Code: Select all

if( any( exe=(a>0))) {
nexe=not(exe);

result=nexe & result + exe & ("computation");
};


"Any()" will be a function in the standard pack, that will compute logical OR of all 4 channels and will put the result in all of them. Using it, the code will only skip, when all 4 channels are false. In above example "exe" holds the result of per-channel comparison and "nexe" holds the opposite value. You may use them to update only the channels, that are "true" in a way shown in the example. (also notice, that you may use assignment in a middle of a statement).
The same will apply for conditional loops and variable length loops and hops.

Also there will be "all()" function to output true only when all channels are true, as well as "shuffle()" to shuffle the channels (ie. bring any desired channel to channel0 to serve as a condition). "min()" and "max()" functions will output the maximum/minimum of 4 channels, when provided with only one parameter.

[Perfect Human Interface]

Ok, so there will definitely be some quirks where things don't necessarily give you what you expect at face value. But still, that's huge for efficiency to be able to skip/turn off chunks of code!

[KG_is_back]

Ok, so there will definitely be some quirks where things don't necessarily give you what you expect at face value. But still, that's huge for efficiency to be able to skip/turn off chunks of code!

Well... DSPcode is not really a fullblown programming language with a compiler - it is not meant to produce a full program. It's just a code-generator for Flowstone - a module. Because it's meant to generate 4-channel code that is meant to run in flowstone's assembler, it's sort of a one trick pony. Although its syntax will remind C and Ruby it will not work in the same way and there will be a couple of things to be aware of (like above-mentioned if statement behaviour).

The efficiency of code-jumping (in "If" statements and loops) is surprisingly questionable. The negative side-effects of unpredictably (understand: in irregular patterns) jumping from one part of a code to another may actually create bigger CPU waste then running both branches and then using only one of the results. It is only viable if the jumping pattern is regular enough (like in hop statement in stock DSPcode) OR the branches are considerably big (having 20+ ASM instructions, or very CPU heavy ones like division and square root). I've done some research on this in the past and written articles about it on FSguru. Also I will definitely mention it in the DSPcode2 manual.

[Perfect Human Interface]

Should the "if" statements maybe support SSE by default? You said it is possible, maybe it is less efficient, but wouldn't consistency in such a modular environment be more important than efficiency? I'm referring to the idea that in Flowstone we operate under the assumption that using SSE is as simple as wiring a mono4 through a stream connector. It would be frustrating if people started releasing all sorts of components and it ended up a bunch of them weren't SSE compatible. How would the end-user know or find out? Especially for those who are working with less knowledge of Flowstone, both in using and writing DSP2 components, it could be an important issue.

Would it be possible to create two different "ifs," one which is more robust and supports SSE fully and predictably and one which is basic and functions as you've described?

[KG_is_back]

Should the "if" statements maybe support SSE by default? You said it is possible, maybe it is less efficient, but wouldn't consistency in such a modular environment be more important than efficiency? I'm referring to the idea that in Flowstone we operate under the assumption that using SSE is as simple as wiring a mono4 through a stream connector. It would be frustrating if people started releasing all sorts of components and it ended up a bunch of them weren't SSE compatible. How would the end-user know or find out? Especially for those who are working with less knowledge of Flowstone, both in using and writing DSP2 components, it could be an important issue.

Would it be possible to create two different "ifs," one which is more robust and supports SSE fully and predictably and one which is basic and functions as you've described?

You've got a good point there. When I started the DSPcode2 project, it mainly meant to be a DSPcode for mono, that enables several possibilities that are only mono-compatible (namely branching and looping). SSE itself was not meant to contain such features - it was meant to reduce size of loops and allow some parallel processing (by processing 4 identical operations in parallel). Above-mentioned SSE compatible "If" statement is more of a "hack". It becomes especially complicated in nested loops (when you need different number of iterations for each channel) and branching within branching. Frankly, I can't even do it at my current level of programming knowledge. There's a loot of stuff to be considered and done if such behaviour should be implemented (for example when global variable is being updated inside a function, or when data travels form one channel to another). I'm not saying it can't be done, I'm just saying it's beyond my skills at the moment, but maybe in the future... who knows.

What I personally consider the main update vs. the old DSPcode is to allow bigger customization. To combine low level features (like inline assembler) with high-level features (ifs, loops, no need to manage registers and labels) and to wrap low-level stuff into usable high-level features (declaring new functions as ASM blueprints).

Once the main work is done, I will write a manual that will describe all features (and point out all their differences vs. the "mono-channel" programming languages like C and ruby) along with several useful and clear examples (like the one I pointed out above). I will clearly point out which features are mono-only and provide poly-compatible versions (or at least examples how to mimic poly compatibility) if it's even possible.

I'm glad we're having this discussion, it gives me an idea what people actually expect from this...