Audio to Polyphonic MIDI

[KG_is_back]

Well... not just yet. I have made an FFT based pitch detector. It detects peaks in FFT spectrum and estimates it's frequencies and amplitudes. Now I need a Ruby module that will remove zeros from the array and leave nonzero values, and then create midi events based on the array of pitches and magnitudes.

The schematic contains custom Threshold for the spectrum, but I'll have to make it logarithmic to be usable.

Apart from that also another function should be added - overtone excluding. You enter a harmonic series of the soundsource (like on the additive osc). Algorithm will check whether the pitch is (near) a integer multiple of some of previous detected notes. If it is, then it will locally rise the threshold by supposed amplitude of that harmonic. That should prevent false triggering by overtones.

[tester]

Sounds interesting.

I generally look for pitch detector, but modules found earlier were having small problem - band size (detection) vs blind spots inside. These few to ten Hz spots were due to split of the spectra into smaller bands.

I need a pitch detector that follows continuous voice (like slowly changing vowels), in a way that allows to track amplitudes of first 20 (up to 32) harmonics of the voice. The problem I had were the buffers and pitch detection stability (the higher band measured above fundamental, the greater error) plus these blind spots in the spectra.

[KG_is_back]

This should be the thing you are looking for. It can detect the pitch downto 0.04Hz precision (thanks to interpolation) with higher FFTsizes and it detects pitches and amplitudes of all frequency peaks above threshold.

With Ruby module that can remove the zero values form the array you can get an array of all the pitches of all overtones with their respective amplitudes with ease. Can you do that? it would help me a lot too... It can possibly do that for polyphonic material too - simply by splitting the array into arrays that contain (roughly) integer multiples of one frequency.

[tester]

I'm not too good in ruby, but perhaps Trog or Nubeat7 could help. Shouldn't be complicated I think. I suspect this have to be fast working module, for on-the-fly updates, so it would be a matter of selecting proper method.

p.s.: looks you need to remove upper half of the array too (FFT symmetry aspect).

[Nubeat7]

wow sounds great!

Code: Select all

array.delete_if {|x| x == 0 }

like this all zero values gets deleted from array

[Tronic]

Code: Select all

output @in.reject! {|x| x == 0}

[KG_is_back]

So.... now it outputs corresponding pitches and amplitudes. Now we have to turn them into midi somehow. It should work something like this:
Ruby module stores previous array into its internal memory. scans input and memory arrays. All pitches that are the same in both arrays will do nothing. If pitch is present in input but not memory array, then note-on midi signal will be send. If pitch is present in memory but not input note-off midi signal will be send.
Some syncing between FFT and the ruby will be needed, so frames maybe? ...

[Nubeat7]

..It should work something like this:
Ruby module stores previous array into its internal memory. scans input and memory arrays. All pitches that are the same in both arrays will do nothing. If pitch is present in input but not memory array, then note-on midi signal will be send. If pitch is present in memory but not input note-off midi signal will be send.
Some syncing between FFT and the ruby will be needed, so frames maybe? ...

sorry but i dont really get it which arrays should be compared, do you mean the 2 float arrays at the end (freq/amplitude)?, sorry i`m a dsp noob? if scanning the input, shouldnt this be done samplewise? if yes ruby wouldnt be usable, its also would be difficult to get the frames in sync..

[KG_is_back]

..It should work something like this:
Ruby module stores previous array into its internal memory. scans input and memory arrays. All pitches that are the same in both arrays will do nothing. If pitch is present in input but not memory array, then note-on midi signal will be send. If pitch is present in memory but not input note-off midi signal will be send.
Some syncing between FFT and the ruby will be needed, so frames maybe? ...

sorry but i dont really get it which arrays should be compared, do you mean the 2 float arrays at the end (freq/amplitude)?, sorry i`m a dsp noob? if scanning the input, shouldnt this be done samplewise? if yes ruby wouldnt be usable, its also would be difficult to get the frames in sync..

The first one that contain pitches (this new version really contains midi note numbers,...might be convenient to simply turn them to integers). The module periodically updates the array. I need a ruby module that would scan the changes of the array and turn every new entity into note-on signal and every entity that disappears to note-off. The array that contains amplitude might be used to determine velocity of the note, however that is not that crucial.

Also I've made a little updates. the Threshold bar-graph editor is now working and it is scaled roughly exponentially. The same for Harmonic model. When new note is detected it is stored and everytime the algorithm gets close to a frequency that might be an overtone it uses supposed overtone amplitude from the Harmonic model as a new threshold. That gives the algorithm better idea of what is an overtone and what is new independent note. It uses trogs array to memory code to instantly transfer the model to the detector.
The code is still very likely to crash when loaded, so be careful.

[KG_is_back]

what I've meant is this. First input is the current array of pitches, second input is a previous array of pitches (btw how to delay an array in green?). the first output array contains notes that are new (they should be converted to note-on signals) and second one contains notes that were released (they should be converted to note-off signals)