Fragment Audio Server

Oscillator bank / spectral synthesizer built for the Fragment Synthesizer, a web-based collaborative & graphical audio synthesizer

This program should compile on most platforms!

Fragment Audio Server

About

Fragment Audio Server (FAS) is a high performance pixels-based graphical audio synthesizer implemented as a WebSocket server with the C language (C11).

One can see this as a limitless bank of generators / filters.

The versatility of its sound engine allow a wide variety of synthesis methods (aka instruments) to produce sounds:

additive / spectral with per partial effects (bitcrush, noise)
phase modulation (PM/FM)
granular (asynchronous / synchronous)
subtractive synthesis
physical modelling (models : Karplus-strong, droplet)
wavetable synthesis

There is a second type of synthesis methods (or modifiers) which use any synthesis methods from above as input:

second-order band-pass Butterworth filter (bandpass filter bank)
formant synthesis (formant filter bank)
string resonator synthesis (complex filter bank similar to Karplus-Strong)
modal synthesis (resonant filter bank)
phase distorsion

There is a third method which can do both; modify or synthesize sounds :

spectral (via STFT)

There is also the Faust option which allow to import / write custom generators and effects built out of Faust DSP specification language and load them at runtime to extend FAS dynamically with custom DSP code.

Other type of synthesis (Linear Arithmetic Synthesis, Vector synthesis, Walsh, stacked waves etc.) may be supported out of the box easily by a combination of the methods above.

There is also an input instrument which just play input audio so amplitude envelope, effects or second synthesis type can be applied.

There is also a modulation instrument which can be used to modulate fx or instruments.

Multiple instruments can be used at the same time with dedicated virtual output channels (which can be routed to any device output channels), the maximum number of instruments / virtual channels can be changed on launch.

FAS is focused on real-time performances, being cross-platform and pixels-based.

This project was built for the Fragment Synthesizer client, a web-based graphical audio / spectral collaborative synthesizer

The most pixels-adapted synthesis methods are (in order) additive/spectral, wavetable, granular/PM/Physical modelling; re-synthesis is possible with all of them.

Requirement : by default this synth require ~1 Gb RAM (~2 Gb RAM to be safe) which enable a maximum of 24 instruments / virtual channels with a frames queue size of 3, the memory requirement can be lowered greatly by lowering the amount of instruments / virtual channels or the frame queue size parameter (a frame queue size of 1 would divide the requirement by two but would potentially introduce hiccups).

Pixels-based

Unlike other synthesizers, the notes data format understood by FAS is entirely pixels-based, the notes data format can be

8-bit RGBA
32-bit float RGBA

The RGBA data collected is 1px wide with an user-defined height, the height is mapped to frequencies with an user-defined logarithmic frequency map. Multiple 1px slices are defined as separate instruments.

FAS collect the slices RGBA data over WebSocket (streaming) at an user-defined rate (commonly 60 or 120 Hz), convert the RGBA data to a suitable internal data structure and produce sounds in real-time.

It can be said that FAS/Fragment is a generic image-synth (also called graphical audio synthesizer): any RGBA images may be used to produce an infinite variety of sounds by streaming bitmap data to FAS.

Each note events carry amplitude level (stereo) in RED and GREEN component plus additional synthesis parameters in BLUE and ALPHA channels.

With a light wrapper its architecture can also be used as a generic synth right out of the box; just deal with RGBA notes.

As a fun side note FAS can be considered as a basis to build a simple DAW due to architecture similarities.

Specifications

Here is some architectural specifications as if it were made by a synth. manufacturer :

polyphonic; unlimited number of voices (depend on input data height parameter)
multitimbral; unlimited number of timbres / parts / instruments with configurable virtual output channel
distributed architecture; more than one instance can run on same machine / a network with independent processing of voice / part, example : FAS relay
driven by pixels data over the wire; this synth can be used as a basis for other softwares and has about no limitations as-is, it is typically used with a client that implement higher order features like MIDI such as Fragment client
envelopes ? has it all due to stream based architecture, you can build any types (ADSR etc.) with any interpolation scheme (linear / exp etc.)
multiple sound engine; additive / spectral, sample-based, subtractive, wavetable, physical modeling, frequency modulation, spectral, filters bank, phase distorsion...
virtual channels: instruments -> virtual channels -> output device channels
allow to extend the sound engine at runtime with user-defined generators and effects written with Faust DSP specification language
high quality stereophonic audio with low latency and multiple input / output (with Jack)
fully microtonal / spectral (oscillator-bank concept for every instruments)
unlimited effects slot per virtual channel (24 by default but adaptable); reverb, convolution, comb, delay, filters, phaser... 30 high quality effects type provided by Soundpipe are available (most of them stereophonic)
per partial effect slot for additive synthesis
per voice filtering for subtractive synthesis with one multi mode filter
- per voice effects is limited by RGBA note data, this may be seen as a limitation since only one multi mode filter per voice is allowed
highly optimized real-time architecture; run on low-power embedded hardware such as Raspberry
events resolution can be defined as you wish and can be changed dynamically (typically display refresh rate; 60 Hz but you can go above that through a real-time parameter)
cross-platform; run this about anywhere !

Note : "Unlimited" is actually an architectural term, in real conditions it is limited by the available processing power, amount of memory available, frequency resolution is also limited by slice height so it may be out of tune with low resolution! (just like analog but still more precise!)

Additive synthesis

Additive synthesis is a mean to generate sounds by adding sine waves together, it is an extremely powerful type of sound synthesis able to reproduce any waveforms in theory.

When compiled with PARTIAL_FX defined there is a fx slot available per partial, the effect type must be provided as an integer value from the BLUE component, the following effects are available :

0: none
1: bitcrush (Bitdepth / Sampling rate : B component [0, 1] / A component [0, 1])
2: noise (B added white noise factor to sine wave phase, maximum defined by command-line parameter)

Any combination of these can be applied to each partials with real-time parameters change. This feature may allow to easily add character to the additive sound.

Partials effects can be disabled to speed up additive synthesis (or if FAS doesn't use the Soundpipe library).

Additive synthesis use either magic circle algorithm or a wavetable, speed depend on architecture, magic circle algorithm is recommended. (note : noise is not available with magic circle)

RGBA interpretation

Stereophonic

| Components | Interpretations | | ---------: | :--------------------------------------- | | R | Amplitude value of the LEFT channel | | G | Amplitude value of the RIGHT channel | | B | see above | | A | see above |

Granular synthesis

Granular synthesis is a mean to generate sounds by using small grains of sounds blended together and forming a continuous stream.

FAS grains source are audio files (.wav, .flac or any formats

Fas

Install / Use

README