Pianolizer

Description

A simple and efficient algorithm for spectral analysis of music. Examples for browser and Raspberry Pi are provided.

tl;dr

• Pianolizer app - runs directly in the browser. Also, mobile browser. Chrome is recommended for the best experience.
• Algorithm benchmark - test the speed of the core algorithm, in the browser. WASM results closely match what is expected from the native binary performance, on the same machine. 44100 samples per second is enough for realtime performance. As a matter of fact, one can get decent results with as little as 8000 samples per second (saving both CPU & RAM resources!); however the resampling algorithm implementation is left as an exercise to the reader (that is, for the JS/browser; the pianolizer CLI utility is meant to be paired with ffmpeg).

Building the hardware

Parts list

0. Any Raspberry Pi or a compatible device.
1. ReSpeaker 2-Mics Pi hat by Seeed. Any ALSA-compatible audio input device should work (also the external USB ones), but YMMV.
2. WS2812B RGB LED strip, 1m-long with 144 LEDs/m. I used this specific one.
3. A cable with a connector compatible with this one, for connecting the LED strip to the ReSpeaker device.
4. An SD card with at least 4GB capacity.
5. A power source for the Raspberry Pi.

Assembly instructions

0. Connect the ReSpeaker hat to the Raspberry Pi.
1. Connect the LED strip to the GP12 port of the ReSpeaker hat. Connect GND of the strip to pin 4; +5V to pin 3 & DATA to pin 1 (this standard is called Grove System).
2. Download the pianolizer-2023-03-22.img.xz image file (it is based on 2022-09-22-raspios-bullseye-arm64-lite.img, BTW).
3. Write the image to the SD card. You have two options:
   1. I recommend the Raspberry Pi Imager, since it works with any OS and has a GUI to conveniently setup WiFi/SSH before booting. Besides, it decompresses the image on-fly.
   2. xzcat -v pianolizer-2023-03-22.img.xz | sudo dd bs=1M of=/dev/<YOUR_SD_CARD> - this works out-of-box! But you probably want to enable SSH & configure WiFi, and add your own user (pi is already taken, although).
4. Customize the pianolizer.txt file on the initialized SD card (or leave it as-is when using the same parts as listed above).
5. Power up! After the boot completes, the LEDs start blinking when you talk to the microphone :)

Building the software

The C++ implementation should compile just fine on any platform that supports C++11, there are no dependencies as the code uses C++11 standard data types. It is known to compile & run successfully with Clang, GCC and Emscripten. The target platform should support double float operations efficiently (in other words, hardware FPU is rather mandatory).

Compile the native binary (AKA the pianolizer CLI utility) and to WebAssembly:

make

Compile only the native binary:

make pianolizer

To compile only to WebAssembly:

make emscripten

Test and benchmark the C++ implementation (optional; depends on GoogleTest):

make test

Delete all the compiled files:

make clean

Using the CLI utility

General instructions

$ ./pianolizer -h
Usage:
	arecord -f FLOAT_LE -t raw | ./pianolizer -s 8000 | sudo misc/hex2ws281x.py

Options:
	-h	this
	-b	buffer size; default: 256 (samples)
	-c	number of channels; default: 1
	-s	sample rate; default: 44100 (Hz)
	-p	A4 reference frequency; default: 440 (Hz)
	-k	number of keys on the piano keyboard; default: 61
	-r	reference key index (A4); default: 33
	-a	average window (effectively a low-pass filter for the output); default: 0.04 (seconds; 0 to disable)
	-t	noise gate threshold, from 0 to 1; default: 0
	-x	frequency tolerance, range (0.0, 1.0]; default: 1
	-y	return the square root of each value; default: false
	-d	serialize as space-separated decimals; default: hex

Description:
Consumes an audio stream (1 channel, 32-bit float PCM)
and emits the volume levels of 61 notes (from C2 to C7) as a hex string.

The pianolizer CLI utility receives an input stream of following specifications, by default:

Channels       : 1
Sample Rate    : 44100
Sample Encoding: 32-bit Floating Point PCM

And emits a stream of 122-character hexadecimal strings representing the volume level of the 61 consecutive notes (from C2 to C6):

...
000000000000000000010004020100000000000001010419110101010102182e040203192202040b080201010001000105010901060101000000000000
0000000000000000000100040201000000000000010104151301010101021634030202192202040a070101010001000104010801060101000000000000
00000000000000000000000402010000000000000101031215010101010215380302021922020309060101010001000104010701050101000000000000
000000000000000000000004030100000000000000010310160101010103163a0302021a20020308050101010001000104010601040101000000000000
00000000000000000000000404010000000000000001030e1601010102041839030202181f020307050101000001000103010601040101000000000000
...

Each level uses 2 hexadecimal characters (therefore, the value range is 0-255). A new string is emitted every 256 samples (adjustable with -b option); that amounts to ~6ms of audio.

ffmpeg is recommended to provide the input for pianolizer when decoding an audio file. It should be trivial to convert the pianolizer output into a static spectrogram image (TODO). When using a microphone source on a Raspberry Pi, use arecord.

Desktop Linux

On a desktop linux pc - without any 'native' gpios - it is possible to use an arduino that is running an AdaLight (or compatible) sketch. Capturing and visualizing the audio that Pulse-Audio receives and pipes to it's speakers:

RATE=22050
pacat --record --rate $RATE --device=alsa_output.pci-0000_00_1b.0.analog-stereo.monitor | \
    ffmpeg -f s16le -ar $RATE -ac 2 -i - -f f32le -ar $RATE -ac 1 - | \
    ./pianolizer -s $RATE -t 0.03 -a 0.02 | ./misc/hex2adalight.py /dev/ttyACM0

Note that pacat also can directly convert to '--format float32le', but for some reason leaving this up to ffmpeg introduces less latency between notes beeing played and the visualization showing the corresponding keys.

Raspberry Pi specific

The included Python script consumes the hexadecimal output of pianolizer and drives a WS2812B LED strip (depends on the rpi_ws281x library). Conveniently, 1m LED strip with 144 diodes/meter matches precisely the standard piano keyboard dimensions and is enough to cover 61 keys.

Raspberry Pi has no audio input hardware at the time of writing, therefore an expansion board is required. I am using ReSpeaker 2-Mics Pi HAT by Seeed.

Check pianolizer.sh for an example of how to drive the LED strip with a microphone. You will probably need to adjust the sample rate and volume in this script, and also the I/O pin number in the Python script.

Using the library

The main purpose of Pianolizer is music visualization. Because of this, the volume level values are squared (more contrast, less CPU usage) and averaged (effectively, a low-pass filter of the output, otherwise it is unpleasant and potentially harmful to look at, due to flickering). However, the library is modular by design, so you can shuffle things around and implement other stuff like DTMF decoder or even a vocoder (YMMV!).

In a nutshell, first you need to create an instance of SlidingDFT class, which takes an instance of PianoTuning class as a parameter. PianoTuning requires the sample rate parameter. Sample rate should be at least 8kHz. By default, PianoTuning defines 61 keys (from C2 to C7), with A4 tuned to 440Hz. Why not 88 keys, like most of the acoustic pianos? Essentially, it is because the frequencies below C2 (65.4Hz) would require some extra processing to be visualized properly.

Once you have an instance of SlidingDFT, you can start pumping the audio samples into the process method (I recommend doing it in chunks of 128 samples, or more). process then returns an array of 61 values (or whatever you defined instantiating PianoTuning) ranging from 0.0 to 1.0, each value being the squared amplitude of the fundamental frequency component for that key.

C++

Standard: C++11 (but C++14 or higher is recommended)

• Include [p