BeatNet: Real-time and Offline Joint Music Beat, Downbeat, Tempo, and Meter Tracking System

BeatNet is the state-of-the-art AI-based Python library for joint music beat, downbeat, tempo, and meter tracking. This repo includes the BeatNet neural structure along with the efficient two-stage cascade particle filtering algorithm that is proposed in the paper. It offers four distinct working modes, as follows:

• Streaming mode: This mode captures streaming audio directly from the microphone.
• Real-time mode: In this mode, audio files are read and processed in real-time, yielding immediate results.
• Online mode: Similar to Real-time mode, Online mode employs the same causal algorithm for track processing. However, rather than reading the files in real-time, it reads them faster, while still producing identical outcomes to the real-time mode.
• Offline mode: Inferes beats and downbeats in an offline fashion.

New in v1.2.0: The official training pipeline is now included. You can train the BeatNet CRNN from scratch on your own data or reproduce the paper's results using the provided training scripts, dataset handlers, and test suite.

To gain a better understanding of each mode, please refer to the Usage examples provided in this document.

This repository contains the user package and the source code of the Monte Carlo particle flitering inference model of the "BeatNet" music online joint beat/downbeat/tempo/meter tracking system. The arxiv version of the original ISMIR-2021 paper:

In addition to the proposed online inference, we added madmom's DBN beat/downbeat inference model for the offline usages. Note that, the offline model still utilize BeatNet's neural network rather than that of Madmom which leads to better performance and significantly faster results.

Note: All pre-trained models are included in the models folder. The official training script is now part of this repository (see Training section below).

System Input:

Raw audio waveform object or directory.

• By using the audio directory as the system input, the system automatically resamples the audio file to 22050 Hz. However, in the case of using an audio object as the input, make sure that the audio sample rate is equal to 22050 Hz.

System Output:

A vector including beats and downbeats columns, respectively with the following shape: numpy_array(num_beats, 2).

Input Parameters:

model: An scalar in the range [1,3] to select which pre-trained CRNN models to utilize.

mode: An string to determine the working mode. i.e. 'stream', 'realtime', 'online' and 'offline'.

inference model: A string to choose the inference approach. i.e. 'PF' standing for Particle Filtering for causal inferences and 'DBN' standing for Dynamic Bayesian Network for non-causal usages.

plot: A list of strings to plot. It can include 'activations', 'beat_particles' and 'downbeat_particles' Note that to speed up plotting the figures, rather than new plots per frame, the previous plots get updated. However, to secure realtime results, it is recommended to not plot or have as less number of plots as possible at the time.

thread: To decide whether accomplish the inference at the main thread or another thread.

device: Type of device being used. Cuda or cpu (by default).

Installation command:

Approach #1: Installing binaries from the pypi website:

pip install BeatNet

Approach #2: Installing directly from the Git repository:

pip install git+https://github.com/mjhydri/BeatNet

• Note: Before installing the BeatNet make sure Librosa and Madmom packages are installed. Also, pyaudio is a python binding for Portaudio to handle audio streaming. If Pyaudio is not installed in your machine, depending on your machine type either install it thorugh pip (Mac OS and Linux) or download an appropriate version for your machine (Windows) from here. Then, navigate to the file location through commandline and use the following command to install the wheel file locally:

pip install <Pyaduio_file_name.whl>     

Usage example 1 (Streaming mode):

from BeatNet.BeatNet import BeatNet

estimator = BeatNet(1, mode='stream', inference_model='PF', plot=[], thread=False)

Output = estimator.process()

*In streaming usage cases, make sure to feed the system with as loud input as possible to leverage the maximum streaming performance, given all models are trained on the datasets containing mastered songs.

Usage example 2 (Realtime mode):

from BeatNet.BeatNet import BeatNet

estimator = BeatNet(1, mode='realtime', inference_model='PF', plot=['beat_particles'], thread=False)

Output = estimator.process("audio file directory")

Usage example 3 (Online mode):

from BeatNet.BeatNet import BeatNet

estimator = BeatNet(1, mode='online', inference_model='PF', plot=['activations'], thread=False)

Output = estimator.process("audio file directory")

Usage example 4 (Offline mode):

from BeatNet.BeatNet import BeatNet

estimator = BeatNet(1, mode='offline', inference_model='DBN', plot=[], thread=False)

Output = estimator.process("audio file directory")

Training:

The official training pipeline is included in this repository. Training involves three steps: data preparation, training, and evaluation.

Prerequisites

Install the required packages:

pip install BeatNet

Or install from source:

pip install -e .

Note on madmom compatibility: madmom 0.16.1 has compatibility issues with Python >= 3.10 and NumPy >= 1.24. If you encounter ImportError: cannot import name 'MutableSequence' from 'collections' or AttributeError: module 'numpy' has no attribute 'float', you can either (a) use Python 3.9, or (b) apply the following fixes to your madmom installation:

• In madmom/processors.py, change from collections import MutableSequence to from collections.abc import MutableSequence
• For numpy alias errors in compiled Cython extensions, add this to your Python's sitecustomize.py:

import numpy as np
if not hasattr(np, 'float'): np.float = np.float64
if not hasattr(np, 'int'): np.int = np.int_

Step 1: Prepare Data

Organize your raw dataset with the following structure:

raw_datasets/
    ballroom/
        audio/ChaChaCha/track001.wav
        audio/Waltz/track002.wav
        ...
        annotations/track001.beats
        annotations/track002.beats
        ...
    gtzan/
        audio/blues/track001.wav
        ...
        annotations/track001.beats
        ...

The .beats annotation format is one line per beat: <time_in_seconds> <beat_number>, where beat_number == 1 indicates a downbeat.

Extract features and annotations:

python -m BeatNet.prepare_data --config src/BeatNet/configs/default.yaml \
    --raw_dir /path/to/raw_datasets \
    --dataset BALLROOM GTZAN BEATLES CMR ROCK_CORPUS

This creates pickled per-track feature files under ./data/ (configurable via --data_dir).

Step 2: Train

python -m BeatNet.train --config src/BeatNet/configs/default.yaml

Key configuration options (set in configs/default.yaml or as CLI overrides):

python -m BeatNet.train --config src/BeatNet/configs/default.yaml \
    learning_rate=0.001 batch_size=128 device=cuda

| Parameter | Default | Description | |-----------|---------|-------------| | batch_size | 200 | Training batch size | | learning_rate | 5e-4 | Adam optimizer learning rate | | seq_len | 400 | Training sequence length in frames (8 seconds at 50fps) | | max_epochs | 10000 | Maximum training epochs | | patience | 20 | Earl