audacitorch

This package contains utilities for prepping PyTorch audio models for use in Audacity. More specifically, it provides abstract classes for you to wrap your waveform-to-waveform and waveform-to-labels models (see the Deep Learning for Audacity website to learn more about deep learning models for audacity).

Table of Contents

• Downloading Audacity with Deep Learning
• Installing
• Contributing Models to Audacity
• Choosing an Effect Type
  • Waveform to Waveform models
  • Waveform to Labels models
• Model Metadata
• Making Your Model Built-In to Audacity
• Debugging Your Model in Audacity
• Example - Waveform-to-Waveform
  • Making sure your model is compatible with torchscript
• Examples

---

<a name="download"/>

Download Audacity with Deep Learning

Our work has not yet been merged to the main build of Audacity, though it will be soon. You can keep track of its progress by viewing our pull request. In the meantime, you can download an alpha version of Audacity + Deep Learning here.

Installing

<a name="pip"/>

You can install audacitorch using pip:

pip install audacitorch

<a name="contrib"/>

Contributing Models to Audacity

Supported Torch versions

audacitorch requires for your model to be able to run in Torch 1.9.0, as that's what the Audacity torchscript interpreter uses.

Deep Learning Effect and Analyzer

Audacity is equipped with a wrapper framework for deep learning models written in PyTorch. Audacity contains two deep learning tools: Deep Learning Effect and Deep Learning Analyzer.
Deep Learning Effect performs waveform to waveform processing, and is useful for audio-in-audio-out tasks (such as source separation, voice conversion, style transfer, amplifier emulation, etc.), while Deep Learning Analyzer performs waveform to labels processing, and is useful for annotation tasks (such as sound event detection, musical instrument recognition, automatic speech recognition, etc.). audacitorch contains two abstract classes for serializing two types of models: waveform-to-waveform and waveform-to-labels. The classes are WaveformToWaveformBase, and WaveformToLabelsBase, respectively.

<a name="effect-types"/>

Choosing an Effect Type

<a name="effect-diagram"/>

<a name="wav2wav"/>

Waveform to Waveform models

As shown in the effect diagram, Waveform-to-waveform models receive a single multichannel audio track as input, and may write to a variable number of new audio tracks as output.

Example models for waveform-to-waveform effects include source separation, neural upsampling, guitar amplifier emulation, generative models, etc. Output tensors for waveform-to-waveform models must be multichannel waveform tensors with shape (num_output_channels, num_samples). For every audio waveform in the output tensor, a new audio track is created in the Audacity project.

<a name="wav2labels"/>

Waveform to Labels models

As shown in the effect diagram, Waveform-to-labels models receive a single multichannel audio track as input, and may write to an output label track as output. The waveform-to-labels effect can be used for many audio analysis applications, such as voice activity detection, sound event detection, musical instrument recognition, automatic speech recognition, etc. The output for waveform-to-labels models must be a tuple of two tensors. The first tensor corresponds to the class indexes for each label present in the waveform, shape (num_timesteps,). The second tensor must contain timestamps with start and stop times for each label, shape (num_timesteps, 2).

What If My Model Uses a Spectrogram as Input/Output?

If your model uses a spectrogram as input/output, you'll need to wrap your forward pass with some torchscript-compatible preprocessing/postprocessing. We recommend using torchaudio, writing your own preprocessing transforms in their own nn.Module, or writing your PyTorch-only preprocessing and placing it in WaveformToWaveform.do_forward_pass or WaveformToLabels.do_forward_pass. See the compatibility section for more info.

<a name="metadata"/>

Scripting a Model

Once you have chosen the appropriate class type for your model from the provided audacitorch Deep Learning Effect and the Deep Learning Analyzer classes, you will need to create a model file for use in Audacity. This model file allows a trained model to be executed in Audacity.

There are several methods available to create a file for an executable deep learning model. The purpose of serializing the model into a file is to enable our C++ code to execute your model. To serialize a model, our framework utilizes files generated from TorchScript. An important note is that TorchScript does not facilitate model training. When investigating TorchScript, you may also come across the term LibTorch, which is a PyTorch C++ API. LibTorch contains the core components of PyTorch, allowing TorchScript files to be executed in C++. However, you do not need to interact directly with LibTorch to serialize your model.

TorchScript enables the serialization of PyTorch code and is included with the PyTorch module - no additional modules are required. Currently, the deep learning tools for Audacity do not support models running on the GPU. More information on TorchScript can be found in the PyTorch documentation.

TorchScript features a JIT module, where JIT stands for Just-In-Time Compiler. The TorchScript JIT analyzes PyTorch code and translates it into TorchScript. There are two methods for converting PyTorch code into TorchScript:

• Tracing: torch.jit.trace constructs the computational graph of your model by tracing the path of sample inputs through your model.

• Scripting: This method parses the PyTorch code to compile a graph for TorchScript. Scripting is a more robust method for generating a TorchScript version of your model, as tracing can overlook certain logic embedded in the Python code of your model.

These two approaches can be combined, with more information available in the TorchScript documentation. We recommend using TorchScript scripting whenever possible for more robust model serialization.

Serializing a model can be a challenging task with many unique edge cases. To help you navigate this process, we have provided several examples.

Model Metadata

Certain details about the model, such as its sample rate, tool type (e.g. waveform-to-waveform or waveform-to-labels), list of labels, etc. must be provided by the model contributor in a separate metadata.json file. In order to help users choose the correct model for their required task, model contributors are asked to provide a short and long description of the model, the target domain of the model (e.g. speech, music, environmental, etc.), as well as a list of tags or keywords as part of the metadata. Note that you do not need to manually create a metadata file, we provide utility function to automatically create and test metadata files from a Python dictionary. For an example of creating a metadata file from a Python dictionary, see here.

Metadata Spec

required fields:

• sample_rate (int)
  • range (0, 396000)
  • Model sample rate. Input tracks will be resampled to this value.
• domains (List[str])
  • List of data domains for the model. The list should contain any of the following strings (any others will be ignored): ["music", "speech", "environmental", "other"]
• short_description(str)
  • max 60 chars
  • short description of the model. should contain a brief message with the model's purpose, e.g. "Use me for separating vocals from the background!".
• long_description (str)
  • max 280 chars
  • long description of the model. Shown in the detailed view of the model UI.
• tags (List[str])
  • list of tags (to be shown in the detailed view)
  • each tag should be 15 characters max
  • max 5 tags per model.
• labels (List[str)
  • output labels for the model. Depending on the effect type, this field means different things
  • waveform-to-waveform
    • name of each output source (e.g. drums, bass, vocal). To create the track name for each output source, each one of the labels will be appended to the mixture track's name.
  • waveform-to-labels:
    • This should be classlist for model. The class indexes output by the model during a forward pass will be used to index into this classlist.
• effect_type (str)
  • Target effect for this model. Must be one of ["waveform-to-waveform", "waveform-to-labels"].
• multichannel (bool)
  • If multichannel is set to true, stereo tracks are passed to the model as multichannel audio tensors, with shape (2, n). Note that this means that the input could either be a mono track with shape (1, n) or stereo track with shape (2, n).
  • If multichannel is set to false, stereo tracks are downmixed, meaning that the input audio tensor will always be shape (1, n).

---

<a name="builtin"/>

Making Your Model Built-In To Audacity

By default, users have to click on the Add From HuggingFace button on the Audacity Model Manager and enter the desired repo's ID to install a community contributed model. If you, instead, would like your community contributed model t