ParaCLAP – Towards a general language-audio model for computational paralinguistic tasks

This repo includes the official PyTorch implementation of ParaCLAP – Towards a general language-audio model for computational paralinguistic tasks

Abstract

Contrastive language-audio pretraining (CLAP) has recently emerged as a method for making audio analysis more generalisable. Specifically, CLAP-style models are able to ‘answer’ a diverse set of language queries, extending the capabilities of audio models beyond a closed set of labels. However, CLAP relies on a large set of (audio, query) pairs for pretraining. While such sets are available for general audio tasks, like captioning or sound event detection, there are no datasets with matched audio and text queries for computational paralinguistic (CP) tasks. As a result, the community relies on generic CLAP models trained for general audio with limited success. In the present study, we explore training considerations for ParaCLAP, a CLAP-style model suited to CP, including a novel process for creating audio-language queries. We demonstrate its effectiveness on a set of computational paralinguistic tasks, where it is shown to surpass the performance of open-source state-of-the-art models.

Instruction

The goal of this work is to create a CLAP-style model for computational paralinguistics. This is done by training an acoustic model (audeering-w2v2-emo) and a text model (bert-base-uncased) with the CLIP objective (contrastive loss).

The text-audio pairs are generated through a novel art of templating which accepts as input eGeMAPS features and generates pseudo-captions.

The templates are generated by preprocessing/template_creation.py. This script can be adapted to generate more pseudo-captions and control which variables are used.

The remaining scripts in this repository can be used as follows:

• train.py trains the CLAP model
• features.py extracts eGeMAPS features for MSP-Podcast
• evaluate.py evaluates the CLAP model on emotion recognition on different datasets
  • Depending on which data has been used for training, this is may be a zero-shot scenario

Please find our best checkpoint at HuggingFace trained and evalatuted on the MSP-Podcast.

Evaluation

Evaluation on file

1. update your candidit list and filepath in eval.py, and the checkpoint will be automaticly downloaded

2. run the following command:

python ./eval.py

Evaluation on dataset

1. update your dataset path in eval_dataset.py, and the checkpoint will be automaticly downloaded

2. run the following command:

python ./eval_dataset.py

Evaluation on Your Dataset

Dataset Requirements

Please check the './Data/cremad.py' for the dataset template. There are two requirements for your dataset to be compatible with the evaluation script:

1. Initialization (__init__(self)):

   • Define the following variables in your dataset class:
     • self.emotion_map: A dictionary mapping emotion labels to corresponding numerical values (e.g., {'happy': 0, 'sad': 1}). This mapping is used for statistical analysis of predictions.
     • self.emo_list: A list of available emotion labels (e.g., ['happy', 'sad']). This will serve as candidates during evaluation.

2. Dataset Return Format:
   Your dataset should return the following in each iteration:

   • torch.Tensor(waveform): The audio waveform to be processed.
   • emotion: A placeholder for additional information (optional).
   • self.emotion_map[emotion]: The numerical label corresponding to the emotion of the sample.

Citation Info

ParaCLAP has been accept at InterSpeech 2024 for presentation.

@inproceedings{Jing24_PTA,
  title     = {ParaCLAP – Towards a general language-audio model for computational paralinguistic tasks},
  author    = {Xin Jing and Andreas Triantafyllopoulos and Björn Schuller},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {1155--1159},
  doi       = {10.21437/Interspeech.2024-1315},
  issn      = {2958-1796},
}