Communication Dans Un Congrès Année : 2025

Investigating the Sensitivity of Pre-trained Audio Embeddings to Common Effects

Résumé

In recent years, foundation models have significantly advanced data-driven systems across various domains. Yet, their underlying properties, especially when functioning as feature extractors, remain under-explored. In this paper, we investigate the sensitivity to audio effects of audio embeddings extracted from widely-used foundation models, including OpenL3, PANNs, and CLAP. We focus on audio effects as the source of sensitivity due to their prevalent presence in large audio datasets. By applying parameterized audio effects (gain, low-pass filtering, reverberation, and bitcrushing), we analyze the correlation between the deformation trajectories and the effect strength in the embedding space. We propose to quantify the dimensionality and linearizability of the deformation trajectories induced by audio effects using canonical correlation analysis. We find that there exists a direction along which the embeddings move monotonically as the audio effect strength increases, but that the subspace containing the displacements is generally high-dimensional. This shows that pre-trained audio embeddings do not globally linearize the effects. Our empirical results on instrument classification downstream tasks confirm that projecting out the estimated deformation directions cannot generally improve the robustness of pre-trained embeddings to audio effects.
Fichier principal
Vignette du fichier
main.pdf (791.35 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04904470 , version 1 (22-01-2025)

Licence

Copyright (Tous droits réservés)

Identifiants

  • HAL Id : hal-04904470 , version 1

Citer

Victor Deng, Changhong Wang, Gael Richard, Brian McFee. Investigating the Sensitivity of Pre-trained Audio Embeddings to Common Effects. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Apr 2025, Hyderabad, India. ⟨hal-04904470⟩
0 Consultations
0 Téléchargements

Partager

More