PyTimbre 0.7.7a0

Python conversion of Timbre Toolbox

PyTimbre

PyTimbre is a Python conversion of the Matlab package Timbre Toolbox, found here: (https://github.com/VincentPerreault0/timbretoolbox).

This package was created in association with work conducted at the United States Air Force Research Laboratory on human perception of sound. Generally, models of perception have focused on sound pressure level spectra, and time histories of sound pressure. But auditory detection, identification/classification, and localization may be described better by attributes of the sound that are more based on perceptual studies of signals.

The concept of Timbre has been used in music perception research for many years. In 2011, Geoffroy Peeters compiled a collection of spectro-temporal attributes (features calculated on a frequency spectrum, that varies with time), and temporal attributes (features calculated on the waveform, or waveform's envelop). This paper forms the basis for the Matlab toolbox referenced above.

Though the Matlab coding functioned, it was cumbersome as Matlab is not a true object-oriented language. This conversion has sought to integrate the calculation of the various timbre auditory features with the structure of classes and provides a more robust method for extension of these ideas and concepts than was available within the Matlab version.

In addition, a generic time waveform object (Waveform) was provided to represent the time signal. From this class, a child class is derived to read wave files. This derived class permits the reading of multiple types of wav files ( canonical, and non-canonical) with bit rates from 8-, 16-, 24-, and 32-bit. Also included are interface methods for reading and adding meta-data that is similar to the MP3 tagging and assists in organizing the audio files by more than name or date.

Over the course of research at the United States Air Force Research Laboratory a number of other features were determined to be of interest for the use of the PyTimbre toolbox. In effort to unify these different extraction methods with the data that PyTimbre represents, the tool kits were added to the requirements list and added as properties of the various classes.

Initially the sound quality metrics found in Mosqito project (https://github.com/Eomys/MoSQITo) were integrated into the interface. However, these functions became less stable across the variety of signals of interest to the Air Force researchers. The timbral_models (https://pypi.org/project/timbral_models/) were integrated into PyTimbre to replace the roughness, sharpness, and loudness calculations. In addition, these codes added a number of other metrics. But, the original code relied on a number of libraries that were available, but cumbersome for off-line installation like soundfile and librosa. As a result the code was integrated into the PyTimbre interface as many of the features within these libraries were already available within PyTimbre.

In addition to this code, PyTimbre has taken the code from libf0 (https://github.com/groupmm/libf0) that assists with the computation of the fundamental frequency. There are a number of methods that exist within the libf0 module, but the package contains a number of dependencies that are not required for the other calculations and computations within PyTimbre. To facilitate the calculation of the fundamental frequency, the swipe class and associated functions were extracted and placed into PyTimbre. The code was extracted from the 1.0.2 version of the code on 1 April 2023. This is based on the report: Justin Salamon and Emilia Gomez: Melody Extraction From Polyphonic Music Signals Using Pitch Contour Characteristics, IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 6, pp. 1759–1770, 2012. Unfortunately, this method became unstable with further development and required the use of librosa. Due to these limitations, the yin function replaced the swipe function to determine the fundamental frequency.

Additionally, this version of PyTimbre included the code for the determination of waveform clipping taken from clipdetect (https://pypi.org/project/clipdetect/#description). In similar manner as libf0, there were dependencies that were required by this package that increased the load of the PyTimbre installation. This code is incorporated within the Waveform class to make it available to all Waveform children classes. Usage of this code can be referenced this paper: Hansen, John H. L., Allen Stauffer, and Wei Xia. Nonlinear Waveform Distortion: Assessment and Detection of Clipping on Speech Data and Systems. Speech Communication 134 (2021): 20–31.

The code within this package has been used in a variety of research publications. These range from loading and saving audio data, to using the features extracted from the spectral time histories to model classification and detection of aircraft.

Usage Example

1. Defining a waveform from an array of values

from pytimbre.waveform import Waveform

fs = 48000
w = 2 * np.pi * f
t = np.arange(0, 10, 1/fs)

wfm = Waveform(0.75 * np.sin(w*t), fs, 0.0)

2. Define a waveform from a wav file

from pytimbre.audio_files.wavefile import WaveFile
wfm = wave_file(filename)

3. Obtain global temporal attributes

from pytimbre.audio_files.wavefile import WaveFile

wfm = WaveFile(filename)
print(wfm.amplitude_modulation)

4. Create single spectrogram and get a feature

from pytimbre.audio_files.wavefile import WaveFile
from pytimbre.spectra import SpectrumByFFT

wfm = wave_file(filename)
spectrum = SpectrumByFFT(wfm)
print(spectrum.spectral_roll_off)

Clearance review and publication permission

This software was developed in conjunction with research into the human perception of sound at the 711th Human Performance Wing, Airman Systems Directorate.

It is approved for Distribution A, 88ABW-2020-2147.

A series of audio files employed in classification research within the wing are provided for testing and examples of how to use the interface.

PyTimbre

Development History

2020

April

• Added logic to avoid adding the STFT when it is not within the input configuration
• Added logic to generate and evaluate functions to skip over elements that are not included, or included as None
• Added try catch to trap error on linear algebra analysis
• Added a new function to create a column-wise output file for the results.
• Edited the calculation of the Attack so that if it returns a single value rather than the expected array, it will continue to process the data.
• Added logic to put -9999 in place of the NaN while exporting data that was not calculated

May

• Added logic to ensure that the index is always less than the length of the vector (lines 967 of StaticMethods)

July

• Added the History.md file back into the repository, and changed the column widths in every file to the limit within PyCharm.

October

• Added comments and details regarding classes.

2021

May

• Added the average statistic to the output dictionary

• Added logic to the determination of the Attack start and stop in cases when there is no array matching the constraints.

June

• Added functions to SoundFile to facilitate easier use of the class. Created an 'all' and new 'default' configuration. Added the ability to return only a specific metric from the new 'process' function.
• Updated calculation within Harmonic representation to address potential for index violations.

July

• Updated the default_configuration function

• Updated document string and other formatting issues within the source code.

• Initial design of new interface complete. This design removes the MATLAB style 'eval' command and creates the parent-child classes for the various spectro-temporal representations. The entirety of the feature extraction is contained within the parent class and provides the interface for interaction with the pytimbre classes.

August

• Added a function to the AudioSignal that permits the definition of the names of the features prior to running the analysis. This assists with the definition of the DataFrame that might hold the data.

2022

May

• Added a refactored version of the generic_time_waveform object from external sources. This provides the basis for calculating the features on the waveform
• Added a refactored version of the wav_file class that permits the reading/writing of most wav files that have canonical format. This class also includes the ability to read/write additional chunks for the addition of meta-data to the wave file.

June

• Updated the interface with the deprecation module. This permits the inclusion of the former methods, but marks them with deprectation decoration.
• Added tests for the various new methods for representation of the temporal and spectral characteristics.
• Updated the Data_Chunk to return a single dimension vector when reading a mono wav file.
• Added functions to create the dictionary of featres from a Spectrum object
• Numerous minor fixes to ensure the functioning of the system for the calculation of the various features

August

• Added files from another library to provide test cases for the various shared classes
• Updated the spectrum file with a spectrogram class that creates a short time Fourier transform in a manner similar to the spectrogram function from scipy but in accordance with the Matlab Timbre toolbox code.
• Removed the time history calculation ability
• Moved the chunk classes that support the WaveFile class to the same file.
• Create test cases for the single frequency sine wave. This test case was pushed through the MATLAB version of the toolbox. All spectral features, but the spectral variance, were checked against these results.

2023

January

• Fixed use of automatic headers on history and readme files.
• Added a property to the spectrogram class that returns a dictionary of the various temporal and spectral features, averaged in time.
• Began adding the hooks for inclusion of the mosqito sound quality metrics for extension of the auditory features into this realm.
• Added acoustic weighting and fractional octave band classes to begin facilitation of the inclusion of sound pressure level determination based on FFT and digital filters.
• Added elements of other AFRL spectral calculations in preparation for an increase of the minor revision number.
• Renamed the spectrum.py and time_history.py files to their plural since they contain multiple classes.

February

• There was an issue reading StandardBinaryFile formatted files from another organization. A significant amount of restructuring was conducted on the constructor of this class to permit multiple variations on the required data within the header.
• The get_features function in the Spectrum class did not return all the values of the spectral features in the dictionary. This was updated and a new version pushed.
• Added a function to wrap up the creation of the generic time history with fractional octave resolution calculated from the Narrowband spectra
• Edited the TimeHistory and NarrowbandTimeHistory classes to insert the header, correctly set the integration time on loading the file, and run new tests.
• Changed how WaveFile.audio_info returns the data
• Created a property to obtain the time and spectral Timbre features, sound quality metrics, and level values for a given TimeHistory

March

• Implemented function to scale current waveform based on a calibration signal passed to a function
• Implemented the various forms of equivalent level calculations
• Implemented clipping detection using an external dependency
• Updated the information within the WaveFile to support writing multi-channel audio files to disk
• Re-incorporated classes to read audio files generated as a pre-cursor to the StandardBinaryFile

April

• Version 0.6.4
• Version 0.6.3 integrated the libf0 library. After attempting to integrate this library directly, it was determined that using the raw code would be a better implementation for this package.
• The swipe.py file was taken from version 1.0.2 on 1 April 2023 to provide the fundamental frequency
• Version 0.6.7
• Updates to the calculation of the fundamental frequency
• Updates to the calculation of the harmonic features
• Updates to methods to get the features from the spectrum and time history objects

May

• Added a flag in the constructor of the waveform to remove the finding the maximum of the waveform
• Created the initial version of the Sphinx documentation for the module.
• Integrated code for the reading of impulse waveforms into the Waveform class.
• Integrated code the correct calculation of spectral levels from an impulse waveform into SpectrumByFFT
• Updated the get_features function to return different data in the DataFrame when the waveform is labeled as an impulse waveform.

June

• Updated trim method to copy properties that could have been altered from the defaults in the constructor
• Reduced requirements for standard packages
• Worked on temporal features for EEG data that possesses significantly lower sample rates than audio data
• Added a property to set the method for the thresholding required in the attack determination
• Updated the tests so that they will execute with python setup.py test
• Incremented to 0.7.1 with the completion of all issues slated for the 0.7.1 increment.
• Incremented to 0.7.2
• Added spectrogram class that uses the spectrogram function in scipy.signal to create the matrix of sound pressure levels
• Added a plotting function to the time history class.

July

• Added speech features from python_speech_features
• Fixed errors in start_time function within WaveFile class

August

• Cleaned up some code in Waveform
• Added a split_by_time function that mimics the split function in Numpy
• Updated the calculation of single number metrics. Some were calculated in frequency spectra, others on time windows. To compensate for this, a new temporal metrics namespace was created. Tests were added and elements of the acoustic weights namespace were moved because the are more appropriately in the temporal metric namespace
• Added code to permit the reading of Wavefiles that do not have a LIST chunk, or do not have a 'creation_date' element in the LIST chunk
• Cleaned up some Docustrings.
• Integrated many of the functions from the timbral_models to enable better calculation of the various sound quality metrics that exist.
• With the integration of the timbral_models, the dependency on mosqito was removed

MIT License

Copyright (c) 2021 Frank Mobley, Gregory Bowers

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.