museval 0.3.0b1

Evaluation tools for the SIGSEP MUS database

museval

A python package to evaluate source separation results using the MUSDB18 dataset. This package is part of the MUS task of the Signal Separation Evaluation Campaign (SISEC). Using this package is mandatory for submitting results to SiSEC as it includes the reference implementation of the new BSSEval version 4.

BSSEval v4

The BSSEval metrics, as implemented in the MATLAB toolboxes and their re-implementation in mir_eval are widely used in the audio separation literature. One particularity of BSSEval is to compute the metrics after optimally matching the estimates to the true sources through linear distortion filters. This allows the criteria to be robust to some linear mismatches. Apart from the optional evaluation for all possible permutations of the sources, this matching is the reason for most of the computation cost of BSSEval, especially considering it is done for each evaluation window when the metrics are computed on a framewise basis.

For this package, we enabled the option of having time invariant distortion filters, instead of necessarily taking them as varying over time as done in the previous versions of BSS eval. First, enabling this option significantly reduces the computational cost for evaluation because matching needs to be done only once for the whole signal. Second, it introduces much more dynamics in the evaluation, because time-varying matching filters turn out to over-estimate performance. Third, this makes matching more robust, because true sources are not silent throughout the whole recording, while they often were for short windows.

Installation

Package installation

You can install the museval parsing package using pip:

pip install museval

Usage

The purpose of this package is to evaluate source separation results and write out standardized json files that can easily be parsed by the SiSEC submission system. Furthermore we want to encourage users to use this evaluation output format as the standardized way to share source separation results for processed tracks. We provide two different ways to use museval in conjunction with your source separation results.

Run and Evaluate

• If you want to perform evaluation while processing your source separation results, you can hook museval into your musdb user_function:

Here is an example for such a function separating the mixture into a vocals and accompaniment track:

import musdb
import museval

output_dir = ...
estimates_dir = ...

def estimate_and_evaluate(track):
    # generate your estimates
    estimates = {
        'vocals': track.audio,
        'accompaniment': track.audio
    }

    # Evaluate using museval
    scores = museval.eval_mus_track(
        track, estimates, output_dir=output_dir
    )

    # print nicely formatted mean scores
    print(scores)

    # return estimates as usual
    return estimates

# your usual way to run musdb
mus = musdb.DB()
[estimate_and_evaluate(track) for track in mus]

• Make sure output_dir is set. museval will recreate the musdb file structure in that folder and write the evaluation results to this folder. This whole folder should be submitted for your SiSEC contribution.

Evaluate later

If you have already computed your estimates, we provide you with an easy-to-use function to process evaluation results afterwards.

Simply use the museval.eval_mus_dir to evaluate your estimates_dir and write the results into the output_dir. For convenience, the eval_mus_dir function accepts all parameters of the musdb.run(). That way e.g. multiprocessing can easily be enabled by setting parallel=True:

import musdb
import museval

# initiate musdb
mus = musdb.DB()

# evaluate an existing estimate folder with wav files
museval.eval_mus_dir(
    dataset=mus,  # instance of musdb
    estimates_dir=...,  # path to estimate folder
    output_dir=...,  # set a folder to write eval json files
    subsets="Test",
    is_wav=False
)

:bulb: When evaluating later, please make sure you use the same environment used for separation or use the decoded wav dataset. This is important since the reference sources are loaded from the stems on the fly and certain FFMPEG version produce different zero-padding. We tested several different machines and ffmpeg version and did not run into any problems, but we cannot guarantee that the decoded outputs of two different ffmpeg versions are identical and would not affect the bsseval scores. E.g. when silence > 512 samples would be added in the beginning of a target source.

Aggregate and Analyze Scores

Scores for each track can also be aggregated in a pandas DataFrame for easier analysis or the creation of boxplots. To aggregate multiple tracks in a DataFrame, create museval.EvalStore() object and add the track scores successively.

results = museval.EvalStore(frames_agg='median', tracks_agg='median')
for track in tracks:
    # ...
    results.add_track(museval.eval_mus_track(track, estimates))

When all tracks have been added, the aggregated scores can be shown using print(results) and results may be saved as a pandas DataFrame results.save('my_method.pandas').

To compare multiple methods, create a museval.MethodStore() object add the results

methods = museval.MethodStore()
methods.add_evalstore(results, name="XZY")

To compare against participants from SiSEC MUS 2018, we provide a convenient method to load the existing scores on demand using methods.add_sisec18(). For the creation of plots and statistical significance tests we refer to our list of examples.

Commandline tool

We provide a command line wrapper of eval_mus_dir by calling the museval command line tool. The following example is equivalent to the code example above:

museval -p --musdb path/to/musdb -o path/to/output_dir path/to/estimate_dir

:bulb: you use the --iswav flag to use the decoded wav musdb dataset.

Using Docker for Evaluation

If you don't want to set up a Python environment to run the evaluation, we would recommend to use Docker. Assuming you have already computed your estimates and installed docker in your machine, you just need to run the following two lines in your terminal:

1. Pull Docker Container

Pull our precompiled sigsep-mus-eval image from dockerhub:

docker pull faroit/sigsep-mus-eval

2. Run evaluation

To run the evaluation inside of the docker, three absolute paths are required:

• estimatesdir will stand here for the absolute path to the estimates directory. (For instance /home/faroit/dev/mymethod/musdboutput)
• musdbdir will stand here for the absolute path to the root folder of musdb. (For instance /home/faroit/dev/data/musdb18)
• outputdir will stand here for the absolute path to the output directory. (For instance /home/faroit/dev/mymethod/scores)

We just mount these directories into the docker container using the -v flags and start the docker instance:

docker run --rm -v estimatesdir:/est -v musdbdir:/mus -v outputdir:/out faroit/sigsep-mus-eval --musdb /mus -o /out /est

In the line above, replace estimatesdir, musdbdir and outputdir by the absolute paths for your setting. Please note that docker requires absolute paths so you have to rely on your command line environment to convert relative paths to absolute paths (e.g. by using $HOME/ on Unix).

:warning: museval requires a significant amount of memory for the evaluation. Evaluating all five targets for MUSDB18 may require more than 4GB of RAM. If you use multiprocessing by using the -p switch in museval, this results in 16GB of RAM. It is recommended to adjust your Docker preferences, because the docker container might just quit if its out of memory.

Submission

Please refer to our Submission site.

References

A. If you use the museval in the context of source separation evaluation comparing a method it to other methods of SiSEC 2018, please cite

@InProceedings{SiSEC18,
  author="St{\"o}ter, Fabian-Robert and Liutkus, Antoine and Ito, Nobutaka",
  title="The 2018 Signal Separation Evaluation Campaign",
  booktitle="Latent Variable Analysis and Signal Separation:
  14th International Conference, LVA/ICA 2018, Surrey, UK",
  year="2018",
  pages="293--305"
}

B. if you use the software for any other purpose, you can cite the software release