ConfMatrixCalc 0.7.0

Statistical Analysis of Phoneme Confusion Matrices

Package ConfMatrixCalc implements probabilistic Bayesian analysis of test results recorded in the form of confusion matrices.

The package was developed for analyzing phoneme-identification speech test results, but it can be used to analyze confusion-matrix data from any type of multi-category classification, performed by either humans or machines. The analysis approach was presented and validated in (Leijon et al., 2016).

Phoneme identification tests are used, for example, to evaluate the detailed ("microscopic") speech-recognition ability of listeners using two or more different hearing aids or other sound-transmission instruments or algorithms. Phoneme identification performance is often tested using nonsense "words" with a fixed structure, e.g., CVC, VCV, or CVCVC, where C is a consonant and V is a vowel. This makes the test material more difficult than real words or sentences, because the listener can not make use of prior lexical and semantic knowledge. However, this may actually be an advantage, because interesting test results can be obtained at realistic speech-to-noise ratios, where listeners might otherwise get nearly perfect identification results with an easier test material.

Early speech research showed that the phoneme identification ability is correlated with general sentence understanding (Fletcher and Steinberg, 1929, Fig. 11).

Confusion Matrices

The test results are usually recorded as two-dimensional arrays of confusion counts. A matrix element with index (s, r) shows how many times the listener responded by the rth category, when the sth stimulus was presented.

The statistical analysis of confusion-matrix data is non-trivial, because the matrix is usually quite sparse for each listener. For example, in a consonant-identification test with 16 consonants, each stimulus type might be presented, say, five times. Then each matrix row will have 11 - 15 elements with a zero count. This makes it difficult to estimate underlying response probabilities and to quantify the statistical reliability of observed test results. The Bayesian analysis method handles these problems in a coherent manner.

Analysis Results

1. Overall performance is indicated by two measures, each with a credible range to indicate the uncertainty of the estimate, and credible differences between Test Conditions:

   1. Probability of Correct identification (PC), across all presented test stimuli.

   2. The Mutual Information (MI) between stimulus and response (Miller and Nicely, 1955), sometimes called "transmitted information". This measure indicates the average amount of information about the stimulus category, received by the listener by hearing each presented phoneme.

2. Detailed performance is shown by credible confusion pattern, i.e., a set of stimulus-response pairs where listeners' response probabilities are jointly credibly different between test conditions.

The Bayesian model is hierarchical. The package estimates predictive distributions of results for

• a random individual in the population from which participants were recruited,
• each individual in the group of test participants.

Phoneme Identification Experiments

The package can analyze data from simple or rather complex experimental designs, including the following features:

1. Phoneme identification data may be collected in one or more Test Conditions. Each test condition may be a combination of categories from several Test Factors. For example, the main test factor may be Hearing Aid, with categories A, B, or Unaided. Another test factor may be, e.g., Background, with categories Quiet, or Noisy. A third factor may be Position, with categories C1 or C2, indicating the consonant position in CVC nonsense words. The analysis shows credible differences between categories within the first (main) test factor, for each combination of categories in other (secondary) test factors.

2. One or more Listener Groups may be included. The analysis shows credible differences between groups.

3. The analysis model does not require anything about the number of test presentations for each phoneme category. The validation (Leijon et al., 2016) showed that reliable results could be derived with as few as five presentations per phoneme. The analysis estimates the statistical credibility of all results, given the limited amount of collected data.

Package Documentation

General information is given in the package doc-string that may be accessed by command help(ConfMatrixCalc).

Specific information about the organization and accepted formats of input data files is presented in the doc-string of module cm_data, accessible via help(ConfMatrixCalc.cm_data). The current version can read data from Excel workbook files or from json-serialized text files with a special format.

After running an analysis, the logging output briefly explains the analysis results presented in figures and tables.

Usage

1. Install the most recent package version: python3 -m pip install --upgrade ConfMatrixCalc

2. Copy the template script run_cm.py to your work directory, rename it, and edit the copy as guided by comments in the template, to specify

   • your experimental layout,
   • the top input data directory,
   • a directory where all output result files will be stored.

3. Run your edited script: python3 run_my_cm.py.

Requirements

This package requires Python 3.6 or later with Numpy, Scipy, and Matplotlib, as well as a support package samppy, and the Openpyxl package for reading data from Excel workbook documents. The pip installer will check and install the required packages if needed.

However, it does not check the version of those packages if already installed. In case of some error, try upgrading the required packages manually, e.g., as python3 -m pip install --upgrade matplotlib

References

A. Leijon, G. E. Henter, and M. Dahlquist (2016). Bayesian analysis of phoneme confusion matrices. IEEE Trans Audio, Speech, and Language Proc 24(3):469–482. doi: 10.1109/TASLP.2015.2512039.

G. A. Miller and P. E. Nicely (1955). An analysis of perceptual confusions among some English consonants. J Acoust Soc Amer 27(2):338–352, 1955. doi: 10.1121/1.1907526.

H. Fletcher and J. Steinberg (1929). Articulation testing methods. Bell System Technical Journal 8:806–854. doi: 10.1002/j.1538-7305.1929.tb01246.x.

This Python package is a re-implementation and generalization of a similar MatLab package, developed by Arne Leijon for ORCA Europe, Widex A/S, Stockholm, Sweden. The MatLab development was financially supported by Widex A/S, Denmark.