ConfMatrixCalc 0.6.0

Statistical Analysis of Phoneme Confusion Matrices

Package ConfMatrixCalc implements probabilistic Bayesian analysis of phoneme identification test results. The analysis approach was presented and validated in (Leijon et al., 2016).

Phoneme identification tests are used, for example, to evaluate the detailed ("microscopic") speech-recognition ability of listeners using two or more different hearing aids or other sound-transmission instruments or algorithms. Phoneme identification performance is often tested using nonsense "words" with a fixed structure, e.g., CVC, VCV, or CVCVC, where C is a consonant and V is a vowel. This makes the test material more difficult than real words or sentences, because the listener can not make use of prior lexical and semantic knowledge. However, this may actually be an advantage, because interesting test results can be obtained at realistic speech-to-noise ratios, where listeners might otherwise get nearly perfect identification results with an easier test material.

Early speech research showed that the phoneme identification ability is correlated with general sentence understanding (Fletcher and Steinberg, 1929, Fig. 11).

Phoneme Confusion Matrices

The test results are usually recorded as two-dimensional arrays of confusion counts. A matrix element with index (s, r) shows how many times the listener responded by the rth category, when the sth stimulus was presented.

The statistical analysis of confusion-matrix data is non-trivial, because the matrix is usually quite sparse for each listener. For example, in a consonant-identification test with 16 consonants, each stimulus type might be presented, say, five times, i.e., 80 presentations in total. Then each matrix row will have at least 16 - 5 = 11 elements with a zero count. This makes it difficult to estimate underlying response probabilities and to quantify the statistical reliability of observed test results. The Bayesian analysis method handles these problems in a coherent manner.

Analysis Results

1. Overall performance is indicated by two measures, each with a credible range to indicate the uncertainty of the estimate:

   1. Probability of Correct identification (PC), across all presented phonemes.

   2. The Mutual Information (MI) between stimulus and response (Miller and Nicely, 1955), sometimes called "transmitted information". This measure indicates the average amount of information about the stimulus category, received by the listener by hearing each presented phoneme.

2. Detailed performance is shown by credible confusion pattern, i.e., a set of stimulus-response pairs where listeners' response probabilities are jointly credibly different between test conditions.

The Bayesian model is hierarchical. The package estimates predictive distributions of results for

• a random individual in the population from which participants were recruited,
• each individual in the group of test participants.

Phoneme Identification Experiments

The package can analyse data from simple or rather complex experimental designs, including the following features:

1. Phoneme identification data may be collected in one or more Test Conditions. Each test condition may be a combination of categories from one or more Test Factors. For example, the main test factor may be Hearing Aid, with categories A, B, or Unaided. Another test factor may be, e.g., Background, with categories Quiet, or Noisy. A third factor may be Position, with categories C1 or C2, indicating the consonant position in CVC nonsense words. The analysis shows credible differences between categories within the first (main) test factor, for each combination of categories in other (secondary) test factors.

2. One or more Listener Groups may be included. The analysis shows systematic differences between groups.

3. The analysis model does not require anything about the number of test presentations for each phoneme category. The validation (Leijon et al., 2016) showed that reliable results could be derived with as few as five presentations per phoneme. The analysis estimates the statistical credibility of all observed results, given the amount of collected data.

Package Documentation

General information is given in the package doc-string that may be accessed by command help(ConfMatrixCalc).

Specific information about the organisation and accepted formats of input data files is presented in the doc-string of module cm_data, accessible via help(ConfMatrixCalc.cm_data).

After running an analysis, the logging output briefly explains the analysis results presented in figures and tables.

Usage

1. Install the most recent package version: python3 -m pip install --upgrade ConfMatrixCalc

2. Copy the template script run_cm.py to your work directory, rename it, and edit the copy as guided by comments in the template, to specify

   • your experimental layout,
   • the top input data directory,
   • a directory where all output result files will be stored.

3. Run your edited script: python3 run_my_cm.py.

Requirements

This package requires Python 3.6 with Numpy, Scipy, and Matplotlib, as well as a support package samppy, and the Openpyxl package for reading data from Excel workbook documents. The pip installer will check and install the required packages if needed.

References

A. Leijon, G. E. Henter, and M. Dahlquist (2016). Bayesian analysis of phoneme confusion matrices. IEEE Trans Audio, Speech, and Language Proc 24(3):469–482. doi: 10.1109/TASLP.2015.2512039.

G. A. Miller and P. E. Nicely (1955). An analysis of perceptual confusions among some English consonants. J Acoust Soc Amer 27(2):338–352, 1955. doi: 10.1121/1.1907526.

H. Fletcher and J. Steinberg (1929). Articulation testing methods. Bell System Technical Journal 8:806–854. doi: 10.1002/j.1538-7305.1929.tb01246.x.

This Python package is a re-implementation and generalization of a similar MatLab package, developed by Arne Leijon for ORCA Europe, Widex A/S, Stockholm, Sweden. The MatLab development was financially supported by Widex A/S, Denmark.