channel-capacity-estimator 1.1

1. Calculation of mutual information (for equiprobable input distributions).

In the example below, mutual information is calculated between three sets of points drawn at random from two-dimensional Gaussian distributions, located at (0,0), (0,1), and at (3,3); covariance matrices of all three distributions are identity matrices (this is default in SciPy). Auxiliary function label_all_with() helps to prepare the list of all points, in which each point is labeled according to its distribution of origin.

>>> from scipy.stats import multivariate_normal as mvn
>>> from cce import WeightedKraskovEstimator as wke
>>>
>>> def label_all_with(label, values): return [(label, v) for v in values]
>>>
>>> data = label_all_with('A', mvn(mean=(0,0)).rvs(10000)) \
         + label_all_with('B', mvn(mean=(0,1)).rvs(10000)) \
         + label_all_with('C', mvn(mean=(3,3)).rvs(10000))
>>>
>>> wke(data).calculate_mi(k=10)
0.9552107248613955

In this example, probabilities of input distributions, henceforth referred to as weights, are assumed to be equal for all input distributions. Format of data is akin to [(‘A’, array([-0.4, 2.8])), (‘A’, array([-0.9, -0.1])), …, (‘B’, array([1.7, 0.9])), …, (‘C’, array([3.2, 3.3])), …). Entries of data are not required to be grouped according to the label. Distribution labels can be given as strings, not just single characters. Instead of NumPy arrays, ordinary lists with coordinates will be also accepted. (This example involves random numbers, so your result may vary slightly.)

2. Calculation of mutual information for input distributions with non-equal probabilities.

This example is structured as above, with an addition of weights of each input distributions:

>>> from scipy.stats import multivariate_normal as mvn
>>> from cce import WeightedKraskovEstimator as wke
>>>
>>> def label_all_with(label, values): return [(label, v) for v in values]
>>>
>>> data = label_all_with('A', mvn(mean=(0,0)).rvs(10000)) \
         + label_all_with('B', mvn(mean=(0,1)).rvs(10000)) \
         + label_all_with('C', mvn(mean=(3,3)).rvs(10000))
>>>
>>> weights = {'A': 2/6, 'B': 1/6, 'C': 3/6}
>>> wke(data).calculate_weighted_mi(weights=weights, k=10)
1.0065891280377155

(This example involves random numbers, so your result may vary slightly.)

3. Estimation of channel capacity by maximizing MI with respect to input weights.

>>> from scipy.stats import multivariate_normal as mvn
>>> from cce import WeightedKraskovEstimator as wke
>>>
>>> def label_all_with(label, values): return [(label, v) for v in values]
>>>
>>> data = label_all_with('A', mvn(mean=(0,0)).rvs(10000)) \
         + label_all_with('B', mvn(mean=(0,1)).rvs(10000)) \
         + label_all_with('C', mvn(mean=(3,3)).rvs(10000))
>>>
>>> wke(data).calculate_maximized_mi(k=10)
(1.0154510500713743, {'A': 0.33343804, 'B': 0.19158363, 'C': 0.4749783})

The output tuple contains the maximized mutual information (channel capacity) and probabilities of input distributions that maximize mutual information (argmax). Optimization is performed within TensorFlow with multiple threads and takes less than one minute on a computer with quad-core processor. (This example involves random numbers, so your result may vary slightly.)

Python Package Index

Once the environment has been created, the most convenient manner is to install the package from PyPI using:

$ pip install channel-capacity-estimator

Then, you can directly start using the package:

$ python
>>> from cce import WeightedKraskovEstimator
>>> ...

Local installation

Alternatively, you can build the package locally.

$ git clone https://github.com/pawel-czyz/channel-capacity-estimator.git
$ cd channel-capacity-estimator

To install cce locally via pip, run:

$ make install

Then, you can directly start using the package:

$ python
>>> from cce import WeightedKraskovEstimator
>>> ...