Python package for information theory.
Project description
dit is a Python package for information theory.
Introduction
Information theory is a powerful extension to probability and statistics, quantifying dependencies among arbitrary random variables in a way that is consistent and comparable across systems and scales. Information theory was originally developed to quantify how quickly and reliably information could be transmitted across an arbitrary channel. The demands of modern, datadriven science have been coopting and extending these quantities and methods into unknown, multivariate settings where the interpretation and best practices are not known. For example, there are at least four reasonable multivariate generalizations of the mutual information, none of which inherit all the interpretations of the standard bivariate case. Which is best to use is contextdependent. dit implements a vast range of multivariate information measures in an effort to allow information practitioners to study how these various measures behave and interact in a variety of contexts. We hope that having all these measures and techniques implemented in one place will allow the development of robust techniques for the automated quantification of dependencies within a system and concrete interpretation of what those dependencies mean.
Citing
If you use dit in your research, please cite it as:
@article{dit, Author = {James, R. G. and Ellison, C. J. and Crutchfield, J. P.}, Title = {{dit}: a {P}ython package for discrete information theory}, Journal = {The Journal of Open Source Software}, Volume = {3}, Number = {25}, Pages = {738}, Year = {2018}, Doi = {https://doi.org/10.21105/joss.00738} }
Basic Information
Documentation
Downloads
https://anaconda.org/condaforge/dit
Dependencies 

Optional Dependencies
colorama: colored column heads in PID indicating failure modes
cython: faster sampling from distributions
hypothesis: random sampling of distributions
matplotlib, pythonternary: plotting of various informationtheoretic expansions
numdifftools: numerical evaluation of gradients and hessians during optimization
pint: add units to informational values
scikitlearn: faster nearestneighbor lookups during entropy/mutual information estimation from samples
Install
The easiest way to install is:
pip install dit
If you want to install dit within a conda environment, you can simply do:
conda install c condaforge dit
Alternatively, you can clone this repository, move into the newly created dit directory, and then install the package:
git clone https://github.com/dit/dit.git
cd dit
pip install .
Testing
$ git clone https://github.com/dit/dit.git
$ cd dit
$ pip install r requirements_testing.txt
$ py.test
Code and bug tracker
License
BSD 3Clause, see LICENSE.txt for details.
Implemented Measures
dit implements the following information measures. Most of these are implemented in multivariate & conditional generality, where such generalizations either exist in the literature or are relatively obvious — for example, though it is not in the literature, the multivariate conditional exact common information is implemented here.
Entropies

Mutual Informations

Divergences

Other Measures


Common Informations

Partial Information Decomposition


Secret Key Agreement Bounds

Quickstart
The basic usage of dit corresponds to creating distributions, modifying them if need be, and then computing properties of those distributions. First, we import:
>>> import dit
Suppose we have a really thick coin, one so thick that there is a reasonable chance of it landing on its edge. Here is how we might represent the coin in dit.
>>> d = dit.Distribution(['H', 'T', 'E'], [.4, .4, .2])
>>> print(d)
Class: Distribution
Alphabet: ('E', 'H', 'T') for all rvs
Base: linear
Outcome Class: str
Outcome Length: 1
RV Names: None
x p(x)
E 0.2
H 0.4
T 0.4
Calculate the probability of H and also of the combination H or T.
>>> d['H']
0.4
>>> d.event_probability(['H','T'])
0.8
Calculate the Shannon entropy and extropy of the joint distribution.
>>> dit.shannon.entropy(d)
1.5219280948873621
>>> dit.other.extropy(d)
1.1419011889093373
Create a distribution where Z = xor(X, Y).
>>> import dit.example_dists
>>> d = dit.example_dists.Xor()
>>> d.set_rv_names(['X', 'Y', 'Z'])
>>> print(d)
Class: Distribution
Alphabet: ('0', '1') for all rvs
Base: linear
Outcome Class: str
Outcome Length: 3
RV Names: ('X', 'Y', 'Z')
x p(x)
000 0.25
011 0.25
101 0.25
110 0.25
Calculate the Shannon mutual informations I[X:Z], I[Y:Z], and I[X,Y:Z].
>>> dit.shannon.mutual_information(d, ['X'], ['Z'])
0.0
>>> dit.shannon.mutual_information(d, ['Y'], ['Z'])
0.0
>>> dit.shannon.mutual_information(d, ['X', 'Y'], ['Z'])
1.0
Calculate the marginal distribution P(X,Z). Then print its probabilities as fractions, showing the mask.
>>> d2 = d.marginal(['X', 'Z'])
>>> print(d2.to_string(show_mask=True, exact=True))
Class: Distribution
Alphabet: ('0', '1') for all rvs
Base: linear
Outcome Class: str
Outcome Length: 2 (mask: 3)
RV Names: ('X', 'Z')
x p(x)
0*0 1/4
0*1 1/4
1*0 1/4
1*1 1/4
Convert the distribution probabilities to log (base 3.5) probabilities, and access its probability mass function.
>>> d2.set_base(3.5)
>>> d2.pmf
array([1.10658951, 1.10658951, 1.10658951, 1.10658951])
Draw 5 random samples from this distribution.
>>> dit.math.prng.seed(1)
>>> d2.rand(5)
['01', '10', '00', '01', '00']
Contributions & Help
If you’d like to feature added to dit, please file an issue. Or, better yet, open a pull request. Ideally, all code should be tested and documented, but please don’t let this be a barrier to contributing. We’ll work with you to ensure that all pull requests are in a mergable state.
If you’d like to get in contact about anything, you can reach us through our slack channel.
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.