Skip to main content

dit is a Python package for discrete information theory.

Continuous Integration Status Test Coverage Status Documentation Status Conda installation

JOSS Status DOI dit chat

Introduction

Information theory is a powerful extension to probability and statistics, quantifying dependencies among arbitrary random variables in a way that is consistent and comparable across systems and scales. Information theory was originally developed to quantify how quickly and reliably information could be transmitted across an arbitrary channel. The demands of modern, data-driven science have been coopting and extending these quantities and methods into unknown, multivariate settings where the interpretation and best practices are not known. For example, there are at least four reasonable multivariate generalizations of the mutual information, none of which inherit all the interpretations of the standard bivariate case. Which is best to use is context-dependent. dit implements a vast range of multivariate information measures in an effort to allow information practitioners to study how these various measures behave and interact in a variety of contexts. We hope that having all these measures and techniques implemented in one place will allow the development of robust techniques for the automated quantification of dependencies within a system and concrete interpretation of what those dependencies mean.

Citing

If you use dit in your research, please cite it as:

@article{dit,
  Author = {James, R. G. and Ellison, C. J. and Crutchfield, J. P.},
  Title = {{dit}: a {P}ython package for discrete information theory},
  Journal = {The Journal of Open Source Software},
  Volume = {3},
  Number = {25},
  Pages = {738},
  Year = {2018},
  Doi = {https://doi.org/10.21105/joss.00738}
}

Basic Information

Documentation

https://docs.dit.io

Downloads

https://pypi.org/project/dit/ and https://anaconda.org/conda-forge/dit

Code and bug tracker

https://github.com/dit/dit

License

BSD 3-Clause, see LICENSE.txt for details.

Dependencies

Install

The easiest way to install is:

pip install dit

If you want to install dit within a conda environment, you can simply do:

conda install -c conda-forge dit

Optional features are grouped into extras:

Extra

Provides

optional

matplotlib (plots), plotly (interactive plots), colorama (colored PID tables), PLTable (double- instead of single-line table borders when dit.ditParams['text.font'] = 'linechar'), numdifftools (numerical gradients/Hessians), pint (units), pycddlib-standalone + pypoman (polytope vertex enumeration), scikit-learn (fast nearest-neighbor estimators)

jax

JAX optimization backend with autodiff (jax, jaxlib, optimistix, slsqp-jax)

torch

PyTorch optimization backend with autodiff and GPU support

pytensor

PyTensor optimization backend with autodiff

symbolic

sympy-valued (symbolic) distributions

broja

ECOS cone solver for the BROJA partial information decomposition

music

music21, for the empirical Bach / Corelli example distributions

test

pytest, hypothesis, and plugins for running the test suite

docs

Sphinx and friends for building the documentation

dev

everything above, plus ruff and ty

For example:

pip install "dit[optional,jax]"

For development, we recommend uv:

git clone https://github.com/dit/dit.git
cd dit
uv sync --extra dev

This installs dit in editable mode with all development dependencies (tests, docs, linting, type checking, and optional backends).

Testing

# Using uv (recommended)
uv run pytest

# Or with pip
pip install -e ".[test]"
pytest

By default the test suite skips tests marked slow and very_slow. To run them as well, override the marker selection:

uv run pytest -m ""

Building the documentation

uv run sphinx-build -W --keep-going -b html docs docs/_build/html

Implemented Measures

dit implements the following information measures and tools. Most measures are implemented in multivariate and conditional generality, where such generalizations either exist in the literature or are relatively obvious — for example, though it is not in the literature, the multivariate conditional exact common information is implemented here. Each heading links to the relevant section of the documentation.

Entropies

Shannon entropy (joint and conditional) · Rényi entropy · Tsallis entropy · residual entropy / independent information / variation of information · necessary conditional entropy · cumulative residual entropy (generalized and conditional) · extropy · perplexity · negentropy · disequilibrium and LMPR complexity · α-synergy entropy spectra (synergistic, unique, union, intersection, and self-synergy entropies)

Multivariate mutual informations

mutual information · co-information · interaction information · total correlation / multi-information · dual total correlation / binding information (and its generalization) · CAEKL mutual information · O-information · S-information · cohesion · Kirkwood mutual information · frustration (order-k and pairwise) · transmission · DeWeese mutual informations · cross mutual informations · union information · Quax synergy · synergistic disclosure (and its modified and backbone variants) · logarithmic decomposition · \(\Delta^k\) / \(\Gamma^k\) · TSE complexity · α-synergy information spectra

Common informations

Gács–Körner · stochastic Gács–Körner · Wyner · exact · functional · MSS · Kamath–Anantharam (and its directed variant) · β common information · tension common information · channel synthesis rate · Salamatian–Cohen–Médard maximum entropy function

Other measures

lautum information · Sibson mutual information (and its conditional variants) · maximal leakage

Divergences

Kullback–Leibler divergence / relative entropy · cross entropy · Jensen–Shannon divergence · variational distance · Hellinger distance and divergence · Bhattacharyya coefficient · Chernoff information · Rényi, Tsallis, α-, and f-divergences · earth mover’s distance · coupling metrics · copy mutual information · maximum correlation · hypercontractivity coefficient

Secret key agreement

secret key capacity (multiterminal, with omniscience rate) · no-communication, one-way, and two-way secret key agreement rates · secrecy capacity · intrinsic, reduced intrinsic, minimal intrinsic, two-part intrinsic, relaxed two-part, upper, lower, necessary, interactive, and less-noisy intrinsic mutual informations (with total correlation, dual total correlation, and CAEKL variants) · iterated discarding · secret key cost / information of formation

Partial information decomposition

\(I_{min}\) · \(I_{MMI}\) · \(I_{\wedge}\) · \(I_{proj}\) · \(I_{BROJA}\) · \(I_{ccs}\) · \(I_{\pm}\) · \(I_{sx}\) · \(I_{dep}\) · \(I_{RA}\) · \(I_{RAV}\) · \(I_{RR}\) · \(I_{RDR}\) · \(I_{CT}\) · \(I_{deg}\) · \(I_{MC}\) · \(I_{LN}\) · \(I_{GH}\) · \(I_{IG}\) · \(I_{IPID}\) · \(I_{\delta}\) · \(I_{\delta\Lambda}\) · \(I_{ME}\) · \(I_{\prec}\) · \(I_{do}\) · \(I_{SKAR}\) (four variants) · \(I_{\cup}^{CI}\) (union-information synergy) · partial entropy decompositions (\(H_{cs}\), \(H_{MOS}\), \(H_{sx}\)) · synergistic disclosure decomposition · integrated information decomposition (ΦID) · generalized information decomposition (GID) · representational complexity

Profiles and visualization

Shannon and extropy partitions (information diagrams) · complexity profile · marginal utility of information · connected informations (Schneidman, dual, and m-flat / Amari) · marginal lift profile · binding mixture / shared randomness decomposition · dependency decompositions (and dual) · Shapley decompositions · entropy triangles · information UpSet plots

Channels and coding

channel capacity (certified, accelerated Blahut–Arimoto) · capacity–cost function · Sibson capacity · a catalog of example channels · channel orderings (degraded, less noisy, more capable, Blackwell, Shannon inclusion) and Le Cam / KL deficiencies · source codes (Shannon, Fano, Shannon–Fano–Elias, Huffman, length-limited Huffman, Golomb/Rice, Tunstall, Elias, Fibonacci, unary, polar) · channel codes (repetition, parity-check, Hamming, Reed–Muller, Golay, LDPC, polar, convolutional)

Rate-distortion and Gray–Wyner

rate-distortion curves · information bottleneck (generalized, deterministic, and divergence variants) · privacy funnel (and perfect privacy) · Gray–Wyner network, region, and extension profile · lossy Wyner common information · Körner graph entropy · maximal and private interaction informations · excess functional information

Estimation

distributions from data and time series · 17 plug-in and bias-corrected entropy estimators (Miller–Madow, Grassberger, Chao–Shen, NSB, Zhang, shrinkage, Bayesian, jackknife, …) · alphabet-aware conditional mutual information · k-NN differential entropy and KSG total correlation · permutation entropy · entropy rates (block, Lempel–Ziv, Bayesian posterior) · Markov order tests · surrogate data

Optimization and constructions

maximum entropy distributions · iterative proportional fitting · m-projections · mixtures of products · marginal lifts · convex maximization over polytopes · minimal sufficient statistics and information trimming · lattice joins and meets · Kirkwood, Lancaster, and Fejér–Riesz approximations · symbolic (sympy) distributions · JAX, PyTorch, and PyTensor backends

Quickstart

The basic usage of dit corresponds to creating distributions, modifying them if need be, and then computing properties of those distributions. First, we import:

>>> import dit

Suppose we have a really thick coin, one so thick that there is a reasonable chance of it landing on its edge. Here is how we might represent the coin in dit. Each outcome is a tuple with one entry per random variable; unnamed variables are called X0, X1, ….

>>> d = dit.Distribution(['H', 'T', 'E'], [.4, .4, .2])
>>> print(d)
Class:    Distribution
Alphabet: (('E', 'H', 'T'),)
Base:     linear
<BLANKLINE>
x        p(X0)
('E',)   0.2
('H',)   0.4
('T',)   0.4

Calculate the probability of H and also of the combination H or T.

>>> d['H']
0.4
>>> d.event_probability(['H', 'T'])
0.8

Calculate the Shannon entropy and extropy of the joint distribution.

>>> float(dit.shannon.entropy(d))
1.5219280948873621
>>> float(dit.other.extropy(d))
1.1419011889093373

Create a distribution where Z = xor(X, Y).

>>> import dit.example_dists
>>> d = dit.example_dists.Xor()
>>> d.set_rv_names(['X', 'Y', 'Z'])
>>> print(d)
Class:    Distribution
Alphabet: (('0', '1'), ('0', '1'), ('0', '1'))
Base:     linear
<BLANKLINE>
x                 p(X,Y,Z)
('0', '0', '0')   0.25
('0', '1', '1')   0.25
('1', '0', '1')   0.25
('1', '1', '0')   0.25

Calculate the Shannon mutual informations I[X:Z], I[Y:Z], and I[X,Y:Z].

>>> float(dit.shannon.mutual_information(d, ['X'], ['Z']))
0.0
>>> float(dit.shannon.mutual_information(d, ['Y'], ['Z']))
0.0
>>> float(dit.shannon.mutual_information(d, ['X', 'Y'], ['Z']))
1.0

Calculate the marginal distribution P(X,Z) and print its probabilities as fractions.

>>> d2 = d.marginal(['X', 'Z'])
>>> print(d2.to_string(exact=True))
Class:    Distribution
Alphabet: (('0', '1'), ('0', '1'))
Base:     linear
<BLANKLINE>
x            p(X,Z)
('0', '0')   1/4
('0', '1')   1/4
('1', '0')   1/4
('1', '1')   1/4

Convert the distribution probabilities to log (base 3.5) probabilities, and access its probability mass function.

>>> d2.set_base(3.5)
>>> d2.pmf
array([-1.10658951, -1.10658951, -1.10658951, -1.10658951])

Draw 5 random samples from this distribution.

>>> dit.math.prng.seed(1)
>>> d2.rand(5)
[('0', '1'), ('1', '0'), ('0', '0'), ('0', '1'), ('0', '0')]

Source and Channel Coding

Beyond measures, dit builds explicit codes and ships a catalog of channels.

The dit.coding module constructs lossless source codes (Shannon, Fano, Shannon-Fano-Elias, Huffman, length-limited Huffman, Golomb/Rice, Tunstall, and the universal integer codes) and reports their code-theoretic properties (rate, redundancy, efficiency, the Kraft sum, and whether the code is prefix-free / uniquely decodable / optimal).

>>> from dit.coding import huffman
>>> d = dit.Distribution(['a', 'b', 'c', 'd', 'e'], [0.4, 0.2, 0.2, 0.1, 0.1])
>>> code = huffman(d)
>>> code.average_length()
2.2
>>> float(code.source_entropy())
2.1219280948873624
>>> code.is_optimal(), code.is_prefix_free()
(True, True)

It also builds binary (GF(2)) channel codes — linear block codes (repetition, parity-check, Hamming, Reed-Muller, Golay) as well as LDPC, polar, and convolutional codes — and evaluates them against a noisy channel supplied as a conditional Distribution p(Y|X).

>>> from dit.coding import hamming
>>> code = hamming(3)
>>> code.length, code.dimension, code.minimum_distance()
(7, 4, 3)
>>> bsc = dit.example_channels.binary_symmetric_channel(0.05)
>>> float(code.probability_of_error(bsc, method='exact'))
0.04438054218749993

The dit.example_channels module is a catalog of canonical discrete memoryless channels (binary symmetric, binary erasure, Z-channel, q-ary symmetric/erasure, noisy typewriter, …). Each constructor returns a conditional Distribution p(Y|X) ready for dit.algorithms.channel_capacity or the coding layer above.

>>> from dit.algorithms import channel_capacity
>>> bec = dit.example_channels.binary_erasure_channel(0.25)
>>> round(float(channel_capacity(bec)[0]), 10)
0.75

Contributions & Help

If you’d like a feature added to dit, please file an issue. Or, better yet, open a pull request. Ideally, all code should be tested and documented, but please don’t let this be a barrier to contributing. We’ll work with you to ensure that all pull requests are in a mergeable state.

If you’d like to get in contact about anything, you can reach us through our slack channel.

Thanks

Thank you to everyone who has contributed bug reports or pull requests to dit: Aaron Griffith, Abhishek Bhatia, @alimasn, André F. C. Gomes, Angus Forbes, Artemy Kolchinsky, Benjamin Lee, Bochen Zhao, Casper van Elteren, Charles Kemp, @d-jeon, Daniel Feldspar, David A. Ehrlich, Diego Volpatto, Elias Jaffe, Fabio Anza, Freya Behrens, Haochen Wu, Harald Schilly, Himanshu Mishra, @ivan-marroquin, Jaewon Chung, James K Johnson, Jan Olszak, Jeffrey Emenheiser, Juan Carlos Farah, Keenan Down, @kokokostation, Kunal Marwaha, Lee Junho, Longshan Du, @LuisHeinzlmeier, Marc Harper, Nathaniel Chen, @OrdinarySK, Pattarawat Chormai, QZP, Rion Brattig Correia, Robin Ince, Rodrigo Silva, Scott Sievert, Tai C. Huang, Talha Anwar, Thomas Kluyver, Tobias Mages, Vigneswaran Chandrasekaran, 赵丰 (Zhao Feng), and @zuiyanzuiyan.

Metadata

Release files for dit 2.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dit 2.4
File Size Uploaded Provenance Actions
dit-2.4.tar.gz 986.3 kB With provenance Details for dit-2.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dit 2.4
File Interpreter ABI Platform Actions
dit-2.4-py3-none-any.whl Python 3 none any Details for dit-2.4-py3-none-any.whl

Total release size: 1.8 MB

Release files / dit-2.4.tar.gz

Download URL dit-2.4.tar.gz
Size 986.3 kB
Tags Source
SHA-256 checksum
How to use checksums
9f9fc623985032e3ee35940066e8ab91d5a1c61f375542452e4f436186039532
BLAKE2b-256 checksum
How to use checksums
5402870a1d15418c8ddbb13ecc8c18fcaa96f391cd01f8ac193c198b45943665
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / dit-2.4-py3-none-any.whl

Download URL dit-2.4-py3-none-any.whl
Size 842.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3bbb4db1f031a3d185a2886762a5d3f6b33a9f178726f5fc9029bc8346ff9f5b
BLAKE2b-256 checksum
How to use checksums
0597f3e0e318c69c065185e29c62b791be00c30a88e33542056ee6629665281d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for dit 2.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for dit-2.4.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

2.4 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page