Information Theoretic Measures of Entropy and Divergence
Project description
Divergence
Divergence is a Python package to compute statistical measures of entropy and divergence from probability distributions and samples.
The following functionality is provided:
- (Information) Entropy [1], [2]
- Cross Entropy: [3]
- Relative Entropy or Kullback-Leibler (KL-) Divergence [4], [5]
- Jensen-Shannon Divergence [6]
- Joint Entropy [7]
- Conditional Entropy [8]
- Mutual Information [9]
The units in which these entropy and divergence measures are calculated can be specified by the user.
This is achieved by setting the argument base
, to 2.0
, 10.0
, or np.e
.
In a Bayesian context, relative entropy can be used as a measure of the information gained by moving
from a prior distribution q
to a posterior distribution p
.
Installation
pip install divergence
Examples
See the Jupyter notebook Divergence.
References:
[1] https://en.wikipedia.org/wiki/Entropy_(information_theory)
[2] Shannon, Claude Elwood (July 1948). "A Mathematical Theory of Communication". Bell System Technical Journal. 27 (3): 379–423
[3] https://en.wikipedia.org/wiki/Cross_entropy
[4] https://en.wikipedia.org/wiki/Kullback–Leibler_divergence
[5] Kullback, S.; Leibler, R.A. (1951). "On information and sufficiency". Annals of Mathematical Statistics. 22 (1): 79–86
[6] https://en.wikipedia.org/wiki/Jensen–Shannon_divergence
[7] https://en.wikipedia.org/wiki/Joint_entropy
[8] https://en.wikipedia.org/wiki/Conditional_entropy
[9] https://en.wikipedia.org/wiki/Mutual_information
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file divergence-0.4.2.tar.gz
.
File metadata
- Download URL: divergence-0.4.2.tar.gz
- Upload date:
- Size: 16.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.1
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c545626a9d52aedb69906e95adc44810228173d7c24dd717c5ec966caec62666 |
|
MD5 | 9b6992c344735406a426961976bcdca5 |
|
BLAKE2b-256 | f383125fb9009f7f679a69d672a28243eaff120cd3107cb095bf63855061b4f1 |
File details
Details for the file divergence-0.4.2-py3-none-any.whl
.
File metadata
- Download URL: divergence-0.4.2-py3-none-any.whl
- Upload date:
- Size: 17.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.1
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e254d3ed75abc9b165631596af87650b7a956a9eeda4c1766b464c9668cc725 |
|
MD5 | 76a0ef6bc87c48e1acde5dbc699d9d57 |
|
BLAKE2b-256 | 64c97f5f5668664980e76e6e000382a9374e17e0de6f60985013354d3812b1a9 |