Skip to main content

Information Theoretic Measures of Entropy and Divergence

Project description

Divergence

Divergence is a Python package to compute statistical measures of entropy and divergence from probability distributions and samples.

The following functionality is provided:

  • (Information) Entropy [1], [2]
  • Cross Entropy: [3]
  • Relative Entropy or Kullback-Leibler (KL-) Divergence [4], [5]
  • Jensen-Shannon Divergence [6]
  • Joint Entropy [7]
  • Conditional Entropy [8]
  • Mutual Information [9]

The units in which these entropy and divergence measures are calculated can be specified by the user. This is achieved by setting the argument base, to 2.0, 10.0, or np.e.

In a Bayesian context, relative entropy can be used as a measure of the information gained by moving from a prior distribution q to a posterior distribution p.

Installation

    pip install divergence

Examples

See the Jupyter notebook Divergence.

References:

[1] https://en.wikipedia.org/wiki/Entropy_(information_theory)

[2] Shannon, Claude Elwood (July 1948). "A Mathematical Theory of Communication". Bell System Technical Journal. 27 (3): 379–423

[3] https://en.wikipedia.org/wiki/Cross_entropy

[4] https://en.wikipedia.org/wiki/Kullback–Leibler_divergence

[5] Kullback, S.; Leibler, R.A. (1951). "On information and sufficiency". Annals of Mathematical Statistics. 22 (1): 79–86

[6] https://en.wikipedia.org/wiki/Jensen–Shannon_divergence

[7] https://en.wikipedia.org/wiki/Joint_entropy

[8] https://en.wikipedia.org/wiki/Conditional_entropy

[9] https://en.wikipedia.org/wiki/Mutual_information

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

divergence-0.4.2.tar.gz (16.7 kB view details)

Uploaded Source

Built Distribution

divergence-0.4.2-py3-none-any.whl (17.3 kB view details)

Uploaded Python 3

File details

Details for the file divergence-0.4.2.tar.gz.

File metadata

  • Download URL: divergence-0.4.2.tar.gz
  • Upload date:
  • Size: 16.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.1

File hashes

Hashes for divergence-0.4.2.tar.gz
Algorithm Hash digest
SHA256 c545626a9d52aedb69906e95adc44810228173d7c24dd717c5ec966caec62666
MD5 9b6992c344735406a426961976bcdca5
BLAKE2b-256 f383125fb9009f7f679a69d672a28243eaff120cd3107cb095bf63855061b4f1

See more details on using hashes here.

File details

Details for the file divergence-0.4.2-py3-none-any.whl.

File metadata

  • Download URL: divergence-0.4.2-py3-none-any.whl
  • Upload date:
  • Size: 17.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.1

File hashes

Hashes for divergence-0.4.2-py3-none-any.whl
Algorithm Hash digest
SHA256 1e254d3ed75abc9b165631596af87650b7a956a9eeda4c1766b464c9668cc725
MD5 76a0ef6bc87c48e1acde5dbc699d9d57
BLAKE2b-256 64c97f5f5668664980e76e6e000382a9374e17e0de6f60985013354d3812b1a9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page