Skip to main content

A decision-tree based conditional independence test

Project description

.. image:: https://img.shields.io/badge/License-MIT-yellow.svg
:target: https://opensource.org/licenses/MIT
:alt: License

*A Decision Tree (Conditional) Independence Test (DTIT).*

Introduction
-----------
Let *x, y, z* be random variables. Then deciding whether *P(y | x, z) = P(y | z)*
can be difficult, especially if the variables are continuous. This package
implements a simple yet efficient and effective conditional independence test,
described in [link to arXiv when we write it up!]. Important features that differentiate
this test from competition:

* It is fast. Worst-case speed scales as O(n_data * log(n_data) * dim), where dim is max(x_dim + z_dim, y_dim). However, amortized speed is O(n_data * log(n_data) * log(dim)).

* It applies to cases where some of x, y, z are continuous and some are discrete, or categorical (one-hot-encoded).

* It is very simple to understand and modify.

* It can be used for unconditional independence testing with almost no changes to the procedure.

We have applied this test to tens of thousands of samples of thousand-dimensional datapoints in seconds. For smaller dimensionalities and sample sizes, it takes a fraction of a second. The algorithm is described in [arXiv link coming], where we also provide detailed experimental results and comparison with other methods. However for now, you should be able to just look through the code to understand what's going on -- it's only 90 lines of Python, including detailed comments!

Usage
-----
Basic usage is simple, and the default settings should work in most cases. To perform an *unconditional test*, use dtit.test(x, y):

.. code:: python

import numpy as np
from dtit import dtit

x = np.random.rand(1000, 1)
y = np.random.randn(1000, 1)

pval_i = dtit.test(x, y) # p-value should be uniform on [0, 1].
pval_d = dtit.test(x, x + y) # p-value should be very small.

To perform a conditional test, just add the third variable z to the inputs:

.. code:: python

import numpy as np
from dtit import dtit

# Generate some data such that x is indpendent of y given z.
n_samples = 1000
z = np.random.dirichlet(alpha=np.ones(2), size=n_samples)
x = np.vstack([np.random.multinomial(20, p) for p in z]).astype(float)
y = np.vstack([np.random.multinomial(20, p) for p in z]).astype(float)

# Check that x and y are dependent (p-value should be uniform on [0, 1]).
pval_d = dtit.test(x, y)
# Check that z d-separates x and y (the p-value should be small).
pval_i = dtit.test(x, y, z)

Installation
-----------
pip install dtit


Requirements
------------
Tested with Python 3.6 and

* numpy >= 1.12
* scikit-learn >= 0.18.1
* scipy >= 0.16.1

.. _pip: http://www.pip-installer.org/en/latest/


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dtit-1.2.0.tar.gz (5.9 kB view details)

Uploaded Source

Built Distribution

dtit-1.2.0-py2.py3-none-any.whl (7.6 kB view details)

Uploaded Python 2Python 3

File details

Details for the file dtit-1.2.0.tar.gz.

File metadata

  • Download URL: dtit-1.2.0.tar.gz
  • Upload date:
  • Size: 5.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for dtit-1.2.0.tar.gz
Algorithm Hash digest
SHA256 b352ef77bee229f1fc987205c725792098fb5f3294a70e5619592fc9673f133f
MD5 4552cf8041f9d3b2f067c1732fb2ff08
BLAKE2b-256 6f90ef65ee82665740e4cd1caa8e9e6fe56e2c90b63d74bcdf8c54afe0dfc169

See more details on using hashes here.

File details

Details for the file dtit-1.2.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for dtit-1.2.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 be31eb8b42b882f5f149d0c413ab53ed37128b1282ed5d3a379b7d59fb7a2549
MD5 00ca3ef53ac29ffe86cfbd26703a2a1c
BLAKE2b-256 1b4e2be0f32e3d37fba98112ee9eb8c6f7e31f468a66f0acb3951a1e382033a3

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page