Skip to main content

DataComp: A Python Framework for Systematic Dataset Comparisons

Current version on PyPI Apache 2.0 License Stable Supported Python Versions Development Documentation Status

Description

DataComp is an open source Python package for domain independent multimodal longitudinal dataset comparisons. It serves as an investigative toolbox to assess differences between multiple datasets on feature level. DataComp empowers data analysts to identify significantly different and not significantly difference between datasets and thereby is helpful to identify comparable dataset combinations.

Typical application scenarios are:

  • Identifying comparable datasets that can be used in machine learning approaches as training and independent test data

  • Evaluate if, how and where simulated or synthetic datasets deviate from real world data

  • Assess (systematic) differences across multiple datasets (for example multiple sampling sites)

  • Conducting multiple statistical comparisons

  • Comparative visualizations

./docs/source/DataComp_workflow.png

The figure above depicts a typical DataComp workflow.

Main Features

DataComp supports:

  • Evaluating and visualizing the overlap in features across datasets

  • Parametric and nonparametric statistical hypothesis testing to compare feature value distributions

  • Creating comparative plots of feature value distributions

  • Normalizing time series data to baseline and statistically comparing the progression of features over time

  • Comparative visualization of feature progression over time

  • Hierarchical clustering of the entities in the data sets to evaluate if dataset membership labels are evenly distributed across clusters or assigned to distinct clusters

  • Performing a MANOVA to assess the influence of features onto the dataset membership

Installation

pip install datacomp

Documentation

The full package documentation can be found here.

Application examples

Example notebooks showcasing Datacomp workflows and results on simulated data can be found at DataComp_Examples:

Metadata

Release files for datacomp 0.0.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datacomp 0.0.6
File Size Uploaded
datacomp-0.0.6.tar.gz 21.5 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for datacomp 0.0.6
File Interpreter ABI Platform
datacomp-0.0.6-py3-none-any.whl Python 3 none any Details

Total release size: 49.0 kB

Release files / datacomp-0.0.6.tar.gz

Download URL datacomp-0.0.6.tar.gz
Size 21.5 kB
Tags Source
SHA-256 checksum
How to use checksums
3980c5300702c3561da8e8f61709d8adcc581af14e7007a129c5d8e8e2c8ef6a
BLAKE2b-256 checksum
How to use checksums
ef12f528202bdd6edfebd7f56999ba017ebc5519457cd82abbdb5af05055daf4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.5.2

Release files / datacomp-0.0.6-py3-none-any.whl

Download URL datacomp-0.0.6-py3-none-any.whl
Size 27.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
897a43ab8d835d5fc5b7937e8628557a2ab5bba236d308a918d9b4e283fa002c
BLAKE2b-256 checksum
How to use checksums
39b8c40f97d8c2d220778c49bb004ceacc84494f9ea008c3f5ec30986a21563f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.5.2

Release history Release notifications | RSS feed

This release

0.0.6 This release

2 release files

0.0.5

2 release files

0.0.4

2 release files

0.0.3

2 release files

0.0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page