decon_dtn_toolkit 0.1.1

A toolkit for addressing confounding effects in text classification problems

DeconDTN-Toolkit

DeconDTN Toolkit is a PyTorch suite containing benchmark datasets and algorithms for confounding effects in text classification.

[Docs] [Dataset Access]

Features

If a dataset is drawn from two different sources, one may be enriched for the outcome of interest (i.e. $P(Y \mid \text{source1}) \neq P(Y \mid \text{source2})$ ). In this situation a model may learn to recognize the data source, and make predictions in accordance with their class distribution, rather than on the basis of relevant features. This scenario, which we refer to as provenance shift, was the primary motivating use case for the development of the DeconDTN toolkit, though the same evaluation framework and methods of mitigation can apply to other confounding variables also.

• An evaluation framework for assessment of robustness to confounding shifts in which the proportion of positive examples changes with a confounding variable.
• A range of algorithms with the potential to mitigate for confounding shift
• A range of benchmark datasets to evaluate performance.

Available Algorithms

The currently available algorithms are:

• Empirical Risk Minimization (ERM, Vapnik, 1998)
• Data Re-Sampling (ReSample, Japkowicz, 2000)
• Domain Adversarial Neural Network (DANN, Ganin et al., 2015)
• Conditional Domain Adversarial Neural Network (CDANN, Li et al., 2018)
• Deep Correlation Alignment (CORAL, Sun and Saenko, 2016)
• Maximum Mean Discrepancy (MMD, Li et al., 2018)
• Mixup (Mixup, Zhang et al., 2018)
• Learning Invariant Predictors with Selective Augmentation (LISA, Yao et al., 2022)
• Invariant Risk Minimization (IRM, Arjovsky et al., 2019)
• Group Distributionally Robust Optimization (GroupDRO, Sagawa et al., 2020)
• Gradient Matching for Domain Generalization (Fish, Shi et al., 2021)
• Learning from Failure (LfF, Nam et al., 2020)
• Just Train Twice (JTT, Liu et al., 2021)
• Deep Feature Reweighting (DFR, Kirichenko et al., 2022)
• Optimal Representations for Covariate Shift (CAD & CondCAD, Ruan et al., 2022)
• Backdoor Adjustment (BackDoor, Ding et al., 2024)
• Dual Filter (DualFilter, Sheng et al., 2025)

Send us a PR to add your algorithm! Our implementations use the hyper-parameter grids described here.

Available Datasets

The currently available datasets are:

• CivilComments (Borkan et al., 2019) from the WILDS benchmark
• MultiNLI (Williams et al., 2017)
• MIMICNotes (Johnson et al., 2016) from the SubpopBench
• AmazonReviews (Ni et al., 2019) from Veith et al., 2021
• SHAC (Lybarger et al., 2021)
• HateSpeech (Vidgen et al., 2021 & Gibert et al., 2018) from Ding et al., 2023

Installation

Prerequisites

• Ubuntu 18.04 or higher
• CUDA 12.1 or higher
• Python 3.10 or higher
• pip

Python Package

pip install --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple/ \
            decon-dtn-toolkit

From source

Option 1: uv (Recommended)

# Create virtual environment
uv venv dedtn-env
source dedtn-env/bin/activate  # On Linux/Mac
# dedtn-env\Scripts\activate  # On Windows

# Install the package in editable mode (for development)
uv pip install -e .

Option 2: conda

Conda is ideal for managing complex dependencies, especially with CUDA/PyTorch installations. It provides both package and environment management.

# Create environment with Python 3.12
conda create -n dedtn-tool python=3.12
conda activate dedtn-tool

# Install the package in editable mode
pip install -e .

Option 3: venv (Python Built-in)

venv is Python's built-in virtual environment tool - lightweight and requires no additional installation. Good for standard Python projects.

# Create virtual environment
python -m venv dedtn-env

# Activate environment
# On Linux/Mac:
source dedtn-env/bin/activate
# On Windows:
# dedtn-env\Scripts\activate

# Install the package
pip install -e .

Verify Installation

After activating your chosen environment, you can verify the installation with:

python -c "import decon_dtn_toolkit; print('DeconDTN-Toolkit installed successfully')"

Quick Start

To train DANN on the Amazon_Reviews dataset

from decon_dtn_toolkit import datasets
from decon_dtn_toolkit.trainer import TrainConfig, Trainer

data_dir = "PATH_TO_Amazon_Reviews_2018"
amazon_reviews = vars(datasets)["Amazon_Reviews_2018"](root=data_dir)
config = TrainConfig(algorithm='DANN')
model = Trainer(dataset=dataset, config=config)
model.train()

Unittest

python -m unittest discover

Acknowledgement

This project is built upon

• DomainBed - A PyTorch suite containing benchmark datasets and algorithms for domain generalization in computer vision - MIT license
• WILDS - A benchmark of in-the-wild distribution shifts spanning diverse data modalities and applications - MIT license
• SubpopBench - A benchmark of subpopulation shift - MIT license

Citation

Below are citations of the DeconDTN line of work.

@inproceedings{ding2024backdoor,
  title={Backdoor adjustment of confounding by provenance for robust text classification of multi-institutional clinical notes},
  author={Ding, Xiruo and Sheng, Zhecheng and Yeti{\c{s}}gen, Meliha and Pakhomov, Serguei and Cohen, Trevor},
  booktitle={AMIA Annual Symposium Proceedings},
  volume={2023},
  pages={923},
  year={2024}
}
@article{ding2025tailoring,
  title={Tailoring task arithmetic to address bias in models trained on multi-institutional datasets},
  author={Ding, Xiruo and Sheng, Zhecheng and Hur, Brian and Tauscher, Justin and Ben-Zeev, Dror and Yeti{\c{s}}gen, Meliha and Pakhomov, Serguei and Cohen, Trevor},
  journal={Journal of Biomedical Informatics},
  pages={104858},
  year={2025},
  publisher={Elsevier}
}
@inproceedings{sheng2025mitigating,
  title={Mitigating confounding in speech-based dementia detection through weight masking},
  author={Sheng, Zhecheng and Ding, Xiruo and Hur, Brian and Li, Changye and Cohen, Trevor and Pakhomov, Serguei VS},
  booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={10419--10434},
  year={2025}
}

Key Contributors (listed alphabetically)

Trevor Cohen

Xiruo Ding

Yongsen Tan

Zhecheng Sheng