midaspy 1.4.2

Multiple Imputation with Denoising Autoencoders

MIDASpy

Deprecation notice

MIDASpy is deprecated. Please use midasverse-midas, which replaces MIDASpy with a faster PyTorch-based backend, a simpler sklearn-style API (no manual preprocessing), and fewer dependencies (no TensorFlow). MIDASpy will remain on PyPI for existing users but will not receive new features or bug fixes.

A migration guide is available below: Migrating to midasverse-midas.

Install the replacement:

pip install midasverse-midas

---

Overview

MIDASpy is a Python package for multiply imputing missing data using deep learning methods. The MIDASpy algorithm offers significant accuracy and efficiency advantages over other multiple imputation strategies, particularly when applied to large datasets with complex features. In addition to implementing the algorithm, the package contains functions for processing data before and after model training, running imputation model diagnostics, generating multiple completed datasets, and estimating regression models on these datasets.

For an implementation in R, see rMIDAS2.

Background and suggested citations

For more information on MIDAS, the method underlying the software, see:

Lall, Ranjit, and Thomas Robinson. 2022. "The MIDAS Touch: Accurate and Scalable Missing-Data Imputation with Deep Learning." Political Analysis 30, no. 2: 179-196. doi:10.1017/pan.2020.49. Published version. Accepted version.

Lall, Ranjit, and Thomas Robinson. 2023. "Efficient Multiple Imputation for Diverse Data in Python and R: MIDASpy and rMIDAS." Journal of Statistical Software 107, no. 9: 1-38. doi:10.18637/jss.v107.i09. Published version.

Installation

To install via pip, enter the following command into the terminal: pip install MIDASpy

The latest development version (potentially unstable) can be installed via the terminal with: pip install git+https://github.com/MIDASverse/MIDASpy.git

MIDAS requires:

• Python (>=3.6; <3.11)
• Numpy (>=1.5, <=1.26.4)
• Pandas (>=0.19)
• TensorFlow (<2.12)
• Matplotlib
• Statmodels
• Scipy
• TensorFlow Addons (<0.20)

Tensorflow also has a number of requirements, particularly if GPU acceleration is desired. See https://www.tensorflow.org/install/ for details.

Examples

For a simple demonstration of MIDASpy, see our Jupyter Notebook examples.

---

Migrating to midasverse-midas

Why midasverse-midas?

	MIDASpy	midasverse-midas
Backend	TensorFlow 1.x / 2.x	PyTorch
Preprocessing	Manual (binary_conv(), cat_conv(), column sorting)	Automatic column-type detection
API style	Separate init / build_model() / train_model() / generate_samples()	sklearn-style fit() / transform() / fit_transform()
Python versions	3.6--3.10	3.9+
TensorFlow required	Yes	No

Installation

pip install midasverse-midas

Side-by-side comparison

1. Preprocessing

MIDASpy required manual conversion of binary and categorical columns before building the model:

# --- MIDASpy ---
from MIDASpy import Midas, binary_conv, cat_conv

df['income'] = binary_conv(df['income'])
cat_encoded, cat_cols = cat_conv(df[['workclass', 'marital_status']])
df = pd.concat([df.drop(['workclass', 'marital_status'], axis=1), cat_encoded], axis=1)

midasverse-midas detects column types automatically:

# --- midasverse-midas ---
from midas2 import MIDAS

# No preprocessing needed -- just pass your DataFrame directly

2. Model construction and training

MIDASpy required three separate steps -- instantiate, build, train:

# --- MIDASpy ---
imputer = Midas(
    layer_structure=[256, 256, 256],
    learn_rate=0.0004,
    input_drop=0.8,
    train_batch=16,
    seed=42,
)
imputer.build_model(
    imputation_target=df,
    binary_columns=['income'],
    softmax_columns=cat_cols,
)
imputer.train_model(training_epochs=20)

midasverse-midas combines these into a single fit() call:

# --- midasverse-midas ---
mod = MIDAS(hidden_layers=[256, 128, 64], dropout_prob=0.5)
mod.fit(df, epochs=20, lr=0.001, corrupt_rate=0.8, seed=42)

Parameter name changes:

MIDASpy (__init__ / build_model / train_model)	midasverse-midas (__init__ / fit)	Notes
layer_structure	hidden_layers	Default changed from [256,256,256] to [256,128,64]
learn_rate	lr	Default changed from 0.0004 to 0.001
input_drop	corrupt_rate	Moved to fit()
train_batch	batch_size	Default changed from 16 to 64
dropout_level	dropout_prob	Moved to __init__()
cont_adj	num_adj	Moved to fit()
softmax_adj	cat_adj	Moved to fit()
binary_adj	bin_adj	Moved to fit()
training_epochs	epochs	Moved to fit()
binary_columns / softmax_columns	Automatic	No manual specification needed

3. Generating imputations

MIDASpy used generate_samples() to store imputations in an attribute:

# --- MIDASpy ---
imputer.generate_samples(m=10)
completed_datasets = imputer.output_list

midasverse-midas uses transform(), which returns a generator:

# --- midasverse-midas ---
imputations = list(mod.transform(m=10))

Or use fit_transform() for an all-in-one approach:

# --- midasverse-midas ---
imputations = list(mod.fit_transform(df, m=10, epochs=20))

4. Rubin's rules regression

MIDASpy combine() took separate y_var and X_vars arguments along with a list of DataFrames:

# --- MIDASpy ---
from MIDASpy import combine
results = combine(
    y_var='income',
    X_vars=['age', 'hours_per_week'],
    df_list=imputer.output_list,
)

midasverse-midas combine() takes dfs, y, and optional ind_vars (defaults to all non-outcome columns):

# --- midasverse-midas ---
from midas2 import combine
results = combine(imputations, y='income', ind_vars=['age', 'hours_per_week'])

# Or use all predictors:
results = combine(imputations, y='income')

5. Mean imputation (new)

midasverse-midas adds an imp_mean() utility:

# --- midasverse-midas only ---
from midas2 import imp_mean
mean_imputed = imp_mean(mod.transform(m=10), pandas=True)

Complete migration example

MIDASpy (old)

from MIDASpy import Midas, binary_conv, cat_conv, combine

# 1. Preprocess
df['income'] = binary_conv(df['income'])
cat_encoded, cat_cols = cat_conv(df[['workclass', 'marital_status']])
df_processed = pd.concat([df.drop(['workclass', 'marital_status'], axis=1), cat_encoded], axis=1)

# 2. Build and train
imputer = Midas(layer_structure=[256, 256, 256], seed=42)
imputer.build_model(df_processed, binary_columns=['income'], softmax_columns=cat_cols)
imputer.train_model(training_epochs=20)

# 3. Generate imputations
imputer.generate_samples(m=5)

# 4. Analyse
combine(y_var='income', X_vars=['age', 'hours_per_week'], df_list=imputer.output_list)

midasverse-midas (new)

from midas2 import MIDAS, combine

# 1. Fit (no preprocessing needed)
mod = MIDAS()
mod.fit(df, epochs=20, seed=42)

# 2. Impute
imputations = list(mod.transform(m=5))

# 3. Analyse
combine(imputations, y='income', ind_vars=['age', 'hours_per_week'])

Quick-reference cheat sheet

Task	MIDASpy	midasverse-midas
Import	from MIDASpy import Midas	from midas2 import MIDAS
Preprocess binary	binary_conv(col)	Automatic
Preprocess categorical	cat_conv(df[cols])	Automatic
Instantiate	Midas(layer_structure, ...)	MIDAS(hidden_layers, ...)
Build model	imputer.build_model(df, binary_columns, softmax_columns)	Not needed
Train	imputer.train_model(training_epochs)	mod.fit(df, epochs, ...)
Generate imputations	imputer.generate_samples(m)	mod.transform(m)
All-in-one	Not available	mod.fit_transform(df, m, ...)
Access imputations	imputer.output_list	list(mod.transform(m))
Mean imputation	Not available	imp_mean(imputations)
Rubin's rules	combine(y_var, X_vars, df_list)	combine(dfs, y, ind_vars)

---

Version history

Version 1.4.1 (August 2024)

• Adds support for non-negative output columns, with a positive_columns argument

Version 1.3.1 (October 2023)

• Minor update to reflect publication of accompanying article in Journal of Statistical Software
• Further updates to make documentation and URLs consistent, including removing unused metadata

Version 1.2.4 (August 2023)

• Adds support for Python 3.9 and 3.10
• Addresses deprecation warnings and other minor bug fixes
• Resolves dependency issues and includes an updated setup.py file
• Adds GitHub Actions workflows that trigger automatic tests on the latest Ubuntu, macOS, and Windows for Python versions 3.7 to 3.10 each time a push or pull request is made to the main branch
• An additional Jupyter Notebook example that demonstrates the core functionalities of MIDASpy

Version 1.2.3 (December 2022)

v1.2.3 adds support for installation on Apple Silicon hardware (i.e. M1 and M2 Macs).

Version 1.2.2 (July 2022)

v1.2.2 makes minor efficiency changes to the codebase. Full details are available in the Release logs.

Version 1.2.1 (January 2021)

v1.2.1 adds new pre-processing functionality and a multiple imputation regression function.

Users can now automatically preprocess binary and categorical columns prior to running the MIDAS algorithm using binary_conv() and cat_conv().

The new combine() function allows users to run regression analysis across the complete data, following Rubin's combination rules.

Previous versions

Version 1.1.1 (October 2020)

Key changes:

• Update adds full Tensorflow 2.X support:

  • Users can now run the MIDAS algorithm in TensorFlow 2.X (TF1 support retained)

  • Tidier handling of random seed setting across both TensorFlow and NumPy

• Fixes a minor dependency bug

• Other minor bug fixes

Version 1.0.2 (September 2020)

Key changes:

• Minor, mainly cosmetic, changes to the underlying source code.
• Renamed 'categorical_columns' argument in build_model() to 'binary_columns' to avoid confusion
• Added plotting arguments to overimputation() method to suppress intermediary overimputation plots (plot_main) and all plots (skip_plot).
• Changed overimputation() plot titles, labels and legends
• Added tensorflow 2.0 version check on import
• Fixed seed-setting bug in earlier versions

Alpha 0.2:

Variational autoencoder enabled. More flexibility in model specification, although defaulting to a simple mirrored system. Deeper analysis tools within .overimpute() for checking fit on continuous values. Constructor code deconflicted. Individual output specification enabled for very large datasets.

Key added features:

• Variational autoencoder capacity added, including encoding to and sampling from latent space

Alpha 0.1:

• Basic functionality feature-complete.
• Support for mixed categorical and continuous data types
• An "additional data" pipeline, allowing data that may be relevant to the imputation to be included (without being included in error generating statistics)
• Simplified calibration for model complexity through the "overimputation" function, including visualization of reconstructed features
• Basic large dataset functionality