ifcfill

ifcfill is a Python library for preparing tabular data for synthetic-data generation. It transforms real tabular data into Integer, Float, and Categorical (IFC) variables with fast NumPy-powered missing data imputation, then applies the same learned inverse transformation to generated synthetic data so it can be mapped back to the original table structure.

Purpose

ifcfill is designed to sit around tabular synthetic-data generators:

1. Fit IFCTransformer on real data.
2. Transform the real data into generator-friendly IFC variables.
3. Train or run any tabular generator on the transformed data.
4. Apply inverse_transform() to the synthetic output using the mappings learned from the real data.

If inverse transformation happens later or on another machine, save the fitted transformation state with save() and load it back with IFCTransformer.load().

The package is intentionally unsupervised: it does not require or model a target variable.

Features

• Flexible input — accepts a pandas.DataFrame or a path to a CSV file
• Automatic type inference — detects integer, float, categorical, and datetime columns automatically; user overrides supported per column
• Configurable imputation — choose fill strategy independently for each type:
  • Integer: mean, median, mode, zero
  • Float: mean, median, mode, zero
  • Categorical: constant (default), mode
• Synthesis-safe categorical missingness — missing categorical values use the internal __ifcfill_missing__ sentinel during transformation
• Marginal or joint reconstruction — inverse transformation always reconstructs either fitted per-variable missingness or the empirical joint row-pattern distribution
• Optional categorical label encoding — fill categorical values first, then encode categories as integer codes through a separate label-encoding layer with inverse mappings
• Datetime → integer conversion — converts date/time columns to integers relative to a configurable anchor date and time unit (days, seconds, ms, …)
• Constant column removal — automatically drops true constant columns while preserving categorical missing categories when they are learnable
• Missing value tracking — records the count and fraction of missing values per column at fit time, accessible via missing_report_
• Full transformation bookkeeping — inverse_transform() restores semantic dtypes, dropped constants, original column order, and fitted missingness
• Portable fitted state — save all learned transformations to JSON and load them later for consistent transform/inverse-transform workflows on another machine
• Synthetic-data workflow support — apply one inverse transformation consistently to both transformed real data and generated synthetic data

Installation

pip install ifcfill

Quick Start

import pandas as pd
from ifcfill import IFCTransformer

df = pd.DataFrame({
    "age":    [25, 30, None, 40],
    "salary": [50_000.5, None, 75_000.0, 90_000.25],
    "city":   ["London", None, "Paris", "London"],
    "joined": pd.to_datetime(["2020-01-01", "2021-06-15", None, "2023-03-10"]),
    "flag":   ["yes", "yes", "yes", "yes"],   # constant → will be dropped
})

tf = IFCTransformer(
    int_fill="median",
    float_fill="mean",
    datetime_anchor="1970-01-01",
    datetime_unit="D",
)

transformed = tf.fit_transform(df)
print(transformed)

# Inspect missing-value distribution captured at fit time
print(tf.missing_report_)

# Restore original structure and the default marginal missingness distribution.
restored = tf.inverse_transform(
    transformed,
    missingness_restore="marginal",
    random_state=42,
)
print(restored)

# Save everything needed to transform or inverse-transform later.
tf.save("ifcfill-state.json")

# On another machine or in another process:
loaded_tf = IFCTransformer.load("ifcfill-state.json")
restored_again = loaded_tf.inverse_transform(transformed)

Missingness reconstruction

inverse_transform() always reconstructs missingness across every original column, including categorical and internally dropped constant columns:

• IFCFill-Marginal (missingness_restore="marginal", the default) preserves each fitted per-variable distribution, P(M_j). For n synthetic rows it assigns exactly round(fitted_fraction * n) missing positions per column.
• IFCFill-Joint (missingness_restore="joint") preserves the fitted empirical row-wise pattern distribution, P(M_1, ..., M_p), using largest-remainder count allocation followed by a reproducible row shuffle.

The categorical sentinel __ifcfill_missing__ only makes incomplete training data synthesis-ready. It does not control final missingness. IFCFill replaces excess generated sentinels and applies its reconstructed mask as the authority. Integers are rounded without clipping and returned as nullable pandas Int64; floats remain floating-point and missing datetimes use NaT.

From a CSV file

transformed = IFCTransformer().fit_transform("data.csv")

Parallel column processing

# Use all available CPUs for per-column fit/transform work
tf = IFCTransformer(n_jobs=-1)
transformed = tf.fit_transform(df)

Override column types

tf = IFCTransformer(col_types={"age": "categorical", "score": "float"})
transformed = tf.fit_transform(df)

Label encoding for generators

tf = IFCTransformer(cat_encoding="label")
transformed = tf.fit_transform(df)

# Safe copies of learned category/code dictionaries
print(tf.get_category_mappings())

restored = tf.inverse_transform(transformed)

Documentation

Full documentation including the API reference is available at: https://eulerlettersai.github.io/ifcfill

License

Apache 2.0 — see LICENSE.