Skip to main content

Haphazard

A Python package for haphazard dataset and model management.
Provides a standardized interface for loading datasets (with online normalization) and models, running experiments, and extending with custom datasets or models.


Table of Contents


Installation

Install via pip (after packaging):

pip install haphazard

Project Structure

The Haphazard package follows a modular, extensible design:

haphazard/
├── __init__.py
├── data/                          # Dataset management
│   ├── __init__.py
│   ├── base_dataset.py
│   ├── mask.py
│   └── datasets/
│       ├── __init__.py
│       ├── dummy_dataset/
│       ├── magic04/
│       ├── a8a/
│       ├── imdb/
│       ├── susy/
│       ├── higgs/
│       ├── dry_bean/
│       └── gas/
├── models/                        # Model management
│   ├── __init__.py
│   ├── base_model.py
│   └── model_zoo/
│       ├── __init__.py
│       ├── dummy_model/
│       ├── dynfo/
│       ├── fae/
│       ├── nb3/
│       ├── ocds/
│       ├── olifl/
│       ├── olvf/
│       ├── orf3v/
│       └── ovfm/
├── normalization/                 # New in v1.1.0
│   ├── __init__.py
│   ├── base_normalizer.py
│   └── normalizer_zoo/
│       ├── __init__.py
│       ├── decimal_scale.py
│       ├── mean.py
│       ├── minmax.py
│       ├── no_normalization.py
│       ├── unit_vector.py
│       └── zscore.py
└── utils/                         # Utilities
    ├── __init__.py
    ├── file_utils.py
    ├── metrics.py
    └── seeding.py

Notes

  • data/base_dataset.py defines the BaseDataset class and integrates normalization support.
  • normalization/base_normalizer.py defines a universal BaseNormalizer base class.
  • normalizer_zoo/ provides built-in normalizers (e.g., zscore, mean, decimal_scale, no_normalization).
  • models/base_model.py defines BaseModel, used by all models in model_zoo/.
  • Dynamic registration of datasets, models, and now normalizers is handled via decorators.

Quick Start

from haphazard import load_dataset, load_model

# Load dataset
dataset = load_dataset("dummy", n_samples=100, n_features=10, norm="zscore")

# Load model
model = load_model("dummy")

# Run model
outputs = model(dataset)
print(outputs)

Datasets

  • All datasets inherit from BaseDataset.
  • Example dataset: DummyDataset.
  • Main interface:
from haphazard import load_dataset
dataset = load_dataset(
   "magic04", 
   base_path="./data", 
   scheme="probabilistic", 
   availability_prob=0.5,
   norm="none"
   )

x, y = dataset.x, dataset.y
mask = dataset.mask

Dataset Attributes

  • name: str - dataset name
  • task: "classification" | "regression"
  • haphazard_type: "controlled" | "intrinsic"
  • n_samples, n_features: int
  • num_classes: int (for classification)
  • normalizer: optional (default="none") defines a normalization scheme, if used

Available Datasets

  • WPBC ("wpbc")
  • Ionosphere ("ionosphere")
  • WDBC ("wdbc")
  • Australian ("australian")
  • WBC ("wbc")
  • Diabetes-f ("diabetes_f")
  • Crowdsense(c3) ("crowdsense_c3")
  • Crowdsense(c5) ("crowdsense_c5")
  • German ("german")
  • IPD ("ipd")
  • Svmguide3 ("svmguide3")
  • KR-vs-KP ("krvskp")
  • Spambase ("spambase")
  • Spamassasin ("apamassasin")
  • Magic04 ("magic04")
  • IMDB ("imdb")
  • A8A ("a8a")
  • Diabetes US ("diabetes_us")
  • SUSY ("susy")
  • HIGGS ("higgs")
  • Dry Bean (dry_bean"")
  • GAS ("gas")

Models

  • All models inherit from BaseModel.
  • Example model: DummyModel.
from haphazard import load_model
model = load_model("dummy")
model_params = {}  # Hyperparameters of the model
outputs = model(dataset, model_params)

Output

  • Classification: labels, preds, logits, time_taken, is_logit
  • Regression: targets, preds, time_taken

Available Models

  • Dummy - testing/prototyping.
  • NB3, FAE - Naive Bayes based models.
  • DynFo, ORF3V - Decision stump based models.
  • OLVF, OLIFL, OVFM, OCDS - Linear classifier based models.

Normalization

Overview

Introduced in v1.1.0, the normalization module provides standardized interfaces for online feature normalization across datasets and models.

Using Built-in Normalizers

from haphazard import load_dataset, load_normalizer

dataset = laod_dataset(
   "a8a",
   base_path="./",
   scheme="sudden",
   num_chunks=4,
   norm="none"  # No normalization applied internally
)

# Load z-score normalization
normalizer = load_normalizer("zscore", num_features=dataset.n_features)
for x, mask, y in dataset:
   x_norm = normalizer(x, mask)
   ...  
   # Further processing as required

# Load mean normalization
normalizer = load_normalizer("mean", num_features=dataset.n_features)
X, Mask, Y = dataset.x, dataset.y, dataset.mask
for x, mask, y in zip(X, Mask, Y):
   x_norm = normalizer(x, mask)
   ...  
   # Further processing as required

or

from haphazard import load_dataset, load_normalizer

dataset = laod_dataset(
   "a8a",
   base_path="./",
   scheme="sudden",
   num_chunks=4,
   norm="zscore"  # Apply normalization internally
)

# Load z-score normalization
# Iterating through the dataset normalizes the input at every step
for x_norm, mask, y in dataset:
   ...  
   # Further processing as required


# Un-normalized values can still be extracted using the following
X, Mask, Y = dataset.x, dataset.y, dataset.mask

# Load mean normalization
normalizer = load_normalizer("mean", num_features=dataset.n_features)
for x, mask, y in zip(X, Mask, Y):
   x_norm = normalizer(x, mask)
   ...  # Further processing as required

Available Normalizers

Normalizer Name Description
decimal_scale Scales feature values by powers of 10
mean Online mean normalization by substracting running mean
minmax Scales features by the range (max-min) of the feature value observed
no_normalization Pass-through, no normalization applied
unit_vector Normalizes observed values into a unit-vector (scales by L2 norm)
zscore Online mean normalization using running mean and variance (substract mean, scale by variance)

Extending Normalization

Developers can register their own normalization schemes-see Contributing.


Versions

v1.2.8

  • Add code to remove empty rows in Crowdsense datasets.
  • Added GPU support for Aux-Drop and Aux-Net.
  • Minor code changes in DynFo for better logic implementation.

v1.2.7

  • Logic update in HI2 for choosing device ('cpu' or 'gpu') for running model.
  • Use tqdm instead of tqdm.auto.

v1.2.6

  • Added models Aux-Drop and Aux-Net.
  • Suppress numpy warning for loading Crowdsense(c3), Crowdsense(c5) and Spamassasin datasets.
  • Set OLIFL as a deterministic model.

v1.2.5

  • Some datasets, not intrinsically haphazard, (like wpbc, wbc) may have some NaN values in them. These feature values are treated as missing even in controlled haphazadness setting.

v1.2.4

  • Freeze required package versions.

v1.2.3

  • Remove internal normalization from HI2.
  • Freeze package versions of torch and tochvision for compatibility.

v1.2.2

  • Debug imports in HI2 model.

v1.2.1

  • Removed check for passing empty hyperparameter dict from BaseModel to support default hyperparameter values.

v1.2.0: datasets module changed to dataset_zoo

  • Added 15 dataset: WPBC, Ionosphere, WDBC, Australian, WBC, Diabetes-f, Crowdsense(c3), Crowdsense(c5), German, IPD, Svmguide3, KR-vs-KP, Spambase, Spamassasin and Diabetes US.
  • Set default value of mask scheme as "probabilistic". For datasets with intrinsic haphazardness scheme value is forced to "intrinsic".
  • Minor updates (name and comments) in A8A, GAS, SUSY, HIGGS and Dry Bean datasets.
  • Seed the random shuffle of IMDB dataset appropriately.

v1.1.8

  • Hardened seeding implementation in haphazard/utils/seeding.py.
  • Immplemented optional imports in HI2 model.
  • Update string regresentation of BaseDataset and BaseModel.
  • Strict data-typing of x, y and mask in DummyModel.

v1.1.7

  • Bug-fix: Resolve dtype of y in v2f.py for different criterion.

v1.1.6

  • Add task heads to V2F: Transformer, MLP, ODL
  • Available task heads for V2F: LSTM, Transformer, MLP, ODL

v1.1.5

  • Bug fix: Squeezed output logit of V2F for criterion compatibility

v1.1.4

  • Added model V2F

v1.1.3

  • Bug fix: HI2 is not deterministic.
  • Set self.determinitic = False in HI2 module.

v1.1.2

  • Add model HI2

v1.1.1

  • Added model HapTransformer
  • Update __repr__ of base_dataset

v1.1.0

Major Features

  • Added Normalization Framework

    • Introduced new module normalization/ with base and zoo submodules.
    • Built-in normalizers: mean, zscale, no_normalization, etc.
    • Unified registration via @register_normalizer.
    • Datasets and models now support integrated normalization.

Modifications

  • Updated:

    • data/base_dataset.py - normalization integration.
    • models/base_model.py - normalization compatibility.
    • model_zoo and datasets modules - decorator consistency.

v1.0.9

  • Added model FAE

  • Bug Fix

Update the X2 calculation in NB3 model

v1.0.8

  • Added model NB3.

v1.0.7

  • Bug Fix
  • Set RunOCDS.determministic = False as it uses random initialization.
  • Not passing 'tau' (or passing None) hyperparameter in OCDS will now result in using tau=np.sqrt(1.0/t) as a varied step size, as mentioned in OCDS paper (but not GLSC paper).

v1.0.6

  • Added datasets A8a, IMDB, Susy, and Higgs.

v1.0.5

  • Added model OCDS.

  • Bug Fixes and Improvements:

  • In haphazard/models/model_zoo/dynfo/dynfo.py:
    Updated the dropLearner() method to prevent errors when attempting to remove the last remaining weak learner.
    def dropLearner(self, i):
        if len(self.learners) == 1:
            return
        self.learners.pop(i)
        self.weights.pop(i)
        self.acceptedFeatures.pop(i)
        assert len(self.weights) == len(self.learners) == len(self.acceptedFeatures)
    
    This ensures stability in low-learner configurations and prevents IndexError during runtime.

v1.0.4

  • Added model ORF3V.

NOTE:

  • ORF3V also requires an initial buffer, which works similarly to DynFo.

  • ORF3V depends on the optional package tdigest, which requires Microsoft Visual C++ Build Tools.

  • To install with this dependency:

    1. Visit: https://visualstudio.microsoft.com/visual-cpp-build-tools/

    2. Download and install Build Tools for Visual Studio. During installation:

      • Select “Desktop development with C++” workload.
      • Ensure MSVC v143 or later, Windows 10/11 SDK, and CMake tools are checked.
  • After installation, restart your terminal and re-run:

    pip install haphazard[orf3v]
    

    or

    pip install haphazard[all]   # installs all optional dependencies
    
  • The package can still be used without installing tdigest; only the ORF3V model will be unavailable.

  • Bug Fixes and Improvements:
  • In haphazard/models/model_zoo/dynfo/__init__.py: corrected docstring from "Initialize the OLVF runner class." -> "Initialize the DynFo runner class."

  • In haphazard/models/model_zoo/dynfo/dynfo.py: changed

    return int(np.argmax(wc)), float(max(wc))
    

    to

    return int(np.argmax(wc)), float(wc[1])
    

    for correct AUROC/AUPRC compatibility.

v1.0.3

  • Added model DynFo

NOTE:

  • DynFo requires an initial buffer.
  • If no initial buffer size is provided, it is set to 1.
  • The length of the output labels/preds/logits is reduced by the initial buffer size.

v1.0.2

  • Added model OVFM

v1.0.0

(Considered to be the base version, ignore versions before this)

  • Includes models OLVF and OLIFL natively.
  • Includes datasets Magic04, Dry Bean and Gas. (Does not include raw files to read from, please use base_path argument to point to relevant path containing the raw files).

Contributing

Haphazard supports easy extensibility for new datasets, models, and normalizers.

Adding a new dataset

  1. Create a new folder under haphazard/data/dataset_zoo/, e.g., my_dataset/.
  2. Add __init__.py:
from ...base_dataset import BaseDataset
from ...dataset_zoo import register_dataset
import numpy as np

@register_dataset("my_dataset")
class MyDataset(BaseDataset):
    def __init__(self, base_path="./", **kwargs):
        self.name = "my_dataset"
        self.haphazard_type = "controlled"
        self.task = "classification"
        super().__init__(base_path=base_path, **kwargs)

    def read_data(self, base_path="./"):
        # Load or generate x, y
        x = np.random.random((100, 10))
        y = np.random.randint(0, 2, 100)
        return x, y
  1. The dataset is automatically registered and can be loaded with load_dataset("my_dataset").

Adding a new model

  1. Create a new folder under haphazard/models/model_zoo/, e.g., my_model/.
  2. Add __init__.py:
from ...base_model import BaseModel, BaseDataset
from ...model_zoo import register_model
import numpy as np

@register_model("my_model")
class MyModel(BaseModel):
    def __init__(self, **kwargs):
        self.name = "MyModel"
        self.tasks = {"classification", "regression"}
        self.deterministic = True
        self.hyperparameters = set()
        super().__init__(**kwargs)

    def fit(self, dataset: BaseDataset, model_params=None, seed=42):
        # Dummy implementation
        preds = []
        for x, mask, y in dataset:
            preds.append(int(np.random.randint(0, 2)))
        if dataset.task == "classification":
            return {
                "labels": y,
                "preds": preds,
                "logits": preds,
                "time_taken": 0.0,
                "is_logit": True
            }
        elif dataset.task == "regression":
            return {
                "targets": dataset.y,
                "preds": preds,
                "time_taken": 0.0,
            }
  1. The model is automatically registered and can be loaded with load_model("my_model").

Adding a New Normalizer

  1. Create a folder under:

    haphazard/normalization/normalizer_zoo/my_normalizer/
    
  2. Add __init__.py:

    from ...base_normalizer import OnlineNormalization
    from ...normalizer_zoo import register_normalizer
    import numpy as np
    from numpy.typing import NDArray
    
    @register_normalizer("my_normalizer")
    class MyNormalizer(OnlineNormalization):
        def __init__(self, num_features: int, replace_with: float | str = "nan"):
            # initialize required parameters
            super().__init__(num_features, replace_with)
    
        def update_params(self, x: NDArray[np.float64], indices: NDArray[np.int64]) -> None:
            # Update parameters
    
        def normalize(self, x: NDArray[np.float64], indices: NDArray[np.int64]) -> NDArray[np.float64]:
            # normalize x
            x_norm = ...
            return x_norm
    
  3. Load dynamically with:

    from haphazard import load_normalizer
    normalizer = load_normalizer("my_normalizer", num_features=10)
    

License

MIT License.

Metadata

Release files for haphazard 1.2.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for haphazard 1.2.8
File Size Uploaded Provenance Actions
haphazard-1.2.8.tar.gz 112.8 kB Without provenance Details for haphazard-1.2.8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for haphazard 1.2.8
File Interpreter ABI Platform Actions
haphazard-1.2.8-py3-none-any.whl Python 3 none any Details for haphazard-1.2.8-py3-none-any.whl

Total release size: 295.4 kB

Release files / haphazard-1.2.8.tar.gz

Download URL haphazard-1.2.8.tar.gz
Size 112.8 kB
Tags Source
SHA-256 checksum
How to use checksums
dd82d6a277c82bfbab2fe694c9a23d24b377e062b1ae2186f4d3e857f889a386
BLAKE2b-256 checksum
How to use checksums
4ffc8b72634b5319222d0719c2b04ccf1d465a10bb17f322869b8d873db8023b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / haphazard-1.2.8-py3-none-any.whl

Download URL haphazard-1.2.8-py3-none-any.whl
Size 182.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d2a168c71b98e9dc96a8f37b0808d7e796c235423630237892f5544cee60dc1f
BLAKE2b-256 checksum
How to use checksums
4f5fd2df436dc99402e2352ebb141be723f670640ef2dedb70576ddc5b136077
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for haphazard 1.2.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.8 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page