Skip to main content

OMR — Omni Data Refinement

OMR is a pure Python framework for dataset quality, validation, profiling, monitoring, and reliability.

The goal is to become the standard tool developers use immediately after loading a dataset, similar to how Pandas is used for data manipulation.

OMR is designed to be Pure Python, Open Source, Modular, Extensible, Enterprise-ready, and entirely independent of external AI APIs or LLMs.

Core Vision

Instead of writing dozens of exploratory scripts, you just pass your dataset to OMR.

Current workflow:

import pandas as pd
df = pd.read_csv("data.csv")
# Write 50 lines of .isnull(), .duplicated(), and data exploration code...

Desired workflow:

import pandas as pd
from omr import Dataset

df = pd.read_csv("data.csv")
report = Dataset(df).health()

print(report.score)  # e.g. 87

Capabilities

OMR handles 12 complete domains of data intelligence:

  1. Health Engine: 5-pillar quality score (Completeness, Uniqueness, Consistency, Validity, Conformity).
  2. Cleaning Engine: Auto-resolution of missing values, duplicates, and type mismatches.
  3. Profiling Engine: Deep statistical profile of every column.
  4. Validation Engine: Schema-based validation with typed constraints (PositiveInteger, Email, etc).
  5. Statistical Engine: Outlier detection, multicollinearity, skewness, class imbalance.
  6. Drift Engine: Distribution shift detection using PSI, KS Test, and JS Divergence.
  7. Monitoring System: Continuous tracking and alerting for data decay.
  8. Explainability: Rule-based explanations for data science issues.
  9. Versioning: Snapshots and rollback capabilities for transformations.
  10. Reporting: Export to HTML, Markdown, or JSON.
  11. Pipelines: Fluent, chainable API for applying operations.
  12. Plugin Registry: Extensible via third-party domain-specific packages.

Installation

pip install omni-data-refinement

Dependencies: pandas, numpy, rich. (Optional: scipy for KS Drift tests). Compatible with Pandas, Polars, and NumPy.


Quickstart

1. Diagnostic Health Check

Get an immediate quality score and detailed issue list.

import pandas as pd
from omr import Dataset

df = pd.read_csv("messy_data.csv")
dataset = Dataset(df)

# Run health check
report = dataset.health()
print(f"Health Score: {report.score}/100")

2. Auto-Cleaning

Automatically fix missing values, duplicate rows, and mixed data types.

dataset.clean()
dataset.explain_changes() # View transformation log

3. Schema Validation

Enforce strict business rules on your dataset.

from omr import schemas

schema = {
    "age": schemas.PositiveInteger(max=120),
    "salary": schemas.PositiveFloat(min=10000),
    "status": schemas.OneOf("active", "inactive"),
    "email": schemas.Email()
}
dataset.validate(schema)

4. Advanced Analytics & Drift

Detect complex issues or compare against production data.

# Detect outliers, multicollinearity, and zero variance features
dataset.analyze()

# Compare against yesterday's data to detect drift
prod_dataset = Dataset(pd.read_csv("prod_data.csv"))
dataset.compare(prod_dataset)

5. Fluent API (Pipelines)

Chain commands together.

clean_df = (Dataset(df)
    .health()
    .clean()
    .analyze()
    .export())

Architecture (SOLID Principles)

OMR is modular and testable. The architecture is cleanly divided into:

  • omr.core: The main Dataset interface and DatasetMetadata state tracking.
  • omr.health, omr.cleaning, omr.profiling, omr.validation, omr.statistics, omr.drift, omr.monitoring, omr.explainability, omr.versioning, omr.pipelines, omr.reports, omr.plugins, omr.integrations, omr.utils, omr.schemas.

Extensibility (Plugins)

OMR supports domain-specific extensions.

from omr import register_plugin

@register_plugin("medical")
class MedicalPlugin:
    def validate_ehr(self, df):
        pass

License

BSD 3-Clause License. Copyright (c) 2026 Omar Alshafai. See LICENSE for details.

Metadata

Release files for omni-data-refinement 0.3.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for omni-data-refinement 0.3.7
File Size Uploaded Provenance Actions
omni_data_refinement-0.3.7.tar.gz 60.9 kB Without provenance Details for omni_data_refinement-0.3.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for omni-data-refinement 0.3.7
File Interpreter ABI Platform Actions
omni_data_refinement-0.3.7-py3-none-any.whl Python 3 none any Details for omni_data_refinement-0.3.7-py3-none-any.whl

Total release size: 118.9 kB

Release files / omni_data_refinement-0.3.7.tar.gz

Download URL omni_data_refinement-0.3.7.tar.gz
Size 60.9 kB
Tags Source
SHA-256 checksum
How to use checksums
7f47a164cca19b211ee8c890d5e8d80857e1d2e4d444faee3055a25e35212ad5
BLAKE2b-256 checksum
How to use checksums
ab6304dc535a9cc3776424fba1677a71733670ccd08668bff993bd3040f67cc1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.14

Release files / omni_data_refinement-0.3.7-py3-none-any.whl

Download URL omni_data_refinement-0.3.7-py3-none-any.whl
Size 57.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
763d773e8c15daa1a835f5c0174b8cabcefd345937b979ac82774b890d318881
BLAKE2b-256 checksum
How to use checksums
b073bfc6af2cfa1b4fa098a7402a501179cf94492d261f014e85b01a8d980c0a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.14

Security for omni-data-refinement 0.3.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page