OMR — Omni Data Refinement
OMR is a pure Python framework for dataset quality, validation, profiling, monitoring, and reliability.
The goal is to become the standard tool developers use immediately after loading a dataset, similar to how Pandas is used for data manipulation.
OMR is designed to be Pure Python, Open Source, Modular, Extensible, Enterprise-ready, and entirely independent of external AI APIs or LLMs.
Core Vision
Instead of writing dozens of exploratory scripts, you just pass your dataset to OMR.
Current workflow:
import pandas as pd
df = pd.read_csv("data.csv")
# Write 50 lines of .isnull(), .duplicated(), and data exploration code...
Desired workflow:
import pandas as pd
from omr import Dataset
df = pd.read_csv("data.csv")
report = Dataset(df).health()
print(report.score) # e.g. 87
Capabilities
OMR handles 12 complete domains of data intelligence:
- Health Engine: 5-pillar quality score (Completeness, Uniqueness, Consistency, Validity, Conformity).
- Cleaning Engine: Auto-resolution of missing values, duplicates, and type mismatches.
- Profiling Engine: Deep statistical profile of every column.
- Validation Engine: Schema-based validation with typed constraints (
PositiveInteger,Email, etc). - Statistical Engine: Outlier detection, multicollinearity, skewness, class imbalance.
- Drift Engine: Distribution shift detection using PSI, KS Test, and JS Divergence.
- Monitoring System: Continuous tracking and alerting for data decay.
- Explainability: Rule-based explanations for data science issues.
- Versioning: Snapshots and rollback capabilities for transformations.
- Reporting: Export to HTML, Markdown, or JSON.
- Pipelines: Fluent, chainable API for applying operations.
- Plugin Registry: Extensible via third-party domain-specific packages.
Installation
pip install omni-data-refinement
Dependencies: pandas, numpy, rich. (Optional: scipy for KS Drift tests).
Compatible with Pandas, Polars, and NumPy.
Quickstart
1. Diagnostic Health Check
Get an immediate quality score and detailed issue list.
import pandas as pd
from omr import Dataset
df = pd.read_csv("messy_data.csv")
dataset = Dataset(df)
# Run health check
report = dataset.health()
print(f"Health Score: {report.score}/100")
2. Auto-Cleaning
Automatically fix missing values, duplicate rows, and mixed data types.
dataset.clean()
dataset.explain_changes() # View transformation log
3. Schema Validation
Enforce strict business rules on your dataset.
from omr import schemas
schema = {
"age": schemas.PositiveInteger(max=120),
"salary": schemas.PositiveFloat(min=10000),
"status": schemas.OneOf("active", "inactive"),
"email": schemas.Email()
}
dataset.validate(schema)
4. Advanced Analytics & Drift
Detect complex issues or compare against production data.
# Detect outliers, multicollinearity, and zero variance features
dataset.analyze()
# Compare against yesterday's data to detect drift
prod_dataset = Dataset(pd.read_csv("prod_data.csv"))
dataset.compare(prod_dataset)
5. Fluent API (Pipelines)
Chain commands together.
clean_df = (Dataset(df)
.health()
.clean()
.analyze()
.export())
Architecture (SOLID Principles)
OMR is modular and testable. The architecture is cleanly divided into:
omr.core: The mainDatasetinterface andDatasetMetadatastate tracking.omr.health,omr.cleaning,omr.profiling,omr.validation,omr.statistics,omr.drift,omr.monitoring,omr.explainability,omr.versioning,omr.pipelines,omr.reports,omr.plugins,omr.integrations,omr.utils,omr.schemas.
Extensibility (Plugins)
OMR supports domain-specific extensions.
from omr import register_plugin
@register_plugin("medical")
class MedicalPlugin:
def validate_ehr(self, df):
pass
License
BSD 3-Clause License. Copyright (c) 2026 Omar Alshafai. See LICENSE for details.
Metadata
Release files for omni-data-refinement 0.3.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| omni_data_refinement-0.3.7.tar.gz | 60.9 kB | Without provenance | Details for omni_data_refinement-0.3.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| omni_data_refinement-0.3.7-py3-none-any.whl | Python 3 | none | any | Details for omni_data_refinement-0.3.7-py3-none-any.whl |
Total release size: 118.9 kB
Release files / omni_data_refinement-0.3.7.tar.gz
| Download URL | omni_data_refinement-0.3.7.tar.gz |
|---|---|
| Size | 60.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7f47a164cca19b211ee8c890d5e8d80857e1d2e4d444faee3055a25e35212ad5
|
|
BLAKE2b-256 checksum How to use checksums |
ab6304dc535a9cc3776424fba1677a71733670ccd08668bff993bd3040f67cc1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.14
|
Release files / omni_data_refinement-0.3.7-py3-none-any.whl
| Download URL | omni_data_refinement-0.3.7-py3-none-any.whl |
|---|---|
| Size | 57.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
763d773e8c15daa1a835f5c0174b8cabcefd345937b979ac82774b890d318881
|
|
BLAKE2b-256 checksum How to use checksums |
b073bfc6af2cfa1b4fa098a7402a501179cf94492d261f014e85b01a8d980c0a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.14
|
Security for omni-data-refinement 0.3.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.