data-reliability-index 0.2.0

Data Reliability Index - native validation and filtering of datasets.

Data Reliability Index

Data Reliability Index is a Python package for attaching reliability metadata to data points, enforcing trust policies, and filtering unreliable records before they reach analysis or API boundaries.

The package is built around a simple rule: data should carry the evidence needed to decide whether it is safe to use.

Features

• Pydantic models for reliability metadata and policies.
• Scanning engine for computing reliability scores from validation evidence.
• Tiered trust classification with scores from 0 to 100.
• Policy-based acceptance checks for individual records.
• Pandas helpers for filtering DataFrames by reliability metadata.
• FastAPI example for rejecting low-reliability input at ingestion time.
• MkDocs documentation for concepts and API usage.

Installation

pip install data-reliability-index

For local development from this repository:

pip install -e ".[test]"

Quick Start

from data_reliability import DataTier, ReliabilityPolicy, ReliabilityScanner, ValidationEvidence

scanner = ReliabilityScanner()
data = scanner.scan(
    {"temperature": 21.4, "unit": "celsius"},
    source_id="sensor-a",
    evidence=ValidationEvidence(
        completeness=1.0,
        consistency=1.0,
        provenance=1.0,
        cryptographic_verification=1.0,
        calibration=1.0,
        schema_compliance=1.0,
        anomaly_detection=1.0,
        duplicate_detection=1.0,
        metadata_quality=1.0,
    ),
)

policy = ReliabilityPolicy(
    minimum_score=90,
    maximum_tier=DataTier.TIER_2,
)

assert policy.resolve(data) == {"temperature": 21.4, "unit": "celsius"}

Pandas Filtering

import pandas as pd
from data_reliability import DataTier, ReliabilityMetadata, ReliabilityPolicy, filter_reliable_df

df = pd.DataFrame([
    {
        "value": 10,
        "reliability": ReliabilityMetadata(
            score=95,
            tier=DataTier.TIER_1,
            source_id="sensor-a",
            trace_hash="abc123",
        ),
    },
])

policy = ReliabilityPolicy(minimum_score=90, maximum_tier=DataTier.TIER_2)
trusted = filter_reliable_df(df, policy)

Documentation

The project documentation lives in docs/ and can be served locally with MkDocs:

pip install mkdocs-material
mkdocs serve

Start with:

• Concepts
• Core models
• Pandas extension
• FastAPI example

The longer project rationale is available in data-reliability.md.

Development

Run the test suite:

pip install -e ".[test]"
pytest

Build package artifacts:

pip install -e ".[build]"
python -m build
python -m twine check dist/*

Run the FastAPI example:

uvicorn examples.fastapi_app:app --reload

Contributing

Contributions are welcome. See CONTRIBUTING.md for the local workflow and pull request expectations.

Security

Please report security issues privately. See SECURITY.md.

License

Licensed under the Apache License 2.0.