IoT-DQA

The IoT-DQA library is a Python package designed to streamline Data Quality Assessment (DQA) for IoT time-series data. It provides robust tools for validating and analyzing IoT data streams, ensuring reliable data for downstream applications.

Documentation: https://jeafreezy.github.io/iot-dqa/

Source Code: https://github.com/jeafreezy/iot-dqa

Key Features

• Optimized Performance: Handles large-scale IoT datasets efficiently, powered by the high-performance Polars library.
• Streamlined Validation: Simplifies the process of validating and analyzing IoT data streams.
• Custom Metrics: Tailor metrics to meet specific requirements.
• Comprehensive Scoring: Generates detailed data quality scores across multiple dimensions.
• Seamless Integration: Export results in formats like CSV and GeoJSON for easy integration with other tools.

Dimensions of Data Quality

• Validity: Verifies data adherence to expected formats and ranges.
• Accuracy: Identifies and quantifies outliers using advanced techniques.
• Completeness: Evaluates the presence of missing or null values.
• Timeliness: Measures data arrival punctuality based on timestamps.

Note:

• Designed for cumulative time-series data (e.g., utility consumption).
• Sample data is available in tests/test_data.csv.

Installation

pip install iot-dqa

Quick Start

Example: Calculate Data Quality Score for IoT time-series data

from iot_dqa import DataQualityScore, Dimension, OutlierDetectionAlgorithm, CompletenessStrategy

# Initialize and compute the Data Quality Score
dq_score = DataQualityScore(
    "./data/sample_data.csv",
    multiple_devices=True,
    dimensions=[
        Dimension.VALIDITY.value,
        Dimension.ACCURACY.value,
        Dimension.COMPLETENESS.value,
        Dimension.TIMELINESS.value,
    ],
    col_mapping={
        "latitude": "LAT",
        "longitude": "LONG",
        "date": "DATE",
        "value": "VALUE",
        "id": "DEVICE_ID",
    },
    metrics_config={
        "timeliness": {"iat_method": "min"},
        "accuracy": {
            "ensemble": True,
            "strategy": "validity",
            "algorithms": [
                OutlierDetectionAlgorithm.IF.value,
                OutlierDetectionAlgorithm.IQR.value,
                OutlierDetectionAlgorithm.MAD.value,
            ],
        },
        "completeness_strategy": CompletenessStrategy.ONLY_NULLS.value,
    },
).compute_score(
    weighting_mechanism="ahp",
    output_format="geojson",
    output_path="./output",
    ahp_weights={
        Dimension.VALIDITY.value: 0.3,
        Dimension.ACCURACY.value: 0.3,
        Dimension.COMPLETENESS.value: 0.3,
        Dimension.TIMELINESS.value: 0.1,
    },
)

print("Data Quality Score computed successfully!")

Configuration Overview

For more details on configuration, refer to the documentation.

Documentation

Visit the documentation for comprehensive details.

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License. See the LICENSE file for more information.