iot-dqa 0.0.4

Fast and efficient data quality assessment for IoT timeseries data.

IoT-DQA

The IoT-DQA library is a Python package designed to streamline Data Quality Assessment (DQA) for IoT time-series data. It provides robust tools for validating and analyzing IoT data streams, ensuring reliable data for downstream applications.

---

Documentation: https://jeafreezy.github.io/iot-dqa/

Source Code: https://github.com/jeafreezy/iot-dqa

---

Key Features

• Optimized Performance: Handles large-scale IoT datasets efficiently, powered by the high-performance Polars library.
• Streamlined Validation: Simplifies the process of validating and analyzing IoT data streams.
• Custom Metrics: Tailor metrics to meet specific requirements.
• Comprehensive Scoring: Generates detailed data quality scores across multiple dimensions.
• Seamless Integration: Export results in formats like CSV and GeoJSON for easy integration with other tools.

Dimensions of Data Quality

• Validity: Verifies data adherence to expected formats and ranges.
• Accuracy: Identifies and quantifies outliers using advanced techniques.
• Completeness: Evaluates the presence of missing or null values.
• Timeliness: Measures data arrival punctuality based on timestamps.

Note:

• Designed for cumulative time-series data (e.g., utility consumption).
• Sample data is available in tests/test_data.csv.

Installation

pip install iot-dqa

Quick Start

Example: Calculate Data Quality Score for IoT time-series data

from iot_dqa import DataQualityScore, Dimension, OutlierDetectionAlgorithm, CompletenessStrategy

# Initialize and compute the Data Quality Score
dq_score = DataQualityScore(
    "./data/sample_data.csv",
    multiple_devices=True,
    dimensions=[
        Dimension.VALIDITY.value,
        Dimension.ACCURACY.value,
        Dimension.COMPLETENESS.value,
        Dimension.TIMELINESS.value,
    ],
    col_mapping={
        "latitude": "LAT",
        "longitude": "LONG",
        "date": "DATE",
        "value": "VALUE",
        "id": "DEVICE_ID",
    },
    metrics_config={
        "timeliness": {"iat_method": "min"},
        "accuracy": {
            "ensemble": True,
            "strategy": "validity",
            "algorithms": [
                OutlierDetectionAlgorithm.IF.value,
                OutlierDetectionAlgorithm.IQR.value,
                OutlierDetectionAlgorithm.MAD.value,
            ],
        },
        "completeness_strategy": CompletenessStrategy.ONLY_NULLS.value,
    },
).compute_score(
    weighting_mechanism="ahp",
    output_format="geojson",
    output_path="./output",
    ahp_weights={
        Dimension.VALIDITY.value: 0.3,
        Dimension.ACCURACY.value: 0.3,
        Dimension.COMPLETENESS.value: 0.3,
        Dimension.TIMELINESS.value: 0.1,
    },
)

print("Data Quality Score computed successfully!")

Configuration Overview

Configuration	Attribute	Default Value	Description
Isolation Forest	n_estimators	100	Number of trees in the forest.
	max_samples	0.8	Proportion of samples for training each base estimator.
	contamination	0.1	Proportion of outliers in the dataset.
	max_features	1	Number of features for training each base estimator.
	random_state	42	Random seed for reproducibility.
Accuracy	ensemble	True	Use ensemble methods for accuracy.
	mad_threshold	3	Threshold for Median Absolute Deviation (MAD).
	optimize_iqr_with_optuna	True	Enable IQR optimization using Optuna.
	iqr_optuna_q1_max	0.5	Maximum value for Q1 in IQR optimization.
	iqr_optuna_q3_min	0.5	Minimum value for Q3 in IQR optimization.
	iqr_optuna_q3_max	1	Maximum value for Q3 in IQR optimization.
	algorithms	All algorithms	List of outlier detection algorithms.
	strategy	NONE	Strategy for accuracy computation.
Timeliness	iat_method	min	Method to calculate inter-arrival time.
Completeness	completeness_strategy	ONLY_NULLS	Strategy for handling completeness.

For more details on configuration, refer to the documentation.

Documentation

Visit the documentation for comprehensive details.

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License. See the LICENSE file for more information.