LDIMBenchmark

Leakage Detection and Isolation Methods Benchmark

Instead of collecting all the different dataset to benchmark different methods on. We wanted to create a Benchmarking Tool which makes it easy to reproduce the results of the different methods locally on your own dataset.

It provides a close to real-world conditions environment and forces researchers to provide a reproducible method implementation, which is supposed to run automated on any input dataset, thus hindering custom solutions which work well in one specific case.

Usage

Installation

pip install ldimbenchmark

Python

from ldimbenchmark.datasets import DatasetLibrary, DATASETS
from ldimbenchmark import (
    LDIMBenchmark,
    BenchmarkData,
    BenchmarkLeakageResult,
)
from ldimbenchmark.classes import LDIMMethodBase
from typing import List

class YourCustomLDIMMethod(LDIMMethodBase):
    def __init__(self):
        super().__init__(
            name="YourCustomLDIMMethod",
            version="0.1.0"
        )

    def train(self, data: BenchmarkData):
        pass

    def detect(self, data: BenchmarkData) -> List[BenchmarkLeakageResult]:
        return [
            {
                "leak_start": "2020-01-01",
                "leak_end": "2020-01-02",
                "leak_area": 0.2,
                "pipe_id": "test",
            }
        ]

    def detect_datapoint(self, evaluation_data) -> BenchmarkLeakageResult:
        return {}


datasets = DatasetLibrary("datasets").download(DATASETS.BATTLEDIM)

local_methods = [YourCustomLDIMMethod()]

hyperparameters = {}

benchmark = LDIMBenchmark(
    hyperparameters, datasets, results_dir="./benchmark-results"
)
benchmark.add_local_methods(local_methods)

benchmark.run_benchmark()

benchmark.evaluate()

CLI

ldimbenchmark --help

Roadmap

v1: Just Leakage Detection
v2: Provides Benchmark of Isolation Methods

https://mathspp.com/blog/how-to-create-a-python-package-in-2022

Development

https://python-poetry.org/docs/basic-usage/

# python 3.10
# Use Environment
poetry config virtualenvs.in-project true
poetry shell
poetry install --without ci # --with ci


# Test
poetry build
cp -r dist tests/dist
cd tests
docker build . -t testmethod
pytest -s -o log_cli=true
pytest tests/test_derivation.py -k 'test_mything'
pytest --testmon
pytest --snapshot-update

# Pytest watch
ptw
ptw -- --testmon

# Watch a file during development
npm install -g nodemon
nodemon -L experiments/auto_hyperparameter.py

# Test-Publish
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry config http-basic.testpypi __token__ pypi-your-api-token-here
poetry build
poetry publish -r testpypi

# Real Publish
poetry config pypi-token.pypi pypi-your-token-here

Documentation

https://squidfunk.github.io/mkdocs-material/ https://click.palletsprojects.com/en/8.1.x/

poetry shell
mkdocs serve

TODO

LDIMBenchmark: Data Cleansing before working with them

per sensor type, e.g. waterflow (cut of at 0)
removing datapoints which are clearly a malfunction

Metadata

Release files for ldimbenchmark 0.2.67

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ldimbenchmark 0.2.67
File	Size	Uploaded
ldimbenchmark-0.2.67.tar.gz	72.8 kB	Jul 29, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for ldimbenchmark 0.2.67
File	Interpreter	ABI	Platform	Reset
ldimbenchmark-0.2.67-py3-none-any.whl 90.5 kB Jul 29, 2023	Python 3	none	any	Details

Total release size: 163.3 kB

Release files / ldimbenchmark-0.2.67.tar.gz

Download URL	ldimbenchmark-0.2.67.tar.gz
Size	72.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`e62d58e7104a6328d0c22e25d0bc49818417159b0fb6b64037bf5f3a42de9a1e`
BLAKE2b-256 checksum How to use checksums	`34906623ad082b92048f043026e9ce592a33637678e864bb3cb0b717f7ceb122`
Upload date	Jul 29, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.8.0 pkginfo/1.9.6 readme-renderer/37.3 requests/2.28.2 requests-toolbelt/0.10.1 urllib3/1.26.14 tqdm/4.64.1 importlib-metadata/6.0.0 keyring/23.13.1 rfc3986/2.0.0 colorama/0.4.6 CPython/3.10.12`

Release files / ldimbenchmark-0.2.67-py3-none-any.whl

Download URL	ldimbenchmark-0.2.67-py3-none-any.whl
Size	90.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`085c388cb12044bfe32e9c688bf224162782b3219b9ab09bfd5f758648c0421e`
BLAKE2b-256 checksum How to use checksums	`d50ae40253cd450fbd91e00aeb54174422981490c24c42f5e6071933ee45e627`
Upload date	Jul 29, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.8.0 pkginfo/1.9.6 readme-renderer/37.3 requests/2.28.2 requests-toolbelt/0.10.1 urllib3/1.26.14 tqdm/4.64.1 importlib-metadata/6.0.0 keyring/23.13.1 rfc3986/2.0.0 colorama/0.4.6 CPython/3.10.12`