Skip to main content

No project description provided

Project description

ldimbenchmark version Documentation badge

LDIMBenchmark

Leakage Detection and Isolation Methods Benchmark

Instead of collecting all the different dataset to benchmark different methods on. We wanted to create a Benchmarking Tool which makes it easy to reproduce the results of the different methods locally on your own dataset.

It provides a close to real-world conditions environment and forces researchers to provide a reproducible method implementation, which is supposed to run automated on any input dataset, thus hindering custom solutions which work well in one specific case.

Usage

Installation

pip install ldimbenchmark

Python

from ldimbenchmark.datasets import DatasetLibrary, DATASETS
from ldimbenchmark import (
    LDIMBenchmark,
    BenchmarkData,
    BenchmarkLeakageResult,
)
from ldimbenchmark.classes import LDIMMethodBase
from typing import List

class YourCustomLDIMMethod(LDIMMethodBase):
    def __init__(self):
        super().__init__(
            name="YourCustomLDIMMethod",
            version="0.1.0"
        )

    def train(self, data: BenchmarkData):
        pass

    def detect(self, data: BenchmarkData) -> List[BenchmarkLeakageResult]:
        return [
            {
                "leak_start": "2020-01-01",
                "leak_end": "2020-01-02",
                "leak_area": 0.2,
                "pipe_id": "test",
            }
        ]

    def detect_datapoint(self, evaluation_data) -> BenchmarkLeakageResult:
        return {}


datasets = DatasetLibrary("datasets").download(DATASETS.BATTLEDIM)

local_methods = [YourCustomLDIMMethod()]

hyperparameters = {}

benchmark = LDIMBenchmark(
    hyperparameters, datasets, results_dir="./benchmark-results"
)
benchmark.add_local_methods(local_methods)

benchmark.run_benchmark()

benchmark.evaluate()

CLI

ldimbenchmark --help

Roadmap

  • v1: Just Leakage Detection
  • v2: Provides Benchmark of Isolation Methods

https://mathspp.com/blog/how-to-create-a-python-package-in-2022

Development

https://python-poetry.org/docs/basic-usage/

# python 3.10
# Use Environment
poetry config virtualenvs.in-project true
poetry shell
poetry install --without ci # --with ci


# Test
poetry build
cp -r dist tests/dist
cd tests
docker build . -t testmethod
pytest -s -o log_cli=true
pytest tests/test_derivation.py -k 'test_mything'
pytest --testmon
pytest --snapshot-update

# Pytest watch
ptw
ptw -- --testmon

# Watch a file during development
npm install -g nodemon
nodemon -L experiments/auto_hyperparameter.py

# Test-Publish
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry config http-basic.testpypi __token__ pypi-your-api-token-here
poetry build
poetry publish -r testpypi

# Real Publish
poetry config pypi-token.pypi pypi-your-token-here

Documentation

https://squidfunk.github.io/mkdocs-material/ https://click.palletsprojects.com/en/8.1.x/

poetry shell
mkdocs serve

TODO

LDIMBenchmark: Data Cleansing before working with them

  • per sensor type, e.g. waterflow (cut of at 0)
  • removing datapoints which are clearly a malfunction

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ldimbenchmark-0.2.67.tar.gz (72.8 kB view details)

Uploaded Source

Built Distribution

ldimbenchmark-0.2.67-py3-none-any.whl (90.5 kB view details)

Uploaded Python 3

File details

Details for the file ldimbenchmark-0.2.67.tar.gz.

File metadata

  • Download URL: ldimbenchmark-0.2.67.tar.gz
  • Upload date:
  • Size: 72.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.9.6 readme-renderer/37.3 requests/2.28.2 requests-toolbelt/0.10.1 urllib3/1.26.14 tqdm/4.64.1 importlib-metadata/6.0.0 keyring/23.13.1 rfc3986/2.0.0 colorama/0.4.6 CPython/3.10.12

File hashes

Hashes for ldimbenchmark-0.2.67.tar.gz
Algorithm Hash digest
SHA256 e62d58e7104a6328d0c22e25d0bc49818417159b0fb6b64037bf5f3a42de9a1e
MD5 c12a47ef113c0e6fba078f64ab658c78
BLAKE2b-256 34906623ad082b92048f043026e9ce592a33637678e864bb3cb0b717f7ceb122

See more details on using hashes here.

File details

Details for the file ldimbenchmark-0.2.67-py3-none-any.whl.

File metadata

  • Download URL: ldimbenchmark-0.2.67-py3-none-any.whl
  • Upload date:
  • Size: 90.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.9.6 readme-renderer/37.3 requests/2.28.2 requests-toolbelt/0.10.1 urllib3/1.26.14 tqdm/4.64.1 importlib-metadata/6.0.0 keyring/23.13.1 rfc3986/2.0.0 colorama/0.4.6 CPython/3.10.12

File hashes

Hashes for ldimbenchmark-0.2.67-py3-none-any.whl
Algorithm Hash digest
SHA256 085c388cb12044bfe32e9c688bf224162782b3219b9ab09bfd5f758648c0421e
MD5 6210c19c3194425db0f8a2a9ffc92525
BLAKE2b-256 d50ae40253cd450fbd91e00aeb54174422981490c24c42f5e6071933ee45e627

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page