No project description provided
Project description
LDIMBenchmark
Leakage Detection and Isolation Methods Benchmark
Instead of collecting all the different dataset to benchmark different methods on. We wanted to create a Benchmarking Tool which makes it easy to reproduce the results of the different methods locally on your own dataset.
It provides a close to real-world conditions environment and forces researchers to provide a reproducible method implementation, which is supposed to run automated on any input dataset, thus hindering custom solutions which work well in one specific case.
Usage
Installation
pip install ldimbenchmark
Python
from ldimbenchmark.datasets import DatasetLibrary, DATASETS
from ldimbenchmark import (
LDIMBenchmark,
BenchmarkData,
BenchmarkLeakageResult,
)
from ldimbenchmark.classes import LDIMMethodBase
from typing import List
class YourCustomLDIMMethod(LDIMMethodBase):
def __init__(self):
super().__init__(
name="YourCustomLDIMMethod",
version="0.1.0"
)
def train(self, data: BenchmarkData):
pass
def detect(self, data: BenchmarkData) -> List[BenchmarkLeakageResult]:
return [
{
"leak_start": "2020-01-01",
"leak_end": "2020-01-02",
"leak_area": 0.2,
"pipe_id": "test",
}
]
def detect_datapoint(self, evaluation_data) -> BenchmarkLeakageResult:
return {}
datasets = DatasetLibrary("datasets").download(DATASETS.BATTLEDIM)
local_methods = [YourCustomLDIMMethod()]
hyperparameters = {}
benchmark = LDIMBenchmark(
hyperparameters, datasets, results_dir="./benchmark-results"
)
benchmark.add_local_methods(local_methods)
benchmark.run_benchmark()
benchmark.evaluate()
CLI
ldimbenchmark --help
Roadmap
- v1: Just Leakage Detection
- v2: Provides Benchmark of Isolation Methods
https://mathspp.com/blog/how-to-create-a-python-package-in-2022
Development
https://python-poetry.org/docs/basic-usage/
# python 3.10
# Use Environment
poetry config virtualenvs.in-project true
poetry shell
poetry install --without ci # --with ci
# Test
poetry build
cp -r dist tests/dist
cd tests
docker build . -t testmethod
pytest -s -o log_cli=true
pytest tests/test_derivation.py -k 'test_mything'
pytest --testmon
pytest --snapshot-update
# Pytest watch
ptw
ptw -- --testmon
# Watch a file during development
npm install -g nodemon
nodemon -L experiments/auto_hyperparameter.py
# Test-Publish
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry config http-basic.testpypi __token__ pypi-your-api-token-here
poetry build
poetry publish -r testpypi
# Real Publish
poetry config pypi-token.pypi pypi-your-token-here
Documentation
https://squidfunk.github.io/mkdocs-material/ https://click.palletsprojects.com/en/8.1.x/
poetry shell
mkdocs serve
TODO
LDIMBenchmark: Data Cleansing before working with them
- per sensor type, e.g. waterflow (cut of at 0)
- removing datapoints which are clearly a malfunction
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file ldimbenchmark-0.2.67.tar.gz
.
File metadata
- Download URL: ldimbenchmark-0.2.67.tar.gz
- Upload date:
- Size: 72.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.9.6 readme-renderer/37.3 requests/2.28.2 requests-toolbelt/0.10.1 urllib3/1.26.14 tqdm/4.64.1 importlib-metadata/6.0.0 keyring/23.13.1 rfc3986/2.0.0 colorama/0.4.6 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e62d58e7104a6328d0c22e25d0bc49818417159b0fb6b64037bf5f3a42de9a1e |
|
MD5 | c12a47ef113c0e6fba078f64ab658c78 |
|
BLAKE2b-256 | 34906623ad082b92048f043026e9ce592a33637678e864bb3cb0b717f7ceb122 |
File details
Details for the file ldimbenchmark-0.2.67-py3-none-any.whl
.
File metadata
- Download URL: ldimbenchmark-0.2.67-py3-none-any.whl
- Upload date:
- Size: 90.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.9.6 readme-renderer/37.3 requests/2.28.2 requests-toolbelt/0.10.1 urllib3/1.26.14 tqdm/4.64.1 importlib-metadata/6.0.0 keyring/23.13.1 rfc3986/2.0.0 colorama/0.4.6 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 085c388cb12044bfe32e9c688bf224162782b3219b9ab09bfd5f758648c0421e |
|
MD5 | 6210c19c3194425db0f8a2a9ffc92525 |
|
BLAKE2b-256 | d50ae40253cd450fbd91e00aeb54174422981490c24c42f5e6071933ee45e627 |