DET Quant Library
Table of Contents
Overview
The DET Quant Library is an internal library containing functions and classes that can be used across Quant models.
Package Contents
The package is split into subpackages, one per functional area. The sections below briefly describe each of them. See List of Exposed Symbols for the symbols that can be imported directly from each subpackage.
Assets
The detquantlib.assets package models physical energy assets:
BatteryStatestores the characteristics and current state of a battery (capacity, power, state of charge, depth of discharge, roundtrip efficiency). Live battery states can be loaded from Redis withBatteryState.build_from_redis(), for the Eniris and Smartbox energy management systems.WindTurbinestores wind turbine information (rated power, cut-in and cut-out wind speeds, number of turbines, power curve) and converts wind speeds to power or energy.WindTurbineModelscontains the specifications of known turbine models, which can be loaded withWindTurbine.build_from_model().
Converters
The detquantlib.converters package converts values between units:
convert_power()andconvert_energy()convert between power units and between energy units.power_to_energy()andenergy_to_power()convert between power and energy, for a given tenor.convert_price()converts a price from one currency and volume unit to another, andflag_cents()detects whether a currency string is expressed in cents.
Data
The detquantlib.data package provides clients for the data sources used by Quant models:
SnowflakeClientandAzureClientare small, safe clients for Snowflake and Azure SQL databases, configured withSnowflakeConfigandAzureConfig(which can be read from environment variables). They can query, load, list and write tables, and exchange data as pandas DataFrames.DetDatabaseis a client for the DET database, with dedicated methods to load and process data such as ENTSOE day-ahead and imbalance prices, futures settlement prices, commodities, instruments, account positions, customer volume forecasts and clients.list_to_sql_set()andlist_to_sql_columns()format Python lists for use in SQL queries.Sftpimports and exports files from and to an SFTP server.Entsoeimports and processes ENTSOE data, such as day-ahead spot prices from the ENTSOE SFTP server.TennetClientis a wrapper around the TenneT publications API, to fetch settlement prices and balance delta data.
Dates
The detquantlib.dates package contains date and time helpers: rounding timestamps up or down
to a given frequency (ts_ceil(), ts_floor()), counting the number of hours between two dates
(count_nr_hours()), computing the number of months between two dates (calc_months_diff()),
and converting dates to month codes (datetime_to_month_code()).
Figures
The detquantlib.figures.plotly_figures module contains helpers for Plotly figures: applying a
standard layout, saving figures to JSON or HTML files, and displaying figures saved as JSON.
Forecasting
The detquantlib.forecasting package contains forecasting strategies. forecast_knife_strategy()
generates a random walk forecast, where each value is set to the most recent observed value with
the same time of day and the same day type (weekday, Saturday or Sunday).
Outputs
The detquantlib.outputs package manages and exports model outputs. OutputItem holds a single
output and exports it to a CSV, JSON, HTML or NPZ file. OutputSet groups several OutputItem
objects, and PathDefinitions defines the folders the outputs are exported to (by default an Outputs
folder in the working directory, or the OUTPUT_DIR environment variable if set).
Snapshot Testing
The detquantlib.snapshot_testing package provides snapshot (regression) tests for Quant models:
each test runs a model end to end on a fixed set of inputs, and compares its outputs with a
reviewed reference copy. Database calls are recorded once, locally, and replayed from disk in CI,
which has no database access. The SnowflakeClient and AzureClient classes are supported out of
the box.
See the snapshot testing guide for how the framework works and how to set it up in a model repository.
Stats
The detquantlib.stats.data_analysis module contains statistical analysis tools.
filter_outliers() iteratively removes outliers from a time series, based on a rolling mean and
standard deviation, and create_outliers_iteration_plot() plots the result of each iteration.
Tradable Products
The detquantlib.tradable_products package contains helpers for tradable products and their
tenors: converting tenors to pandas offset aliases or numbers of hours, counting delivery periods,
and converting between delivery start dates and maturities.
Utils
The detquantlib.utils package contains general utilities: setting up a logger with colored
console output and a log file (setup_logger()), adding timestamped log messages (add_log()),
converting lists to strings (list_to_str()), and dividing arrays element-wise while replacing NaN
and infinite results with user-defined values (divide_with_nan()).
Exposed Symbols
Some of the package's symbols (i.e. functions, classes, modules, etc.) are exposed via
__init__.py files. They can therefore be imported with a more concise notation that does not
require specifying their full path.
This section references all exposed symbols, and for each one of them, provides an exhaustive list of the ways they can be imported.
Note: The list below is auto-generated and can be updated with the following command in the terminal:
poetry run python docs.py
To only check whether the list is up to date, without updating it, add the --check option. The
CI check runs it with that option (see CI Check: Code Formatting),
and inv lint runs it as well.
List of Exposed Symbols
Classes:
AzureClient:from detquantlib.data import AzureClientfrom detquantlib.data.databases.azure_client import AzureClient
AzureConfig:from detquantlib.data import AzureConfigfrom detquantlib.data.databases.azure_client import AzureConfig
BatteryState:from detquantlib.assets import BatteryStatefrom detquantlib.assets.battery import BatteryState
DbCacheMissError:from detquantlib.snapshot_testing import DbCacheMissErrorfrom detquantlib.snapshot_testing.errors import DbCacheMissError
DetDatabase:from detquantlib.data import DetDatabasefrom detquantlib.data.databases.detdatabase import DetDatabase
Entsoe:from detquantlib.data import Entsoefrom detquantlib.data.entsoe.entsoe import Entsoe
NetworkAccessBlockedError:from detquantlib.snapshot_testing import NetworkAccessBlockedErrorfrom detquantlib.snapshot_testing.errors import NetworkAccessBlockedError
OutputItem:from detquantlib.outputs import OutputItemfrom detquantlib.outputs.outputs_interface import OutputItem
OutputSet:from detquantlib.outputs import OutputSetfrom detquantlib.outputs.outputs_interface import OutputSet
PathDefinitions:from detquantlib.outputs import PathDefinitionsfrom detquantlib.outputs.outputs_interface import PathDefinitions
Sftp:from detquantlib.data import Sftpfrom detquantlib.data.sftp.sftp import Sftp
SnapshotCase:from detquantlib.snapshot_testing import SnapshotCasefrom detquantlib.snapshot_testing.cases import SnapshotCase
SnapshotConfig:from detquantlib.snapshot_testing import SnapshotConfigfrom detquantlib.snapshot_testing.config import SnapshotConfig
SnowflakeClient:from detquantlib.data import SnowflakeClientfrom detquantlib.data.databases.snowflake_client import SnowflakeClient
SnowflakeConfig:from detquantlib.data import SnowflakeConfigfrom detquantlib.data.databases.snowflake_client import SnowflakeConfig
TennetAPIError:from detquantlib.data import TennetAPIErrorfrom detquantlib.data.tennet.tennet_client import TennetAPIError
TennetClient:from detquantlib.data import TennetClientfrom detquantlib.data.tennet.tennet_client import TennetClient
WindTurbine:from detquantlib.assets import WindTurbinefrom detquantlib.assets.wind_turbine import WindTurbine
WindTurbineModels:from detquantlib.assets import WindTurbineModelsfrom detquantlib.assets.wind_turbine import WindTurbineModels
Functions:
add_log:from detquantlib.utils import add_logfrom detquantlib.utils.utils import add_log
calc_months_diff:from detquantlib.dates import calc_months_difffrom detquantlib.dates.dates import calc_months_diff
convert_delivery_start_date_to_maturity:from detquantlib.tradable_products import convert_delivery_start_date_to_maturityfrom detquantlib.tradable_products.tradable_products import convert_delivery_start_date_to_maturity
convert_energy:from detquantlib.converters import convert_energyfrom detquantlib.converters.energy import convert_energy
convert_maturity_to_delivery_start_date:from detquantlib.tradable_products import convert_maturity_to_delivery_start_datefrom detquantlib.tradable_products.tradable_products import convert_maturity_to_delivery_start_date
convert_power:from detquantlib.converters import convert_powerfrom detquantlib.converters.energy import convert_power
convert_price:from detquantlib.converters import convert_pricefrom detquantlib.converters.price import convert_price
count_delivery_periods:from detquantlib.tradable_products import count_delivery_periodsfrom detquantlib.tradable_products.tradable_products import count_delivery_periods
count_nr_hours:from detquantlib.dates import count_nr_hoursfrom detquantlib.dates.dates import count_nr_hours
create_case:from detquantlib.snapshot_testing import create_casefrom detquantlib.snapshot_testing.cases import create_case
datetime_to_month_code:from detquantlib.dates import datetime_to_month_codefrom detquantlib.dates.dates import datetime_to_month_code
discover_cases:from detquantlib.snapshot_testing import discover_casesfrom detquantlib.snapshot_testing.cases import discover_cases
divide_with_nan:from detquantlib.utils import divide_with_nanfrom detquantlib.utils.utils import divide_with_nan
energy_to_power:from detquantlib.converters import energy_to_powerfrom detquantlib.converters.energy import energy_to_power
flag_cents:from detquantlib.converters import flag_centsfrom detquantlib.converters.price import flag_cents
forecast_knife_strategy:from detquantlib.forecasting import forecast_knife_strategyfrom detquantlib.forecasting.forecasting import forecast_knife_strategy
list_to_sql_columns:from detquantlib.data import list_to_sql_columnsfrom detquantlib.data.databases.helpers import list_to_sql_columns
list_to_sql_set:from detquantlib.data import list_to_sql_setfrom detquantlib.data.databases.helpers import list_to_sql_set
list_to_str:from detquantlib.utils import list_to_strfrom detquantlib.utils.utils import list_to_str
power_to_energy:from detquantlib.converters import power_to_energyfrom detquantlib.converters.energy import power_to_energy
print_cases:from detquantlib.snapshot_testing import print_casesfrom detquantlib.snapshot_testing.cases import print_cases
setup_logger:from detquantlib.utils import setup_loggerfrom detquantlib.utils.logging import setup_logger
tenor_to_nr_hours:from detquantlib.tradable_products import tenor_to_nr_hoursfrom detquantlib.tradable_products.tradable_products import tenor_to_nr_hours
tenor_to_pd_offset_alias:from detquantlib.tradable_products import tenor_to_pd_offset_aliasfrom detquantlib.tradable_products.tradable_products import tenor_to_pd_offset_alias
ts_ceil:from detquantlib.dates import ts_ceilfrom detquantlib.dates.dates import ts_ceil
ts_floor:from detquantlib.dates import ts_floorfrom detquantlib.dates.dates import ts_floor
Constants:
DEFAULT_DB_CONNECTION_TARGETS:from detquantlib.snapshot_testing import DEFAULT_DB_CONNECTION_TARGETSfrom detquantlib.snapshot_testing.config import DEFAULT_DB_CONNECTION_TARGETS
DEFAULT_DB_QUERY_TARGETS:from detquantlib.snapshot_testing import DEFAULT_DB_QUERY_TARGETSfrom detquantlib.snapshot_testing.config import DEFAULT_DB_QUERY_TARGETS
DEFAULT_DB_WRITE_TARGETS:from detquantlib.snapshot_testing import DEFAULT_DB_WRITE_TARGETSfrom detquantlib.snapshot_testing.config import DEFAULT_DB_WRITE_TARGETS
DEFAULT_IGNORE_PATTERNS:from detquantlib.snapshot_testing import DEFAULT_IGNORE_PATTERNSfrom detquantlib.snapshot_testing.config import DEFAULT_IGNORE_PATTERNS
Configuration
Project Structure
The project mainly follows the standard Poetry structure:
detquantlib
├── pyproject.toml
├── README.md
├── detquantlib
│ └── __init__.py
└── tests
└── __init__.py
Dependencies
Dependency Manager
Project dependencies are managed by Poetry.
Dependabot
Automated dependency updates are executed with Dependabot.
Invoke Tasks
The project uses the Invoke package to simplify certain tasks such as testing and code formatting.
All tasks are defined in a tasks.py file.
Invoke tasks can be executed directly from the terminal, using the inv (or invoke) command
line tool.
For guidance on the available Invoke tasks, execute the following command in the terminal:
inv --list
Use the -h (or --help) argument for help about a particular Invoke task. For example:
inv lint -h
GitHub Actions
The project's CI/CD pipeline is enforced with GitHub actions workflows.
Workflow: Python CI
The Python CI (i.e. continuous integration) workflow runs linters to check the consistency of the codebase's format, and then unit tests to check the integrity of its content.
The workflow was inspired by the following preconfigured templates:
- Python package: A general workflow template for Python packages.
- Poetry action: A GitHub action for installing and configuring Poetry.
CI Check: Code Formatting
Linters are used to check that the code is properly formatted:
- Ruff to check for errors in the code and in the docstrings descriptions, and to sort the imports section
- Black for the main code
- docs.py to check that the list of exposed symbols in README.md is up to date
- Pymarkdown for the markdown file README.md
- Deptry to check the dependencies in pyproject.toml for unused (main dependencies only), missing, transitive and misplaced dependencies. Deptry only reports issues: they must be fixed manually in pyproject.toml.
The CI check is executed with the following Invoke task:
inv lint -c
If the CI check fails, execute the following command in the terminal:
inv lint
This command fixes the parts of the code that should be reformatted. Adding the -c (or
--check) optional argument instructs the command to only check if parts of the code should be
reformatted, without applying any actual changes.
CI Check: Unit Tests
Code changes are tested with the Pytest package.
The CI check is executed with the following Invoke task:
inv test -c
Workflow: Package Publisher
The package publisher workflow checks the validity of package version updates, creates version tags, and publishes package updates to PyPI.
Checking Version Updates
The workflow enforces version control:
- The version number is specified via the
versionfield in the pyproject.toml file. - The version number needs to be updated with every new master commit. If the version is not updated, the GitHub workflow will fail.
- Version numbers should follow semantic versioning (i.e.
X.Y.Z). That is:Xincrements represent major, non-backward compatible updates.Yincrements represent minor, backward compatible functionality updates.Zincrements represent patch/bugfix, backward compatible updates.
Creating Version Tags
If the new package version is valid, the workflow automatically creates a new tag for every new master commit.
Publishing Package Updates to PyPI
The workflow automatically publishes every new master commit to PyPI.
Release Notes
When deemed necessary (especially in case of major updates), developers can document code changes in dedicated GitHub release notes.
Release notes can be created via https://github.com/Dynamic-Energy-Trading/detquantlib/releases.
In any case, all codes changes should always be properly described/documented in GitHub issues and/or pull requests.
Metadata
Release files for detquantlib 5.11.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| detquantlib-5.11.1.tar.gz | 89.1 kB | Without provenance | Details for detquantlib-5.11.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| detquantlib-5.11.1-py3-none-any.whl | Python 3 | none | any | Details for detquantlib-5.11.1-py3-none-any.whl |
Total release size: 189.9 kB
Release files / detquantlib-5.11.1.tar.gz
| Download URL | detquantlib-5.11.1.tar.gz |
|---|---|
| Size | 89.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d2bdcba5d1594cfcaca19aaaec7d53c2904d585feae6d85f0cfb262d2a9f80e0
|
|
BLAKE2b-256 checksum How to use checksums |
cec87d4ec8721148b04ffbc3195c4b7c4add44e545a2f04e937563aafdaac278
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.5.1 CPython/3.11.18 Linux/7.0.0-1019-azure
|
Release files / detquantlib-5.11.1-py3-none-any.whl
| Download URL | detquantlib-5.11.1-py3-none-any.whl |
|---|---|
| Size | 100.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
869ddf6ccd4e66c887128f1c6054678cbccda00dfef09e22830346bcf0a50f04
|
|
BLAKE2b-256 checksum How to use checksums |
0ad94b3003a94a35ca5d9dff98eef0a2162b3d9f3b720c9e3a5fa0ee9d98201f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.5.1 CPython/3.11.18 Linux/7.0.0-1019-azure
|
Security for detquantlib 5.11.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.