Skip to main content

DET Quant Library

Table of Contents

Overview

The DET Quant Library is an internal library containing functions and classes that can be used across Quant models.

Package Contents

The package is split into subpackages, one per functional area. The sections below briefly describe each of them. See List of Exposed Symbols for the symbols that can be imported directly from each subpackage.

Assets

The detquantlib.assets package models physical energy assets:

  • BatteryState stores the characteristics and current state of a battery (capacity, power, state of charge, depth of discharge, roundtrip efficiency). Live battery states can be loaded from Redis with BatteryState.build_from_redis(), for the Eniris and Smartbox energy management systems.
  • WindTurbine stores wind turbine information (rated power, cut-in and cut-out wind speeds, number of turbines, power curve) and converts wind speeds to power or energy. WindTurbineModels contains the specifications of known turbine models, which can be loaded with WindTurbine.build_from_model().

Converters

The detquantlib.converters package converts values between units:

  • convert_power() and convert_energy() convert between power units and between energy units.
  • power_to_energy() and energy_to_power() convert between power and energy, for a given tenor.
  • convert_price() converts a price from one currency and volume unit to another, and flag_cents() detects whether a currency string is expressed in cents.

Data

The detquantlib.data package provides clients for the data sources used by Quant models:

  • SnowflakeClient and AzureClient are small, safe clients for Snowflake and Azure SQL databases, configured with SnowflakeConfig and AzureConfig (which can be read from environment variables). They can query, load, list and write tables, and exchange data as pandas DataFrames.
  • DetDatabase is a client for the DET database, with dedicated methods to load and process data such as ENTSOE day-ahead and imbalance prices, futures settlement prices, commodities, instruments, account positions, customer volume forecasts and clients. It is deprecated and will be removed in a future version: use AzureClient instead.
  • list_to_sql_set() and list_to_sql_columns() format Python lists for use in SQL queries.
  • Sftp imports and exports files from and to an SFTP server.
  • Entsoe imports and processes ENTSOE data, such as day-ahead spot prices from the ENTSOE SFTP server.
  • TennetClient is a wrapper around the TenneT publications API, to fetch settlement prices and balance delta data.

Dates

The detquantlib.dates package contains date and time helpers: rounding timestamps up or down to a given frequency (ts_ceil(), ts_floor()), counting the number of hours between two dates (count_nr_hours()), computing the number of months between two dates (calc_months_diff()), and converting dates to month codes (datetime_to_month_code()).

Figures

The detquantlib.figures.plotly_figures module contains helpers for Plotly figures: applying a standard layout, saving figures to JSON or HTML files, and displaying figures saved as JSON.

Forecasting

The detquantlib.forecasting package contains forecasting strategies. forecast_knife_strategy() generates a random walk forecast, where each value is set to the most recent observed value with the same time of day and the same day type (weekday, Saturday or Sunday).

Outputs

The detquantlib.outputs package manages and exports model outputs. OutputItem holds a single output and exports it to a CSV, JSON, HTML or NPZ file. OutputSet groups several OutputItem objects, and PathDefinitions defines the folders the outputs are exported to (by default an Outputs folder in the working directory, or the OUTPUT_DIR environment variable if set).

Snapshot Testing

The detquantlib.snapshot_testing package provides snapshot (regression) tests for Quant models: each test runs a model end to end on a fixed set of inputs, and compares its outputs with a reviewed reference copy. Database calls are recorded once, locally, and replayed from disk in CI, which has no database access. The SnowflakeClient and AzureClient classes are supported out of the box.

See the snapshot testing guide for how the framework works and how to set it up in a model repository.

Stats

The detquantlib.stats.data_analysis module contains statistical analysis tools. filter_outliers() iteratively removes outliers from a time series, based on a rolling mean and standard deviation, and create_outliers_iteration_plot() plots the result of each iteration.

Tradable Products

The detquantlib.tradable_products package contains helpers for tradable products and their tenors: converting tenors to pandas offset aliases or numbers of hours, counting delivery periods, and converting between delivery start dates and maturities.

Utils

The detquantlib.utils package contains general utilities: setting up a logger with colored console output and a log file (setup_logger()), adding timestamped log messages (add_log()), converting lists to strings (list_to_str()), and dividing arrays element-wise while replacing NaN and infinite results with user-defined values (divide_with_nan()).

Exposed Symbols

Some of the package's symbols (i.e. functions, classes, modules, etc.) are exposed via __init__.py files. They can therefore be imported with a more concise notation that does not require specifying their full path.

This section references all exposed symbols, and for each one of them, provides an exhaustive list of the ways they can be imported.

Note: The list below is auto-generated and can be updated with the following command in the terminal:

inv run-lint-docs

To only check whether the list is up to date, without updating it, add the --check option. The CI check runs it with that option (see Linters (Code Formatting)), and inv lint runs it as well.

List of Exposed Symbols

Classes:

  • ApiCacheMissError:
    • from detquantlib.snapshot_testing import ApiCacheMissError
    • from detquantlib.snapshot_testing.errors import ApiCacheMissError
  • AzureClient:
    • from detquantlib.data import AzureClient
    • from detquantlib.data.databases.azure_client import AzureClient
  • AzureConfig:
    • from detquantlib.data import AzureConfig
    • from detquantlib.data.databases.azure_client import AzureConfig
  • BatteryState:
    • from detquantlib.assets import BatteryState
    • from detquantlib.assets.battery import BatteryState
  • DbCacheMissError:
    • from detquantlib.snapshot_testing import DbCacheMissError
    • from detquantlib.snapshot_testing.errors import DbCacheMissError
  • DetDatabase:
    • from detquantlib.data import DetDatabase
    • from detquantlib.data.databases.detdatabase import DetDatabase
  • Entsoe:
    • from detquantlib.data import Entsoe
    • from detquantlib.data.entsoe.entsoe import Entsoe
  • NetworkAccessBlockedError:
    • from detquantlib.snapshot_testing import NetworkAccessBlockedError
    • from detquantlib.snapshot_testing.errors import NetworkAccessBlockedError
  • OutputItem:
    • from detquantlib.outputs import OutputItem
    • from detquantlib.outputs.outputs_interface import OutputItem
  • OutputSet:
    • from detquantlib.outputs import OutputSet
    • from detquantlib.outputs.outputs_interface import OutputSet
  • PathDefinitions:
    • from detquantlib.outputs import PathDefinitions
    • from detquantlib.outputs.outputs_interface import PathDefinitions
  • SecretLeakError:
    • from detquantlib.snapshot_testing import SecretLeakError
    • from detquantlib.snapshot_testing.errors import SecretLeakError
  • Sftp:
    • from detquantlib.data import Sftp
    • from detquantlib.data.sftp.sftp import Sftp
  • SnapshotCase:
    • from detquantlib.snapshot_testing import SnapshotCase
    • from detquantlib.snapshot_testing.cases import SnapshotCase
  • SnapshotConfig:
    • from detquantlib.snapshot_testing import SnapshotConfig
    • from detquantlib.snapshot_testing.config import SnapshotConfig
  • SnowflakeClient:
    • from detquantlib.data import SnowflakeClient
    • from detquantlib.data.databases.snowflake_client import SnowflakeClient
  • SnowflakeConfig:
    • from detquantlib.data import SnowflakeConfig
    • from detquantlib.data.databases.snowflake_client import SnowflakeConfig
  • TennetAPIError:
    • from detquantlib.data import TennetAPIError
    • from detquantlib.data.tennet.tennet_client import TennetAPIError
  • TennetClient:
    • from detquantlib.data import TennetClient
    • from detquantlib.data.tennet.tennet_client import TennetClient
  • WindTurbine:
    • from detquantlib.assets import WindTurbine
    • from detquantlib.assets.wind_turbine import WindTurbine
  • WindTurbineModels:
    • from detquantlib.assets import WindTurbineModels
    • from detquantlib.assets.wind_turbine import WindTurbineModels

Functions:

  • add_log:
    • from detquantlib.utils import add_log
    • from detquantlib.utils.utils import add_log
  • calc_months_diff:
    • from detquantlib.dates import calc_months_diff
    • from detquantlib.dates.dates import calc_months_diff
  • convert_delivery_start_date_to_maturity:
    • from detquantlib.tradable_products import convert_delivery_start_date_to_maturity
    • from detquantlib.tradable_products.tradable_products import convert_delivery_start_date_to_maturity
  • convert_energy:
    • from detquantlib.converters import convert_energy
    • from detquantlib.converters.energy import convert_energy
  • convert_maturity_to_delivery_start_date:
    • from detquantlib.tradable_products import convert_maturity_to_delivery_start_date
    • from detquantlib.tradable_products.tradable_products import convert_maturity_to_delivery_start_date
  • convert_power:
    • from detquantlib.converters import convert_power
    • from detquantlib.converters.energy import convert_power
  • convert_price:
    • from detquantlib.converters import convert_price
    • from detquantlib.converters.price import convert_price
  • count_delivery_periods:
    • from detquantlib.tradable_products import count_delivery_periods
    • from detquantlib.tradable_products.tradable_products import count_delivery_periods
  • count_nr_hours:
    • from detquantlib.dates import count_nr_hours
    • from detquantlib.dates.dates import count_nr_hours
  • create_case:
    • from detquantlib.snapshot_testing import create_case
    • from detquantlib.snapshot_testing.cases import create_case
  • datetime_to_month_code:
    • from detquantlib.dates import datetime_to_month_code
    • from detquantlib.dates.dates import datetime_to_month_code
  • discover_cases:
    • from detquantlib.snapshot_testing import discover_cases
    • from detquantlib.snapshot_testing.cases import discover_cases
  • divide_with_nan:
    • from detquantlib.utils import divide_with_nan
    • from detquantlib.utils.utils import divide_with_nan
  • energy_to_power:
    • from detquantlib.converters import energy_to_power
    • from detquantlib.converters.energy import energy_to_power
  • flag_cents:
    • from detquantlib.converters import flag_cents
    • from detquantlib.converters.price import flag_cents
  • forecast_knife_strategy:
    • from detquantlib.forecasting import forecast_knife_strategy
    • from detquantlib.forecasting.forecasting import forecast_knife_strategy
  • list_to_sql_columns:
    • from detquantlib.data import list_to_sql_columns
    • from detquantlib.data.databases.helpers import list_to_sql_columns
  • list_to_sql_set:
    • from detquantlib.data import list_to_sql_set
    • from detquantlib.data.databases.helpers import list_to_sql_set
  • list_to_str:
    • from detquantlib.utils import list_to_str
    • from detquantlib.utils.utils import list_to_str
  • power_to_energy:
    • from detquantlib.converters import power_to_energy
    • from detquantlib.converters.energy import power_to_energy
  • print_cases:
    • from detquantlib.snapshot_testing import print_cases
    • from detquantlib.snapshot_testing.cases import print_cases
  • rows_written:
    • from detquantlib.snapshot_testing import rows_written
    • from detquantlib.snapshot_testing.db_recording import rows_written
  • setup_logger:
    • from detquantlib.utils import setup_logger
    • from detquantlib.utils.logging import setup_logger
  • tenor_to_nr_hours:
    • from detquantlib.tradable_products import tenor_to_nr_hours
    • from detquantlib.tradable_products.tradable_products import tenor_to_nr_hours
  • tenor_to_pd_offset_alias:
    • from detquantlib.tradable_products import tenor_to_pd_offset_alias
    • from detquantlib.tradable_products.tradable_products import tenor_to_pd_offset_alias
  • ts_ceil:
    • from detquantlib.dates import ts_ceil
    • from detquantlib.dates.dates import ts_ceil
  • ts_floor:
    • from detquantlib.dates import ts_floor
    • from detquantlib.dates.dates import ts_floor

Constants:

  • DEFAULT_DB_CONNECTION_TARGETS:
    • from detquantlib.snapshot_testing import DEFAULT_DB_CONNECTION_TARGETS
    • from detquantlib.snapshot_testing.config import DEFAULT_DB_CONNECTION_TARGETS
  • DEFAULT_DB_QUERY_TARGETS:
    • from detquantlib.snapshot_testing import DEFAULT_DB_QUERY_TARGETS
    • from detquantlib.snapshot_testing.config import DEFAULT_DB_QUERY_TARGETS
  • DEFAULT_DB_WRITE_TARGETS:
    • from detquantlib.snapshot_testing import DEFAULT_DB_WRITE_TARGETS
    • from detquantlib.snapshot_testing.config import DEFAULT_DB_WRITE_TARGETS
  • DEFAULT_IGNORE_PATTERNS:
    • from detquantlib.snapshot_testing import DEFAULT_IGNORE_PATTERNS
    • from detquantlib.snapshot_testing.config import DEFAULT_IGNORE_PATTERNS
  • DEFAULT_SECRET_ENV_VARS:
    • from detquantlib.snapshot_testing import DEFAULT_SECRET_ENV_VARS
    • from detquantlib.snapshot_testing.config import DEFAULT_SECRET_ENV_VARS

Configuration

Project Structure

The project mainly follows the standard Poetry structure:

detquantlib
├── pyproject.toml
├── README.md
├── detquantlib
│   └── __init__.py
└── tests
    └── __init__.py

Dependencies

Dependency Manager

Project dependencies are managed by Poetry.

Dependabot

Automated dependency updates are executed with Dependabot.

Invoke Tasks

The project uses the Invoke package to simplify certain tasks such as testing and code formatting.

All tasks are defined in a tasks.py file.

Invoke tasks can be executed directly from the terminal, using the inv (or invoke) command line tool.

For guidance on the available Invoke tasks, execute the following command in the terminal:

inv --list

Use the -h (or --help) argument for help about a particular Invoke task. For example:

inv lint -h

GitHub Actions

The project's CI/CD pipeline is enforced with GitHub actions workflows.

Workflow: Python CI/CD

The Python CI/CD (i.e. continuous integration and continuous delivery) workflow checks code changes, and releases new package versions. It consists of three jobs:

  • lint_and_test: Runs linters to check the consistency of the codebase's format, and then unit tests to check the integrity of its content.
  • validate_version: Checks the validity of the package version update.
  • release: Publishes the new package version to PyPI, and creates its version tag.

The lint_and_test and validate_version jobs run in parallel, on every pull request and on every push to master. The release job only runs on pushes to master, after both other jobs succeed.

The workflow was inspired by the following preconfigured templates:

  • Python package: A general workflow template for Python packages.
  • Poetry action: A GitHub action for installing and configuring Poetry.
Linters (Code Formatting)

Linters are used to check that the code is properly formatted:

  • Ruff to check for errors in the code and in the docstrings descriptions, and to sort the imports section
  • Black for the main code
  • docs.py to check that the list of exposed symbols in README.md is up to date
  • Pymarkdown for the markdown file README.md
  • Deptry to check the dependencies in pyproject.toml for unused (main dependencies only), missing, transitive and misplaced dependencies. Deptry only reports issues: they must be fixed manually in pyproject.toml.

The CI check is executed with the following Invoke task:

inv lint -c

If the CI check fails, execute the following command in the terminal:

inv lint

This command fixes the parts of the code that should be reformatted. Adding the -c (or --check) optional argument instructs the command to only check if parts of the code should be reformatted, without applying any actual changes.

Unit Testing

Code changes are tested with the Pytest package.

The CI check is executed with the following Invoke task:

inv test -c
Checking Version Updates

The workflow enforces version control:

  • The version number is specified via the version field in the pyproject.toml file.
  • The version number needs to be updated with every new master commit. If the version is not updated, the GitHub workflow will fail.
  • Version numbers should follow semantic versioning (i.e. X.Y.Z). That is:
    • X increments represent major, non-backward compatible updates.
    • Y increments represent minor, backward compatible functionality updates.
    • Z increments represent patch/bugfix, backward compatible updates.
Publishing Package Updates to PyPI

If the linters and tests pass and the new package version is valid, the workflow automatically publishes every new master commit to PyPI.

Creating Version Tags

After publishing, the workflow automatically creates a version tag (e.g. v1.2.3) for every new master commit. The tag is created last, so that a failed build or publish can be fixed by re-running the workflow.

Release Notes

When deemed necessary (especially in case of major updates), developers can document code changes in dedicated GitHub release notes.

Release notes can be created via https://github.com/Dynamic-Energy-Trading/detquantlib/releases.

In any case, all codes changes should always be properly described/documented in GitHub issues and/or pull requests.

Metadata

Release files for detquantlib 5.13.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for detquantlib 5.13.2
File Size Uploaded Provenance Actions
detquantlib-5.13.2.tar.gz 104.3 kB Without provenance Details for detquantlib-5.13.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for detquantlib 5.13.2
File Interpreter ABI Platform Actions
detquantlib-5.13.2-py3-none-any.whl Python 3 none any Details for detquantlib-5.13.2-py3-none-any.whl

Total release size: 221.4 kB

Release files / detquantlib-5.13.2.tar.gz

Download URL detquantlib-5.13.2.tar.gz
Size 104.3 kB
Tags Source
SHA-256 checksum
How to use checksums
1665868144ebb5ce1d93275e4c665e2e905a117147424b6fe85ae4bfd119a8dd
BLAKE2b-256 checksum
How to use checksums
136ac5fc192b4271e523ff77d651ec8f674f2c0f3241c07d8882eb0583ee5a0e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.11.18 Linux/7.0.0-1019-azure

Release files / detquantlib-5.13.2-py3-none-any.whl

Download URL detquantlib-5.13.2-py3-none-any.whl
Size 117.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
67774a216a2780842bc437b2da5e3b997e351b643e275383818ce8ee5d651c77
BLAKE2b-256 checksum
How to use checksums
27cdfd0f6909fdebb04bc62af660aef5ecdfce1d9e4a543289abeca021020bf0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.11.18 Linux/7.0.0-1019-azure

Security for detquantlib 5.13.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

5.13.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page