Welcome to hdhelpers

What is hdhelpers?

hdhelpers is a package designed for and included in the standard installation of the hetida designer.

Currently, the package provides functions to retrieve metadata from Pandas objects (stored in the attr of the Pandas object). Pandas objects are the standard for processing time series data in the hetida designer. Additionally, a function is provided to handle time zone information. In the future, the package is planned to be expanded, for example by providing functions to facilitate visualization.

The documentation of the package is a GitHub page, on which the functions of the package are described and some tips for getting started with using the package are given.

Getting Started with hdhelpers

Since version >0.13.10 the hetida designer runtime comes with an installed version of hdhelpers. To start the hetida designer we recommend following the hetida designer setup guide.

The plotting helpers (hdhelpers.plotting and their names in hdhelpers) need plotly, which is the optional extra plotting: pip install "hdhelpers[plotting]". Everything else, e.g. the metadata models, works without plotly.

An example is given in GitHub page of how to use functionalities of hdhelpers inside a hetida designer component. Furthermore the base component Single Time Series Plot uses hdhelpers to demonstrate the usage (since version >0.13.10).

Developing for hdhelpers

For dependency management and venv setup, building and publishing, uv is used.

Setting up a Development Environment

Python environment

1. Create a virtual environment with uv venv. This will create a hidden .venv directory.
2. Activate the virtual environment via source .venv/bin/activate
3. Run uv sync --all-extras to install all dependencies given in pyproject.toml, including the optional ones of the plotting helpers.
4. In case you need to add a new dependency, do so via uv add <new_dependency>. That way, uv finds versions of all dependencies that are compatible with each other.
5. In case you need a new requirement for development purposes please use uv add --dev <new_dependency>

Note: To install hdhelpers in editable mode in your venv please run uv pip install -e .

hetida designer with hdhelpers

To test designer images with current hdhelpers version please save a copy of docker-compose.yml from the hetida designer repository.

Then write a Dockerfile to install the current hdhelpers version in the required designer backend.

FROM hetida/designer-backend:<<hd_version>>

USER root

RUN pip install ".[plotting]"

USER hd_app

CMD ["bash", "/app/start.sh"]

Then modify the docker-compose.yml to work with the backend version defined by the modified Dockerfile.

...
  hetida-designer-backend:
    build:
      context: .
      dockerfile: Dockerfile
...

use docker-compose.yaml, e.g. via docker compose -f 'docker-compose.yaml' up -d --build. This compose setup loads the current hetida designer images and installs the hdhelpers package in the runtime. Thus, you can use functions of hdhelpers writing component code.

Code Quality

Once you are done writing your code, including unit tests, use ./run check to see if your code quality is sufficient.

Metadata schema snapshot

tests/metadata/test_conventions_schema.py compares the generated metadata JSON schemas with tests/data/convention_schemas.json. Class docstrings become schema descriptions, so documentation changes can also require a snapshot update.

After an intentional schema change, regenerate the snapshot in Bash/WSL:

UPDATE_SCHEMA_SNAPSHOT=1 uv run pytest tests/metadata/test_conventions_schema.py -q

Or in PowerShell:

$env:UPDATE_SCHEMA_SNAPSHOT = "1"
uv run pytest tests/metadata/test_conventions_schema.py -q
Remove-Item Env:UPDATE_SCHEMA_SNAPSHOT

Review the diff for unintended changes and rerun the test without the update flag:

git diff -- tests/data/convention_schemas.json
uv run pytest tests/metadata/test_conventions_schema.py -q

Commit the updated snapshot together with the corresponding model or docstring changes.

Documentation

For documentation we use the tool sphinx. Please apply ./run build_docs to create the current state of documentation. It will be stored in docs. You can open the documentation by opening docs/index.html, e.g. with your browser.

Build, Release and Publish

The first step for publishing a new package version is creating and merging a pull request from develop to main. In detail the following steps should be executed beforehand:

1. Please execute ./run build_package <version_nr> where version number should follow semantic versioning. This will:

• Runs uv sync --frozen to upgrade dependencies.
• Update version in pyproject.toml
• Update __version__ in __init__.py
• Builds wheels of hdhelpers in ./dist

2. Ensure that listed classifiers in pyproject.tomlare up to date. If not

• Update pyproject.toml accordingly
• Update ./run test-py-versions accordingly for local testing using uv
• Update check_pull_request.yml accordingly for automated pipeline execution of checks
• Update badge in this file for Python versions above

3. Update CHANGELOG.md manually following Keep a Changelog

When the PR is accepted, the package can be published in a second step. To publish the build from the dist subdirectory to PyPI,

1. tag your main branch with the specified package version

2. use uv publish --index testpypi --token <API-token>. You need a Test-PyPI account with a token and you need maintainer/owner access to the hdhelpers Test-PyPI project.

3. verifiy that package can be installed by using:

uv run --with 'hdhelpers==0.0.9' --refresh-package hdhelpers --default-index https://test.pypi.org/simple/ --index https://test.pypi.org/simple/ --no-project -- python -c "import hdhelpers; print(hdhelpers.__version__)"

4. use uv publish dist/* to upload the package on hdhelpers PyPI project.

5. After publishing please communicate to the hetida designer team so upgrade there dependencies. The hetida designer docker compose setup installs hdhelpers from PyPI as it does with any dependency listed in runtime/requirements.in.

Notes

• Please ensure that dependencies specified for hdhelpers do work in current designer versions.
• If you want to upgrade dependencies please run uv lock --upgrade
• to install hdhelpers from testpypi you can use: uv pip install --extra-index-url https://test.pypi.org/simple/ --index-url https://pypi.org/simple --refresh --index-strategy unsafe-best-match hdhelpers