Skip to main content

DOI Supported Python Versions test workflow docs workflow

damast: Creation of reproducible data processing pipelines

The main purpose of this library is to faciliate the reusability of data and data processing pipelines. For this, damast introduces a means to associate metadata with data frames and enables consistency checking.

To ensure semantic consistency, transformation steps in a pipeline can be annotated with allowed data ranges for inputs and outputs, as well as units.

class LatLonTransformer(PipelineElement):
    """
    The LatLonTransformer will consume a lat(itude) and a lon(gitude) column and perform
    cyclic normalization. It will add four columns to a dataframe, namely lat_x, lat_y, lon_x, lon_y.
    """
    @damast.core.describe("Lat/Lon cyclic transformation")
    @damast.core.input({
        "lat": {"unit": "deg"},
        "lon": {"unit": "deg"}
    })
    @damast.core.output({
        "lat_x": {"value_range": MinMax(-1.0, 1.0)},
        "lat_y": {"value_range": MinMax(-1.0, 1.0)},
        "lon_x": {"value_range": MinMax(-1.0, 1.0)},
        "lon_y": {"value_range": MinMax(-1.0, 1.0)}
    })
    def transform(self, df: AnnotatedDataFrame) -> AnnotatedDataFrame:
        lat_cyclic_transformer = CycleTransformer(features=["lat"], n=180.0)
        lon_cyclic_transformer = CycleTransformer(features=["lon"], n=360.0)

        _df = lat_cyclic_transformer.fit_transform(df=df)
        _df = lon_cyclic_transformer.fit_transform(df=_df)
        return _df

Pipeline can also be designed with join / merge operations:

class JoinByTimestamp(PipelineElement):
    def __init__(self):
        pass

    @damast.core.describe("Join data for matching timestamp")
    @damast.core.input({
                           "timestamp": {},
                           "lon": {},
                           "lat": {},
                       })
    @damast.core.input({
                            "timestamp": {},
                            "lat": {},
                            "lon": {},
                            "event_type": {}
                        }, label='other'
    )
    @damast.core.output({ "event_type": {}})
    def transform(self, df: AnnotatedDataFrame, other: AnnotatedDataFrame) -> AnnotatedDataFrame:
        other_timestamp = self.get_name('timestamp', datasource='other')
        df_timestamp = self.get_name('timestamp')

        df.lazyframe = df.join(other.lazyframe, left_on=df_timestamp, right_on=other_timestamp)
        return df

For detailed examples, check the documentation at: https://simula.github.io/damast

Installation and Development Setup

Firstly, you will want to create you an isolated development environment for Python, that being conda or venv-based. The following will go through a venv based setup.

Let us assume you operate with a 'workspace' directory for this project:

    cd workspace

Here, you will create a virtual environment. Get an overview over venv (command):

    python -m venv --help

Create your venv and activate it:

    python -m venv damast-venv
    source damast-venv/bin/activate

From PyPi

To install the package from pypi use:

    pip install damast

Note that machine-learning related elements required additional dependencies, and to install keras and all supported backends (jax, torch, tensorflow) you can use:

    pip install damast[ml]

From Source

Clone the repo and install:

    git clone https://github.com/simula/damast
    cd damast
    pip install -e ".[test,dev]"

or alternatively:

    pip install damast[test,dev]

Docker Container

If you prefer to work or start with a docker container you can build it using the provided Dockerfile

    docker build -t damast:latest -f Dockerfile .

To enter the container:

    docker run -it --rm damast:latest /bin/bash

Usage

To get the usage documentation it is easiest to check the published documentation here.

Otherwise, you can also locally generate the latest documentation once you installed the package:

    tox -e build_docs

Then open the documentation with a browser:

    <yourbrowser> _build/html/index.html

GPU Usage

Install the cuda extra to execute pipelines with the polars GPU engine and to train with a GPU build of the keras backends:

pip install "damast[ml,cuda]"

A GPU is then used automatically when available. To restrict damast to the CPU, hide the devices with CUDA_VISIBLE_DEVICES= (or set POLARS_ENGINE_AFFINITY=streaming for the data processing only). Queries the GPU engine cannot run fall back to polars' in-memory CPU engine - for data larger than memory, prefer the CPU streaming engine.

Testing

Install the project and use the predefined default test environment:

tox -e py

Contributing

This project is open to contributions. For details on how to contribute please check the Contribution Guidelines

License

This project is licensed under the BSD-3-Clause License.

Copyright (c) 2023-2026 Simula Research Laboratory, Oslo, Norway

Acknowledgments

This work has been derived from work that is part of the T-SAR project Some derived work is mainly part of the specific data processing for the 'maritime' domain.

The development of this library is part of the EU-project AI4COPSEC which receives funding from the Horizon Europe framework programme under Grant Agreement N. 101190021.

For the development of this library we take advantage of AI coding assistance, currently Claude Sonnet 5 and Claude Opus 5.

Metadata

Release files for damast 0.4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for damast 0.4.1
File Size Uploaded Provenance Actions
damast-0.4.1.tar.gz 214.2 kB With provenance Details for damast-0.4.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for damast 0.4.1
File Interpreter ABI Platform Actions
damast-0.4.1-py3-none-any.whl Python 3 none any Details for damast-0.4.1-py3-none-any.whl

Total release size: 457.9 kB

Release files / damast-0.4.1.tar.gz

Download URL damast-0.4.1.tar.gz
Size 214.2 kB
Tags Source
SHA-256 checksum
How to use checksums
075e751918b3f196218c76e45a7fa9cd5dc36caaf197ee47c1553ad3d7cd6cb8
BLAKE2b-256 checksum
How to use checksums
396c6ca488885c7526ef7f79846d3ad1ca6d73bfe8002f263d69612430aceaed
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / damast-0.4.1-py3-none-any.whl

Download URL damast-0.4.1-py3-none-any.whl
Size 243.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dbaa1252567671f75779a3244b52fc0490939cbc2c9ab6f57d7ea91cf9fe0393
BLAKE2b-256 checksum
How to use checksums
24ed893df4f3f2d871f8c9ad2315541f0781f565f4d149447d89bcd10e6de8ea
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for damast 0.4.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for damast-0.4.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page