polars-hist-db
A Python library for building bitemporal data pipelines with Polars, MariaDB, and optional XTDB backend support.
It ingests data from DSV files or NATS JetStream subjects, tracks history through a typed temporal table contract, and exposes everything as strongly-typed Polars DataFrames.
Features
- Typed uploads — push Polars DataFrames into MariaDB or XTDB with automatic type mapping between Polars, SQL, and SQLAlchemy/ADBC.
- Typed queries — read tables back into DataFrames with column types inferred from the configured backend schema. Temporal query hints (
asof,span,all) let you slice history without writing SQL. - YAML-driven pipelines — define scrape specifications that handle column typing, enrichment via custom transform functions, normalization across tables, and foreign-key deduction.
- Deduplication — an audit log tracks what has already been ingested so re-runs skip previously processed files or messages.
- Transactional ingestion — each file or message is processed in its own transaction; failures roll back cleanly without affecting other items.
- Dual input sources — crawl directories for DSV files, or consume messages from NATS JetStream with configurable fetch/subscription strategies.
- Delta upserts — staging tables detect changed rows, handle duplicates (
take_first,take_last,error), and optionally mark disappeared rows as dropped.
Full documentation: jr200.github.io/polars-hist-db
Quick Start
uv sync
make test
Development
Development and CI use Python 3.15, selected by .python-version; uv installs
that interpreter automatically. The library continues to support Python 3.12+.
With the current lockfile, Python 3.15 builds pycrdt, scandir-rs, and the
optional ADBC driver manager from source. Install a Rust toolchain supporting
the 2024 edition and a C/C++ compiler before running
uv sync --group dev --extra xtdb.
make check # ruff + mypy
make docs # render and preview quarto docs
make bump PART=patch
make release
License
This project is licensed under the MIT license.
1 maintainer
Metadata
1 maintainer
Release files for polars-hist-db 0.12.81
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| polars_hist_db-0.12.81.tar.gz | 1.0 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| polars_hist_db-0.12.81-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.2 MB
Release files / polars_hist_db-0.12.81.tar.gz
| Download URL | polars_hist_db-0.12.81.tar.gz |
|---|---|
| Size | 1.0 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
07c46e6385e4f4a42b5a4373e9b36f0c8902f512f3e188901542946baa7c1ca4
|
|
BLAKE2b-256 checksum How to use checksums |
6db37de89e597a60990133cbe5e8cc748d630da600f2925143628ec7c1bec37c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.0 {"installer":{"name":"uv","version":"0.13.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / polars_hist_db-0.12.81-py3-none-any.whl
| Download URL | polars_hist_db-0.12.81-py3-none-any.whl |
|---|---|
| Size | 176.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bdc09f226521b90d3f76db0ef49203cda18f7e783f9bc3d56f9aa0793f18d07d
|
|
BLAKE2b-256 checksum How to use checksums |
38c39a0bd20267b4d107eec5668d993a366120aa56fe5624780de5f601c7f583
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.0 {"installer":{"name":"uv","version":"0.13.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
