timeseries-table-format 0.1.3

Append-only time-series table format with gap/overlap tracking (Python bindings).

timeseries-table-format (Python)

Python-first workflow for managing local, append-only time-series tables stored as Parquet segments on disk, with SQL querying (DataFusion) that returns pyarrow.Table.

• PyPI: timeseries-table-format
• Import: timeseries_table_format
• Docs: https://mag1cfrog.github.io/timeseries-table-format/

v0 is local-filesystem-only (no S3/object storage backend yet).

Install

pip install timeseries-table-format

Requires: Python 3.10+. pyarrow is installed automatically (dependency: pyarrow>=23.0.0). If pip tries to build from source (Rust errors), see Troubleshooting below.

Verify installation

import timeseries_table_format as ttf

out = ttf.Session().sql("select 1 as x")
print(type(out))  # pyarrow.Table

Return type and interop

Session.sql(...) returns a pyarrow.Table.

• Polars: pip install polars, then polars.from_arrow(out)

Notebook display (Jupyter/IPython)

In IPython/Jupyter (including VS Code notebooks), pyarrow.Table results will display as a bounded HTML preview by default (the return type is still a real pyarrow.Table).

• Defaults: max_rows=20 (head/tail), max_cols=50 (left/right), max_cell_chars=2000
• Opt-out: set TTF_NOTEBOOK_DISPLAY=0 before importing timeseries_table_format, or call timeseries_table_format.disable_notebook_display()
• Configure: call timeseries_table_format.enable_notebook_display(max_rows=..., max_cols=..., max_cell_chars=..., align=...)
• Config file (TOML): set TTF_NOTEBOOK_CONFIG=path/to/ttf.toml before importing timeseries_table_format (or call timeseries_table_format.load_notebook_display_config("path/to/ttf.toml")) (On Python 3.10, install tomli to enable TOML parsing.)
• Alignment: align="right" (default) or align="auto" (strings left, numbers right); auto-enable can be configured with TTF_NOTEBOOK_ALIGN=auto|left|right
• Cells are visually clipped to a bounded column width with an ellipsis indicator; copying a cell copies the underlying value (up to max_cell_chars).

Example ttf.toml:

[notebook_display]
max_rows = 20
max_cols = 50
max_cell_chars = 2000
align = "auto"

Maintainers: releasing the Python package

The PyPI package version is derived from crates/timeseries-table-python/Cargo.toml (via maturin). If you change the pure-Python sources under python/src/ (or python/pyproject.toml / python/README.md), CI will automatically update crates/timeseries-table-python/python-src.stamp on PRs from branches in this repository.

If you need to update it locally (e.g. working on a fork, or before pushing), run:

python3 scripts/update_python_wheel_stamp.py

If your development environment uses the repo venv, you can also run:

python/.venv/bin/python scripts/update_python_wheel_stamp.py

CI enforces the stamp, and it helps the release automation notice python-only changes for version bumps.

Quickstart: create → append → query

import tempfile
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

import timeseries_table_format as ttf

with tempfile.TemporaryDirectory() as d:
    table_root = Path(d) / "my_table"

    tbl = ttf.TimeSeriesTable.create(
        table_root=str(table_root),
        time_column="ts",
        bucket="1h",
        entity_columns=["symbol"],
        timezone=None,
    )

    seg_path = table_root / "incoming" / "prices.parquet"
    seg_path.parent.mkdir(parents=True, exist_ok=True)

    pq.write_table(
        pa.table(
            {
                "ts": pa.array([0, 3_600 * 1_000_000, 7_200 * 1_000_000], type=pa.timestamp("us")),
                "symbol": pa.array(["NVDA", "NVDA", "NVDA"], type=pa.string()),
                "close": pa.array([10.0, 20.0, 30.0], type=pa.float64()),
            }
        ),
        str(seg_path),
    )

    tbl.append_parquet(str(seg_path))

    sess = ttf.Session()
    sess.register_tstable("prices", str(table_root))

    out = sess.sql("select ts, symbol, close from prices order by ts")
    print(out)  # pyarrow.Table

Bucket size (important): bucket=1h does not resample your data. It defines the time grid used for overlap detection and coverage tracking. Example: with bucket=1h, timestamps 10:05 and 10:55 fall into the same bucket (10:00–11:00). See https://mag1cfrog.github.io/timeseries-table-format/concepts/bucketing_and_overlap/

Join multiple tables

# Aligned with python/examples/register_and_join_two_tables.py
import tempfile
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

import timeseries_table_format as ttf

with tempfile.TemporaryDirectory() as d:
    base_dir = Path(d)

    prices_root = base_dir / "prices_tbl"
    prices = ttf.TimeSeriesTable.create(
        table_root=str(prices_root),
        time_column="ts",
        bucket="1h",
        entity_columns=["symbol"],
        timezone=None,
    )
    prices_seg = base_dir / "prices.parquet"
    pq.write_table(
        pa.table(
            {
                "ts": pa.array([0, 3_600 * 1_000_000], type=pa.timestamp("us")),
                "symbol": pa.array(["NVDA", "NVDA"], type=pa.string()),
                "close": pa.array([1.0, 2.0], type=pa.float64()),
            }
        ),
        str(prices_seg),
    )
    prices.append_parquet(str(prices_seg))

    volumes_root = base_dir / "volumes_tbl"
    volumes = ttf.TimeSeriesTable.create(
        table_root=str(volumes_root),
        time_column="ts",
        bucket="1h",
        entity_columns=["symbol"],
        timezone=None,
    )
    volumes_seg = base_dir / "volumes.parquet"
    pq.write_table(
        pa.table(
            {
                "ts": pa.array([0, 3_600 * 1_000_000], type=pa.timestamp("us")),
                "symbol": pa.array(["NVDA", "NVDA"], type=pa.string()),
                "volume": pa.array([10, 20], type=pa.int64()),
            }
        ),
        str(volumes_seg),
    )
    volumes.append_parquet(str(volumes_seg))

    sess = ttf.Session()
    sess.register_tstable("prices", str(prices_root))
    sess.register_tstable("volumes", str(volumes_root))

    out = sess.sql(
        """
        select p.ts as ts, p.symbol as symbol, p.close as close, v.volume as volume
        from prices p
        join volumes v
        on p.ts = v.ts and p.symbol = v.symbol
        order by p.ts
        """
    )
    print(out)  # pyarrow.Table

Parameterized queries

DataFusion infers placeholder types from context when possible (e.g. in WHERE clauses). If you use placeholders in a SELECT projection without type context, you may need an explicit cast.

# Aligned with python/examples/parameterized_queries.py
import timeseries_table_format as ttf

sess = ttf.Session()

out_positional = sess.sql(
    "select cast($1 as bigint) as x, cast($2 as varchar) as y",
    params=[1, "hello"],
)
out_named = sess.sql(
    "select cast($a as bigint) as x, cast($b as varchar) as y",
    params={"a": 2, "b": "world"},
)

print(out_positional)
print(out_named)

Building from source (contributors)

Prereqs:

• Rust toolchain installed
• Python 3.10+ (CI targets 3.10–3.14; examples below use 3.12)
• uv installed

From the repo root:

uv venv -p 3.12 python/.venv
uv pip install -p python/.venv/bin/python -e python --group dev
python/.venv/bin/python -m pytest

Type checking (ty):

uvx ty check --project python

Alternative (uses the python/ dev environment):

cd python
uv run ty check

Alternative: build with maturin directly:

cd python
uv venv -p 3.12 .venv
uv pip install -p .venv/bin/python pyarrow --group dev
uv run -p .venv/bin/python maturin develop
.venv/bin/python -m pytest

Benchmark: SQL conversion (IPC vs C Stream)

Session.sql(...) returns results as a pyarrow.Table.

By default, results are exported via the Arrow C Data Interface (C Stream) when supported, and fall back to an in-memory Arrow IPC stream otherwise. To compare the two paths and estimate the conversion overhead, run:

cd python
uv pip install -p .venv/bin/python numpy
uv run -p .venv/bin/python maturin develop --features test-utils
.venv/bin/python bench/sql_conversion.py --target-ipc-gb 2

Environment variables (useful for debugging and benchmarks):

• TTF_SQL_EXPORT_MODE=auto|ipc|c_stream (default: c_stream)
• TTF_SQL_EXPORT_DEBUG=1 to emit a debug warning when auto falls back from C Stream → IPC
• TTF_SQL_EXPORT_AUTO_RERUN_FALLBACK=1 to re-run the SQL query on C Stream failure in auto mode (avoids cloning batches on the hot path, but may change results for non-deterministic queries)

Optional: benchmark IPC ZSTD compression (requires building with ipc-zstd):

uv run -p .venv/bin/python maturin develop --features test-utils,ipc-zstd
.venv/bin/python bench/sql_conversion.py --target-ipc-gb 2 --ipc-compression zstd

The script can print a human-friendly terminal summary (--summary) and/or write a JSON payload to a file (--json path). It reports separate timings for:

• end-to-end Session.sql(...)
• Rust-side query+IPC encode (_native._testing._bench_sql_ipc)
• Rust-side query+C Stream export (_native._testing._bench_sql_c_stream)
• Python-side decode/import

Large targets can require high peak RAM (IPC bytes + decoded Table + intermediate buffers). Start with --target-ipc-gb 2 and scale up to 3 or 6 on a machine with plenty of memory.

If you hit Disk quota exceeded, pass --tmpdir /path/with/more/space (the bench uses a temporary directory and cleans it up on exit).

Troubleshooting

• pip is building from source / fails with Rust errors: no wheel is available for your platform/Python; install Rust and retry, or use a supported Python/platform combination.
• DataFusionError about an unknown table name: call sess.register_tstable("name", "/path/to/table") first; use sess.tables() to list registrations.
• Append fails with a time column error: the timestamp column must be an Arrow timestamp(...), and the unit should remain consistent across segments (e.g. timestamp("us")).
• SchemaMismatchError on append: the new Parquet segment schema must match the table's adopted schema (column names and types).
• SQL errors / parameter placeholders: try an explicit CAST(...) for placeholders used in SELECT projections.