timeseries-table-format (Python)

Python-first workflow for managing local, append-only time-series tables stored as Parquet segments on disk, with SQL querying (DataFusion) that returns pyarrow.Table.

• PyPI: timeseries-table-format
• Import: timeseries_table_format
• Docs: https://mag1cfrog.github.io/timeseries-table-format/

v0 is local-filesystem-only (no S3/object storage backend yet).

Install

pip install timeseries-table-format

Requires: Python 3.10+. pyarrow is installed automatically (dependency: pyarrow>=23.0.0). If pip tries to build from source (Rust errors), see Troubleshooting below.

Verify installation

import timeseries_table_format as ttf

out = ttf.Session().sql("select 1 as x")
print(type(out))  # pyarrow.Table

Development: build and test locally

To run the Python test suite against the local editable Rust extension, build the extension into the repo virtualenv first, then run pytest with the same interpreter.

cd crates/timeseries-table-python
uv sync --group dev
uv run maturin develop --features test-utils
uv run pytest -q

Return type and interop

Session.sql(...) returns a pyarrow.Table.

• Polars: pip install polars, then polars.from_arrow(out)

Notebook display (Jupyter/IPython)

In IPython/Jupyter (including VS Code notebooks), pyarrow.Table results will display as a bounded HTML preview by default (the return type is still a real pyarrow.Table).

• Defaults: max_rows=20 (head/tail), max_cols=50 (left/right), max_cell_chars=2000
• Opt-out: set TTF_NOTEBOOK_DISPLAY=0 before importing timeseries_table_format, or call timeseries_table_format.disable_notebook_display()
• Configure: call timeseries_table_format.enable_notebook_display(max_rows=..., max_cols=..., max_cell_chars=..., align=...)
• Config file (TOML): set TTF_NOTEBOOK_CONFIG=path/to/ttf.toml before importing timeseries_table_format (or call timeseries_table_format.load_notebook_display_config("path/to/ttf.toml")) (On Python 3.10, install tomli to enable TOML parsing.)
• Alignment: align="right" (default) or align="auto" (strings left, numbers right); auto-enable can be configured with TTF_NOTEBOOK_ALIGN=auto|left|right
• Cells are visually clipped to a bounded column width with an ellipsis indicator; copying a cell copies the underlying value (up to max_cell_chars).

Example ttf.toml:

[notebook_display]
max_rows = 20
max_cols = 50
max_cell_chars = 2000
align = "auto"

Maintainers: releasing the Python package

The PyPI package version is derived from the local Cargo.toml via maturin.

Published wheels use CPython's stable ABI with a Python 3.10 minimum and carry the cp310-abi3 tag. Release CI builds one wheel for Linux x86_64, macOS arm64, macOS x86_64, and Windows x86_64, then tests the same Linux wheel on CPython 3.10 through 3.14. These wheels target ordinary GIL-enabled CPython. Free-threaded CPython, PyPy, and other Python implementations are not supported by this wheel policy.

Quickstart: create → append → query

import tempfile
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

import timeseries_table_format as ttf

with tempfile.TemporaryDirectory() as d:
    table_root = Path(d) / "my_table"

    tbl = ttf.TimeSeriesTable.create(
        table_root=str(table_root),
        time_column="ts",
        bucket="1h",
        entity_columns=["symbol"],
        timezone=None,
    )

    seg_path = table_root / "incoming" / "prices.parquet"
    seg_path.parent.mkdir(parents=True, exist_ok=True)

    pq.write_table(
        pa.table(
            {
                "ts": pa.array([0, 3_600 * 1_000_000, 7_200 * 1_000_000], type=pa.timestamp("us")),
                "symbol": pa.array(["NVDA", "NVDA", "NVDA"], type=pa.string()),
                "close": pa.array([10.0, 20.0, 30.0], type=pa.float64()),
            }
        ),
        str(seg_path),
    )

    tbl.append_parquet(str(seg_path))

    sess = ttf.Session()
    sess.register_tstable("prices", str(table_root))

    out = sess.sql("select ts, symbol, close from prices order by ts")
    print(out)  # pyarrow.Table

Bucket size (important): bucket=1h does not resample your data. It defines the time grid used for overlap detection and coverage tracking. Example: with bucket=1h, timestamps 10:05 and 10:55 fall into the same bucket (10:00–11:00). See https://mag1cfrog.github.io/timeseries-table-format/concepts/bucketing_and_overlap/

Join multiple tables

# Aligned with examples/register_and_join_two_tables.py
import tempfile
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

import timeseries_table_format as ttf

with tempfile.TemporaryDirectory() as d:
    base_dir = Path(d)

    prices_root = base_dir / "prices_tbl"
    prices = ttf.TimeSeriesTable.create(
        table_root=str(prices_root),
        time_column="ts",
        bucket="1h",
        entity_columns=["symbol"],
        timezone=None,
    )
    prices_seg = base_dir / "prices.parquet"
    pq.write_table(
        pa.table(
            {
                "ts": pa.array([0, 3_600 * 1_000_000], type=pa.timestamp("us")),
                "symbol": pa.array(["NVDA", "NVDA"], type=pa.string()),
                "close": pa.array([1.0, 2.0], type=pa.float64()),
            }
        ),
        str(prices_seg),
    )
    prices.append_parquet(str(prices_seg))

    volumes_root = base_dir / "volumes_tbl"
    volumes = ttf.TimeSeriesTable.create(
        table_root=str(volumes_root),
        time_column="ts",
        bucket="1h",
        entity_columns=["symbol"],
        timezone=None,
    )
    volumes_seg = base_dir / "volumes.parquet"
    pq.write_table(
        pa.table(
            {
                "ts": pa.array([0, 3_600 * 1_000_000], type=pa.timestamp("us")),
                "symbol": pa.array(["NVDA", "NVDA"], type=pa.string()),
                "volume": pa.array([10, 20], type=pa.int64()),
            }
        ),
        str(volumes_seg),
    )
    volumes.append_parquet(str(volumes_seg))

    sess = ttf.Session()
    sess.register_tstable("prices", str(prices_root))
    sess.register_tstable("volumes", str(volumes_root))

    out = sess.sql(
        """
        select p.ts as ts, p.symbol as symbol, p.close as close, v.volume as volume
        from prices p
        join volumes v
        on p.ts = v.ts and p.symbol = v.symbol
        order by p.ts
        """
    )
    print(out)  # pyarrow.Table

Parameterized queries

DataFusion infers placeholder types from context when possible (e.g. in WHERE clauses). If you use placeholders in a SELECT projection without type context, you may need an explicit cast.

# Aligned with examples/parameterized_queries.py
import timeseries_table_format as ttf

sess = ttf.Session()

out_positional = sess.sql(
    "select cast($1 as bigint) as x, cast($2 as varchar) as y",
    params=[1, "hello"],
)
out_named = sess.sql(
    "select cast($a as bigint) as x, cast($b as varchar) as y",
    params={"a": 2, "b": "world"},
)

print(out_positional)
print(out_named)

Building from source (contributors)

Prereqs:

• Rust toolchain installed
• Python 3.10+ (CI targets 3.10–3.14; examples below use 3.12)
• uv installed

From the repo root:

cd crates/timeseries-table-python
uv sync --group dev
uv run maturin develop --features test-utils
uv run pytest -q

Type checking (ty):

uv run ty check python tests

Benchmark: SQL conversion and streaming SQL

Session.sql(...) returns results as a pyarrow.Table.

By default, results are exported via the Arrow C Data Interface (C Stream) when supported, and fall back to an in-memory Arrow IPC stream otherwise. To compare the two paths and estimate the conversion overhead, run:

cd crates/timeseries-table-python
uv pip install -p .venv/bin/python numpy
uv run -p .venv/bin/python maturin develop --features test-utils
.venv/bin/python bench/sql_conversion.py --target-ipc-gb 2

To also benchmark the streaming SQL API (Session.sql_reader(...)), include the streaming mode:

cd crates/timeseries-table-python
uv pip install -p .venv/bin/python numpy
uv run -p .venv/bin/python maturin develop --features test-utils
.venv/bin/python bench/sql_conversion.py --target-ipc-gb 2 --include-streaming --summary

Environment variables (useful for debugging and benchmarks):

• TTF_SQL_EXPORT_MODE=auto|ipc|c_stream (default: c_stream)
• TTF_SQL_EXPORT_DEBUG=1 to emit a debug warning when auto falls back from C Stream → IPC
• TTF_SQL_EXPORT_AUTO_RERUN_FALLBACK=1 to re-run the SQL query on C Stream failure in auto mode (avoids cloning batches on the hot path, but may change results for non-deterministic queries)

Optional: benchmark IPC ZSTD compression (requires building with ipc-zstd):

uv run -p .venv/bin/python maturin develop --features test-utils,ipc-zstd
.venv/bin/python bench/sql_conversion.py --target-ipc-gb 2 --ipc-compression zstd

The script can print a human-friendly terminal summary (--summary) and/or write a JSON payload to a file (--json path). It reports separate timings for:

• end-to-end Session.sql(...)
• Rust-side query+IPC encode (_native._testing._bench_sql_ipc)
• Rust-side query+C Stream export (_native._testing._bench_sql_c_stream)
• Python-side decode/import

With --include-streaming, it also reports:

• time to first batch for Session.sql_reader(...)
• total incremental iteration time and batch counts
• process-as-you-go comparison: Session.sql_reader(...) batch iteration vs Session.sql(...) materialize-then-process
• Session.sql_reader(...).read_all() vs Session.sql(...) parity
• best-effort process peak RSS (ru_maxrss) after each run
• isolated child-process peak RSS for the process-as-you-go comparison (sampled from Linux /proc/<pid>/status)

Sample streaming benchmark results

One local Linux run over a generated dataset of about 10.5M rows:

sql_reader(...).read_all() stayed in the same general performance range as Session.sql(...).

Large targets can require high peak RAM (IPC bytes + decoded Table + intermediate buffers). Start with --target-ipc-gb 2 and scale up to 3 or 6 on a machine with plenty of memory.

If you hit Disk quota exceeded, pass --tmpdir /path/with/more/space (the bench uses a temporary directory and cleans it up on exit).

Troubleshooting

• pip is building from source / fails with Rust errors: no wheel is available for your platform/Python; install Rust and retry, or use a supported Python/platform combination.
• DataFusionError about an unknown table name: call sess.register_tstable("name", "/path/to/table") first; use sess.tables() to list registrations.
• Append fails with a time column error: the timestamp column must be an Arrow timestamp(...), and the unit should remain consistent across segments (e.g. timestamp("us")).
• SchemaMismatchError on append: the new Parquet segment schema must match the table's adopted schema (column names and types).
• SQL errors / parameter placeholders: try an explicit CAST(...) for placeholders used in SELECT projections.