timeseries-table-format (Python)

Build local time-series tables with immutable Parquet segments. The Python API tracks coverage, rejects overlapping appends, updates selected row values, and queries tables with DataFusion SQL. Query results are returned as pyarrow.Table objects.

• Documentation
• PyPI
• Repository
• Changelog

Early MVP: APIs and on-disk layouts may change before v1.0.

What it provides

• Timestamp, Int64, or UInt64 ordered indexes
• Per-entity index-interval uniqueness within and across appends
• Local table roots with managed metadata and table-relative Parquet segment paths
• Dry-run and apply vacuum modes for expired unreferenced table-managed files
• DataFusion SQL over managed tables and standalone Parquet data
• Materialized and streaming Arrow result APIs
• Explicit nullable-column addition through table.add_columns(pyarrow.Schema)
• Atomic keyed value updates through table.update_rows(source, columns=..., expected_version=...)

The current release supports local filesystems only. It does not yet support object storage, small-file compaction, column dropping/renaming, automatic schema merging, row insertion/deletion through updates, merges, or time-travel queries.

add_columns adds new nullable top-level fields after the first successful append establishes the schema. Historical rows read as null without rewriting data; later appends may omit nullable payload fields while retaining all keys and matching the types/nullability of provided fields. Replace SQL registrations with Session.register_tstable after adding columns. See the guide and runnable example.

update_rows backfills or overwrites selected existing fields using complete entity and raw index keys. Capture the source version before reading and computing assignments; any intervening commit conflicts. The operation consumes Arrow batches incrementally and rewrites affected segments in one transaction. Explicit null clears a nullable destination, and unselected fields remain unchanged. Newly planned SQL queries see value changes without re-registration. See the update guide and runnable example for error handling, report semantics, and scratch/rewrite costs.

Install

pip install timeseries-table-format

The package requires Python 3.10 or later and installs pyarrow automatically. See Installation for supported platforms, Python implementations, and source-build guidance.

Verify the installation

import timeseries_table_format as ttf

result = ttf.Session().sql("SELECT 1 AS value")
assert result.column("value").to_pylist() == [1]

Create, append, and query a table

Assuming prices.parquet contains a Timestamp column named ts and a string column named symbol:

import pyarrow as pa
import pyarrow.parquet as pq
import timeseries_table_format as ttf

table = ttf.TimeSeriesTable.create(
    table_root="prices",
    index_column="ts",
    index_type="timestamp",
    index_granularity="1h",
    entity_columns=["symbol"],
)
parquet_file = pq.ParquetFile("prices.parquet")
report = table.append(
    pa.RecordBatchReader.from_batches(
        parquet_file.schema_arrow,
        parquet_file.iter_batches(),
    )
)
version = report.committed_version

session = ttf.Session()
session.register_tstable("prices", "prices")
result = session.sql("SELECT * FROM prices ORDER BY symbol, ts")

For a complete runnable example, follow Create, append, and query your first table.

Documentation

• Decide if the project fits your workload
• Append files incrementally
• Register and join tables
• Use SQL parameters
• Stream query results
• Configure native logging
• Configure notebook display
• Read the API reference
• Troubleshoot common problems

Contributing

See Develop the Python package for local setup, tests, quality checks, documentation builds, benchmarks, and release guidance.

License

MIT. See the repository license.