pqax for Python

pqax reads, validates, and writes PQAX files: the footprint of Parquet tables, as a zip of JSON tables. The format is defined by the serialization and the object model, and the validation rules by the conformance kit.

Install

pip install pqax                       # the package and the pqax command
pip install "pqax[pandas]"             # with pandas, for to_dataframe
pip install "pqax[polars]"             # with polars, for to_dataframe
pip install "pqax[parquet]"            # with pyarrow, for export to Parquet
pip install "pqax[delta]"              # with deltalake and pyarrow, for export to Delta
pip install "pqax[collect]"            # with pyarrow, for the collector
pip install "pqax[azure]"              # with azure-identity, for the sign-in of the collector

From the root of the repository:

pip install -e src/python              # the package and the pqax command
pip install -e "src/python[pandas]"    # with pandas, for to_dataframe
pip install -e "src/python[dev]"       # with pytest and jsonschema, for the tests; on Windows tzdata too, which pyarrow needs to read the UTC timestamps of an export back

Python versions: 3.10 or later. The tests run on 3.10 and 3.14, each without extras and with both pandas and polars. The package has no dependency: the pandas and polars extras only install the library that to_dataframe uses, the parquet and delta extras the libraries that export uses, and the collect and azure extras those that collect uses. The tests run with pyarrow, deltalake, and azure-identity where they run with pandas and polars.

Read a file

import pqax

with pqax.open("footprint.pqax") as doc:
    print(doc.manifest["formatVersion"], doc.part_names)
    tables = doc.summary["tables"]               # read when the file opens
    for table_id, size in zip(tables["id"], tables.get("fileSize", [])):
        print(table_id, size)
    chunks = doc.table("columnChunks")           # read now, from all its parts, and kept
    print(doc.row_count("columnChunks"), sum(chunks["compressedSize"]))
    for row in doc.rows("files"):                # one dictionary for each row
        print(row["id"], row["fileSize"])

A table is a dictionary of fields, and a field is a list with one value for each row, as in the file. The keys keep the names of the file, such as fileCount. A value that the file does not have is None.

• Opening a file reads the zip directory, manifest.json, and summary.json. A detail part is read the first time a table that it holds is asked for.
• doc.table(name) concatenates the parts of a table in the order of the manifest. A field that a part leaves out is None in the rows of that part. A table that the file does not have is {}.
• A field that has no value in any row is not in the dictionary: use table.get("field").
• A value whose JSON type is not the type of its field, such as a string where the model says integer, reads as None. doc.raw(name) returns the table as parsed, with that value and with the members of a table that are not arrays.
• Fields and tables that this version does not define are returned like the others. Members of the manifest that it does not define stay in doc.manifest.

A PQAX file embedded in a VPAX file:

with pqax.open_embedded("model.vpax") as doc:    # the part Footprint.pqax, stored or deflated
    print(doc.embedded_part, doc.embedded_compression)

The part is read into memory, and the VPAX file is closed before open_embedded returns.

Validate a file

result = pqax.validate("footprint.pqax")         # or pqax.validate_embedded("model.vpax")
print(result.is_valid, result.checks)
for issue in result.issues:
    print(issue.code, issue.message, issue.part, issue.table, issue.field)

The validator reports every issue that it finds, with the codes of section 4.1 of the contracts: container.*, json.*, manifest.*, part.*, table.*, depth.*, totals.*, and, for validate_embedded, embedded.partMissing. A rule that cannot be evaluated because an earlier one failed is skipped. A file that cannot be opened raises OSError. A file that is not a zip archive is an issue, container.manifestMissing, and so is a VPAX file given to validate.

Data frames

frame = pqax.to_dataframe(doc.table("columnChunks"))             # pandas when installed, else polars
frame = pqax.to_dataframe(doc.summary["tables"], library="polars")

With pandas, an integer or boolean column with None values gets a nullable type, so it does not turn into floats. Lists and objects stay as Python objects. Timestamps stay strings; pqax.parse_timestamp turns one into a datetime. The rows of doc.rows(name) give the same frame as doc.table(name).

to_dataframe needs polars 1.0 or later, and raises ImportError with an earlier version, which loses the values of a column that mixes types.

Write a file

from datetime import datetime, timezone

builder = pqax.Builder({"collectorApp": "my-tool", "level": 2, "startedAt": datetime.now(timezone.utc)})
source = builder.add_source(kind="Storage folder")
table = builder.add_table(source=source, name="sales", format="Delta", detailDepth="Files")
file_id = builder.add_file(table=table, path="part-0001.parquet", fileSize=47820)
builder.update("tables", table, fileCount=1, fileSize=47820)
builder.write("footprint.pqax")

The fields are named as in the file. An add_ method refuses a field that its table does not have, unless the name has a producer's prefix and an underscore, as in acme_note. A field set to None has no value. A datetime becomes a timestamp in UTC with three decimals; a naive datetime is taken as UTC.

write lays out the file as the serialization says: manifest.json first, then summary.json, always present, then one part for each detail table, named after it, as files.json. A table with more than rows_per_part rows, 250,000 by default and at most, continues in parts numbered from 1, as columnChunks.1.json. A field with no value in any row of a part is left out of that part, and so is a table with no rows. The JSON is compact UTF-8 with no byte order mark, the entries are deflated at level 9 by default (compression_level), and every entry has the same date, so the same footprint gives the same bytes. A path is written through a temporary file.

from_document keeps the ids, the collection, the obfuscation fields, the other members of the manifest, and the fields and tables that this version does not define. It loses what the reader does not keep: the entries that the manifest does not list, the members of a table that are not arrays, and the members of an element of parts other than name and tables. It writes format version 0.2, and its own part names.

Measure the tables, the columns, and the collection

pqax.measures computes every derived measure of section 10 of the object model for the tables, the columns, and the collection, with the names and the numbers of the storage views: the storage stack with its waste, its overhead, and the share of each component, the file sizes, the small files, and the V-Order shares, the history with VACUUM and OPTIMIZE, the ratios, the partitions, the measures of each column, and the totals of the collection. Only the two measures over a set of columns, which the treemap of the views draws, are not here.

import pqax

with pqax.open("footprint.pqax") as doc:
    result = pqax.measures.measure(doc)          # reads the files and partitions tables when the file has them
for table in sorted(result["tables"], key=lambda t: -(t["stack"]["waste"] or 0)):
    print(table["label"], table["stack"]["waste"], table["stack"]["wasteShare"],
          table["files"]["median"], table["maintenance"]["lastVacuumAt"], table["partitions"]["count"])
print([column["label"] for column in result["columns"] if (column["nullShare"] or 0) > 0.5])
print(result["collection"]["totals"]["footprint"], result["collection"]["depths"])

The keys are the names of the view model. The dict of a table has id, name, label, rowCount, fileCount, fileSize, stack with shares, smallFiles, tinyFiles, files with vOrder, measures, maintenance, and partitions; python -m pqax tables --json prints the keys it printed in version 0.2.0. A column has id, table, name, label, and its six measures, and the collection totals and depths. A value that cannot be computed is None, never 0; the waste, the overhead, and the total count a missing component as 0, as the views do, and are None only when every component they add is missing. An integral float is an int, and a time is the text of the file. The conformance kit holds the measures of the views for every sample, in conformance/expected/measures, and the tests compare these with them.

Compare two footprints

pqax.diff compares two footprints of the same lakehouse taken at different times, as section 13 of the object model proposes: the tables and the columns added, removed, and kept, and for every kept one the older value, the newer value, and the delta of each total of section 6 and of each measure of section 10. A table is the same table when its tableId is, and otherwise, when one of the two has none, when its source, namespace, and name are; a column when its fieldId is, and otherwise by name; a partition by its values, a file by its path, and a commit by its commitId.

import pqax
from pqax import diff

with pqax.open("monday.pqax") as older, pqax.open("friday.pqax") as newer:
    result = diff.compare(older, newer)
print(result["counts"], result["collection"]["footprint"])
for kind, entry in diff.rank(result, "total", top=10):
    if kind == "kept":
        print(entry["newer"]["label"], entry["status"], entry["measures"]["stack.total"]["delta"], entry["files"])
    else:
        print(entry["label"], kind, entry["recreated"], entry["total"])

A compared value is {"older": ..., "newer": ..., "delta": ...}. The delta of a time is in milliseconds, a time is written in the form of the format, and a boolean, a text, and a list have no delta; the delta is None when either value is. A kept table has status, matchedBy (tableId or name), renamed, older and newer, totals keyed by the names of the fields, with the members of a distribution as fileSizes.median, measures keyed by the paths of the measures, such as stack.waste, columns with added, removed, and kept, and files, partitions, and commits, None when the two footprints do not both keep them. A measure alone does not make a table changed, since the days since the last commit move with the time of the collection. The conformance kit has two footprints with known differences and their expected diff, in conformance/expected/diff, and the tests compare the comparison with it.

Command line

python -m pqax check footprint.pqax      # exit code 0 when valid, 1 when not, 2 on a usage or I/O error
python -m pqax info footprint.pqax       # format, version, collection, parts with their tables and rows
python -m pqax info model.vpax           # a VPAX file: the PQAX file that it embeds as Footprint.pqax
python -m pqax tables footprint.pqax     # one row per table: rows, files, live bytes, waste, and waste share
python -m pqax export footprint.pqax --to exports   # the tables and their measures as CSV, Parquet, or Delta
python -m pqax diff monday.pqax friday.pqax   # the tables and columns added, removed, and kept, the largest change first
python -m pqax collect --local workspace --output footprint.pqax   # the footprint of the Delta tables of a folder or OneLake

check prints one line for each issue, its code first, then a summary line. The commands take --part NAME for another embedded part. tables takes --sort waste|total|rows|files|name, --top N, --wide for the small files and the overhead, and --json for one JSON object; it writes what pqax tables of Pqax.Cli writes. diff takes --sort total|waste|rows|files|name, --top N, and --json, and --part NAME for both files; it writes what pqax diff of Pqax.Cli writes, the same text and the same JSON. collect takes the options of pqax collect in C#, as the section below says. The installed package also has a pqax command.

Export the tables, day after day

pqax.export writes the tables of a footprint and its derived measures as CSV, Parquet, or Delta, so that a BI tool or a lakehouse can load them. The exports of several days go into the same folder and append to the same tables, and each row says which footprint it comes from. The Power BI template reads them. On a mounted file system without rename, /lakehouse/default in a Fabric notebook or /dbfs on Databricks, the Delta log is committed by a copy and a delete, the option allow_unsafe_rename of deltalake, which the export turns on there by itself.

python -m pqax export footprint.pqax --to exports                    # CSV, with the standard library only
python -m pqax export footprint.pqax --to exports --format parquet   # needs pqax[parquet]
python -m pqax export model.vpax --to Tables --format delta          # needs pqax[delta]; the embedded footprint

from pqax import export

result = export.export("footprint.pqax", "exports", format="parquet")
print(result["footprintId"], result["collectedAt"], result["tables"])

Without the library, the export raises ImportError naming the extra, and the command exits with 2, before anything is written.

The output tables:

Every row starts with footprintId and collectedAt. export.COLUMNS lists the columns of each table, and export.SCHEMAS their kinds.

• footprintId is the SHA-256 of the bytes of the PQAX file, in lowercase hex; for a VPAX, of the PQAX file that it embeds, which has the same bytes as the file on its own. The id comes from the content and from nothing else: no collector has to assign one, the same footprint has the same id wherever it is copied or embedded, and a different collection always has another id, since its manifest holds its times. That is what lets the export refuse a footprint twice.
• collectedAt is completedAt of the collection, else startedAt, as collectedAt of pqax tables --json. footprints also has collectionDate, its date in UTC.
• Names. A table of the format and its fields keep the names of the file. A measure takes its name in section 10 of the object model, with a lower-case first letter, as rule 5 of section 3 of the contracts says for the keys of a table: live, wasteShare, smallFileShare, vOrderShare, lastCommitAt, partitionsKept. Section 10 does not name the share of each stack component, and NonLiveShare is already a measure, so those shares are liveDataShare, nonLiveDataShare, and so on: the component over Data. The file size figures are fileSizesCount, fileSizesSum, fileSizesMinimum, fileSizesFirstQuartile, fileSizesMedian, fileSizesThirdQuartile, and fileSizesMaximum. tableMeasures also has sourceName and some fields of its table, shown beside the stack: detailDepth, nonLiveFileCount, nonLiveOldestRemovedAt, nonLiveNewestRemovedAt, unreferencedFileCount. columnMeasures has tableLabel.
• Types. The type of a column comes from the format, not from the values, so that the files of every day have the same schema. Integers are int64, the measures float64, the booleans bool, the timestamps UTC timestamps in microseconds, and collectionDate a date. A list, a map, or a nested object, such as namespace, properties, or fileSizes, is its compact JSON text. A timestamp that does not parse is null.
• Left out. The tables and the fields that the format does not define, such as those of a producer, acme_note, are not exported; the result lists the tables in ignored, and the command prints them. A table with no rows is not written.

The folder:

A footprint whose id the footprints table of the folder already has is refused with AlreadyExportedError, code export.duplicate, and exit code 1, and nothing is written; if_exists="skip", or --if-exists skip, writes nothing and returns, with exit code 0. footprints is written last, so a footprint in it was exported whole. An export that fails half way, on a full disk for example, can leave rows of the footprint in the CSV files or the Delta tables written before; the Parquet files of a footprint are replaced by the next export of it.

In a Fabric notebook, --to /lakehouse/default/Tables --format delta writes the tables straight into the lakehouse attached to the notebook.

Collect a footprint

pqax.collect collects the footprint of the Delta tables of a lakehouse and writes it as a PQAX file. It is a port of the C# collector, Pqax.Collector, and that README describes what both read and what they keep: the options, the requests, the commits read, the unreferenced files, the locations, the issues, the footer failures, and the footer parser. It reads the Delta log of each table and, at level 2, the footer of each Parquet file, never a row, and it keeps no data value unless asked. On the synthetic lakehouse of the conformance kit its footprint holds the same values as the one of pqax collect in C#, in every table and field, and the requests and bytes too; only the members of the collection that name the collector and the times differ, and cross_check.py checks it.

pip install "pqax[collect]"                                                                        # a folder or a notebook
pip install "pqax[collect,azure]"                                                                  # OneLake with a sign-in
python -m pqax collect --local workspace --output footprint.pqax                                   # a folder in the OneLake layout
python -m pqax collect --mount /lakehouse/default --item Sales.Lakehouse --output sales.pqax       # the lakehouse of a notebook
python -m pqax collect --workspace "My Workspace" --item Sales.Lakehouse --output sales.pqax --access-token-env PQAX_TOKEN
python -m pqax collect --workspace "My Workspace" --item Sales.Lakehouse --output sales.pqax --azure-cli
python -m pqax collect --workspace "My Workspace" --item Sales.Lakehouse --output sales.pqax       # a device code, with pqax[azure]

The command has the options of pqax collect in C#, with the same names, defaults, checks, and messages: --local, --workspace, --item, --output, --depth, --level, --max-footers, --max-commits, --parallelism, --verbose, and the identities of the Pqax.Cli README. It prints the same lines, with counts and never the name of a table, and has the same exit codes: 0 when the file is written and valid, 1 when it is written and not valid, 2 for a wrong command line or a failure, such as a missing folder, no table, a refused sign-in or token, or a missing library. A secret given on the command line, as --client-secret or --access-token, is refused without being repeated. Two options are Python's own:

The function does the same from Python and returns a pqax.Builder:

import pqax.collect as collect

storage = collect.LocalStorage.mount("/lakehouse/default", "Sales.Lakehouse")     # or LocalStorage("workspace")
builder = collect.collect(storage, level=2, max_commits=30)                      # every table of every item
builder.collection = {"collectorApp": "my notebook", "collectorAppVersion": "1", **builder.collection}
builder.write("sales.pqax")

token = notebookutils.credentials.getToken("storage")
storage = collect.OneLakeStorage("My Workspace", lambda: token)                  # a token provider, called per request
tables = collect.discover(storage, "Sales.Lakehouse")
builder = collect.collect(storage, tables, depth="files", progress=print)

from azure.identity import DefaultAzureCredential
provider = collect.identity.TokenCache(collect.identity.azure_credential(DefaultAzureCredential()))
storage = collect.OneLakeStorage("My Workspace", provider)                       # a token cached until 5 minutes before it expires

What differs from C#: the message of an issue that quotes the error of a JSON parser, since the two parsers word their errors differently; the codes are the same. The tables are collected on threads, as many as parallelism, and the requests in flight never exceed it.

API

Errors:

All derive from PqaxError, whose code is the validation code that the error matches, when there is one. A reader refuses a file for the first four, when it opens it, and open_embedded also for a missing embedded part. Opening also fails when the manifest or the summary cannot be read from the zip, for example because the entry is encrypted. A summary.json that the manifest lists and the archive does not have gives empty summary tables, and validate reports it. A detail part that is broken fails only the read of its tables.

Limits

• The collector has the limits of the C# collector: levels 1 and 2, Delta tables only, checkpoints up to 512 MB, the log compaction files counted and not used, and every file of a table in memory until the footprint is built. Its footer parser is plain Python: about 25 microseconds per column chunk on a laptop, so about half a minute of parsing for a million column chunks.
• The reader keeps a table in memory once it is read. A part of 250,000 column chunks takes about 0.15 s to parse.
• A part that holds more than 1 GB of JSON text is not read, and is reported or raised as such.
• The rows of a part are the length of its longest array, not what the manifest states, so that a file that states more rows than it holds cannot make the reader allocate them. In a valid file the two agree.
• The reader replaces a value of another JSON type with None only at the level of the field. An element of a list, or a member of a nested object, is returned as the file has it.
• container.zip64 looks at the central directory and its locator, as the kit says. A ZIP64 field in a local header alone is not reported.
• No pandas or polars type is chosen for timestamps, lists, maps, or nested objects.
• On Python 3.10, parse_timestamp reads fewer of the other forms of ISO 8601, as datetime.fromisoformat does there: a fraction of seconds with other than three or six digits, a comma before the fraction, an offset without a colon, and the basic form, such as 20261005T081530Z, raise ValueError. The form of the format reads on every version.