Schema Quarry

Features

• Convert OpenAPI definitions to Parquet schemas
• Use Schema Quarry as a CLI or as a Python library

Requirements

• Python 3.12+

Installation

You can install Schema Quarry via pip from PyPI:

pip install schema-quarry

For local development with uv:

uv sync --dev

Usage

CLI:

schema-quarry generate \
  --source "https://petstore3.swagger.io/api/v3/openapi.json" \
  --root-schema Pet \
  --output-file pet.parquet \
  --print-schema

Skattemelding example:

schema-quarry generate \
  --source "https://app.swaggerhub.com/apiproxy/registry/skatteetaten/skattemelding-api/4.2.0" \
  --root-schema Skattemelding \
  --output-file skattemelding.parquet \
  --print-schema

Write directly to Google Cloud Storage:

schema-quarry generate \
  --source "https://app.swaggerhub.com/apiproxy/registry/skatteetaten/skattemelding-api/4.2.0" \
  --root-schema Skattemelding \
  --output-file "gs://my-bucket/schemas/skattemelding.parquet"

Library:

from schema_quarry import build_parquet_schema

result = build_parquet_schema(
    source="https://petstore3.swagger.io/api/v3/openapi.json",
    root_schema="Pet",
    output_file="pet.parquet",
)

print(result.schema)
print(result.parquet_path)

output_file also accepts gs://... URIs and writes them with gcsfs.

Please see the Reference Guide for details.

Tests

Run the full test suite locally with:

uv run pytest

The tests are split into two groups:

• tests/unit/ checks the building blocks of the project, such as the CLI, the Python library API, and specific OpenAPI-to-Parquet mapping rules
• tests/golden/ checks that real example inputs still produce exactly the same output as the checked-in reference files

The checked-in test data is stored in tests/resources/:

• tests/resources/master/openapi/ contains OpenAPI documents used as regression inputs
• tests/resources/master/parquet/ contains the expected Parquet schemas for those inputs
• tests/resources/snapshots/schema-text/ contains expected text output for format_schema(...)

In other words:

• if you change the converter logic, the golden tests will tell you whether the generated Parquet schema changed for any of the reference APIs
• if you change schema formatting, the snapshot tests will tell you whether the human-readable schema text changed

You can also run the local quality checks with:

uv run ruff check tests
uv run mypy tests

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the MIT license, Schema Quarry is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Credits

This project was generated from Statistics Norway's SSB PyPI Template.