Two halves of one idea

The JSON is the contract between them. Studio never invents syntax the framework does not support, and the framework never needs Studio to run.

Install

Framework

pip install sparquet

from sparquet import Sparquet

fw = Sparquet(spark={"app_name": "MyJob", "master": "local[*]"})
result = fw.run("pipeline.json", params={"dt_ref": "2026-01-01"})
print(result.summary())
fw.stop()

Or from the CLI:

python -m sparquet.cli pipeline.json

Studio

cd sparquet-studio
npm install
npm run dev            # http://localhost:5273

Full instructions, including the AI setup and the local runner, are in the Studio README.

A pipeline in 30 seconds

{
  "name": "customers_curated",
  "input": { "format": "csv", "path": "/data/landing/customers" },
  "transformations": [
    { "type": "filter", "condition": "status = 'active'" },
    { "type": "cast", "columns": { "created_at": "timestamp" } },
    { "type": "with_column", "column": "loaded_at", "expression": "current_timestamp()" },
    { "type": "drop_duplicates", "columns": ["id"] }
  ],
  "validations": {
    "on_failure": "warn",
    "rules": [
      { "type": "not_null", "columns": ["id"] },
      { "type": "unique", "columns": ["id"] }
    ],
    "report": { "format": "csv", "path": "/dq/customers", "mode": "append" }
  },
  "output": {
    "format": "parquet",
    "path": "/data/curated/customers",
    "mode": "overwrite",
    "partition_by": ["created_at"]
  }
}

That file runs as-is, and opens on the Studio canvas as nine connected nodes.

What the language covers

Transformations — filter select drop rename cast with_column struct drop_duplicates distinct sort fill_na sql group_by join (with broadcast map-side hint) union checkpoint stop_if_empty collect debug

IO — parquet delta iceberg csv txt view kafka read and write; relational via JDBC (postgresql mysql mariadb sqlserver oracle); warehouses (bigquery snowflake redshift); NoSQL/search (mongodb documentdb dynamodb cassandra elasticsearch). Delta and Iceberg support MERGE upserts and Delta time travel; external connectors need their driver JAR on the Spark classpath.

Validations — powered by sparquet-cola, a standalone data-quality library (pyspark-only) that installs as a dependency (pip install sparquet-cola) and is usable on its own. Rules: not_null unique range regex row_count sql (boolean invariant OR failed_rows mode), plus SODA-style check (a metric vs a warn/fail threshold) and schema (columns and types). fail / warn / skip policies, an optional per-rule metrics report, and row-level quarantine (validations.outputs: split valid/invalid to their own sinks) — all written apart from the main output.

Beyond the basics

• Template parameters {param} substituted before parsing, with list and boolean formatting for SQL (IN ('a','b')), plus skip_if_false to switch whole steps on and off per run.
• Runtime variables {{var}} — collect a column into a variable and push it into a later read as a literal IN (...), the declarative form of the collect() + isin() trick that makes Delta data skipping work.
• Multiple destinations with per-destination column projections and per-destination transformations, so one DataFrame can land as Parquet, as a Delta merge and as a Kafka topic in a single pass.
• Includes — { "$include": "shared/filters.json" } to share fragments across pipelines.
• Extensible — register your own readers, writers, transformations and validators; Studio keeps unknown node types intact when it opens the file.

See CLAUDE.md for the complete schema reference.

Documentation

Repository layout

sparquet/        the framework (readers, writers, transformations, validations)
sparquet-studio/        the visual editor (React + TypeScript) and its local runner
website/                landing page and documentation site (Astro + Starlight)
examples/               example pipelines, one per capability
tests/                  unit tests and a full real-world migration case
docs/                   deployment and reference docs

Contributing

Issues and pull requests are welcome, on either half. For Studio, keep npm run typecheck, npm run test and npm run lint clean and run npm run smoke when touching the canvas. For the framework, add an example pipeline covering any new capability.

License

Apache 2.0 — see LICENSE.