meridian-schema-forge 0.1.2

Build a processing pipeline from any schema, in any format: normalize a schema (XML/XSD, JSON/JSON-Schema, YAML, SQL DDL, delimited), classify field roles, then profile (exact tiktoken counts) or de-identify — from files or SQL.

schemaforge

schemaforge turns a schema plus a data source into an inspectable processing plan, then uses that plan to profile or de-identify the data.

Use it when you need to:

• Normalize a schema from SQL DDL, XSD/XML, JSON/JSON-Schema, YAML, JSONL, or a delimited header.
• Classify fields into auditable roles such as identifier, quasi-identifier, free text, date, numeric, categorical, boolean, and structural.
• Profile data with exact token counts and useful per-field metrics.
• De-identify files or SQL tables while preserving keys and joins.

The central artifact is plan.yaml. Generate it, review it, edit any field roles or processor options, then re-run from that same plan.

Install

Python 3.9 or newer is required.

Install from PyPI:

python3 -m pip install meridian-schema-forge

Or install from a wheel supplied to you:

python3 -m pip install meridian_schema_forge-0.1.2-py3-none-any.whl

Or install from a source checkout:

python3 -m pip install .

For local development:

python3 -m pip install -e ".[dev]"

Optional extras:

Extra	Installs	Use when
sql	pyodbc	Connecting to Microsoft Fabric, Azure SQL, or SQL Server
stream	ijson	Working with very large JSON arrays
ner	spacy	Adding statistical name detection in free text
pandas	pandas	Using DataFrame convenience workflows
dev	pytest	Running the test suite

Verify the CLI is available:

schemaforge --version

If you are running directly from a checkout without installing, use:

PYTHONPATH=src python3 -m schemaforge --version

Quick Start

The source checkout includes a quickstart with two CSV files and a matching SQL schema.

cd examples/quickstart

# 1. Inspect the normalized schema and classifier output.
schemaforge inspect-schema --schema schema.sql --classify

# 2. Build the editable plan.
schemaforge plan --schema schema.sql --source ./data --out plan.yaml

# 3. Profile the data.
schemaforge profile --plan plan.yaml --out-json profile.json

# 4. Generate a secret and de-identify the data.
schemaforge init-secret --out secret.key
schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key

# 5. Validate the profile output contract.
schemaforge validate --json profile.json --kind profile

Expected outputs:

• plan.yaml: schema, source binding, roles, and processor options.
• profile.json: token counts and per-field profile metrics.
• clean/*.jsonl: de-identified records, one JSONL file per entity.
• clean-PRIVATE/: private mapping and audit files. Do not deliver this directory with de-identified data.
• secret.key: the HMAC secret used for deterministic pseudonymization. Keep it private.

Standard Workflow

1. Inspect the schema.

   schemaforge inspect-schema --schema schema.sql --classify
   

2. Build a plan from the schema and source data.

   schemaforge plan --schema schema.sql --source ./data --out plan.yaml
   

3. Review plan.yaml.

   Check that each field has the right role. Change any role by hand if the classifier guessed incorrectly.

4. Run a processor from the plan.

   schemaforge profile --plan plan.yaml --out-json profile.json
   schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key
   

5. Validate outputs before sharing.

   schemaforge validate --json profile.json --kind profile
   

Inputs

Schema Inputs

schemaforge accepts schema files, raw schema text, and simple data files that can be used to infer a schema.

Input	Examples	Notes
SQL DDL	.sql, .ddl	Parses CREATE TABLE, primary keys, foreign keys, and column types
XML / XSD	.xml, .xsd	XSD gives declared structure; plain XML can be inferred
JSON / JSON-Schema	.json	JSON-Schema is read directly; plain JSON can be inferred
JSONL / NDJSON	.jsonl, .ndjson	Infers fields from newline-delimited records
YAML	.yaml, .yml	Reads schema-shaped YAML or infers from data
Delimited	.csv, .tsv	Uses the header as fields

Auto-detection uses extension first and then content. Force a format when needed:

schemaforge inspect-schema --schema customers.xsd --format xsd
schemaforge inspect-schema --schema schema.ddl --format sql
schemaforge inspect-schema --schema customers.csv --format delimited

Data Sources

Use files or any SQLAlchemy database URL.

For files, point --source at a directory, glob, or single file:

schemaforge plan --schema schema.sql --source ./data --out plan.yaml
schemaforge profile --schema customer.json --source ./customer.jsonl --out-json profile.json
schemaforge profile --schema schema.sql --source "./exports/*.csv" --out-json profile.json

Directory sources match schema entities to files by filename stem:

data/
  customer.csv
  order.csv

CSV, TSV, TXT, JSONL, JSON, YAML, and XML are supported. CSV, JSONL, and XML are streamed. JSON and YAML are loaded as files, so prefer JSONL or SQL for very large data.

For SQL:

schemaforge plan --schema schema.sql --source "sqlite:///app.db" --out plan.yaml
schemaforge profile --plan plan.yaml --out-json profile.json

Any SQLAlchemy URL can be used. For SQL Server, Azure SQL, or Microsoft Fabric, install the sql extra and provide an ODBC URL, for example:

schemaforge plan \
  --schema schema.sql \
  --source "mssql+pyodbc://user:password@host/database?driver=ODBC+Driver+18+for+SQL+Server" \
  --out plan.yaml

The Plan File

plan.yaml is the reproducible contract for a run. It includes:

• The normalized schema.
• The source location.
• One role for every field.
• Processor options.
• The list of roles counted as profile "content".
• A config digest and schemaforge version.

Example:

schemaforge_version: 0.1.2
source:
  kind: file
  location: ./data
  group_key: customer_id
content_roles: [categorical, free_text, numeric, quasi_identifier]
processors:
  profile: {}
  deidentify:
    date_strategy: shift
    date_max_days: 365
schema:
  entities:
    - name: customer
      primary_key: [customer_id]
      fields:
        - name: customer_id
          type: integer
          is_primary_key: true
          role: structural
        - name: email
          type: string
          role: identifier
        - name: full_name
          type: string
          role: quasi_identifier
        - name: notes
          type: string
          role: free_text

After editing the plan, run processors with --plan plan.yaml. You do not need to regenerate the plan unless the schema, source, or config changed.

Field Roles

Every field gets exactly one role.

Role	Meaning	Common examples
identifier	Directly identifies a person, account, or row	email, phone, SSN, UUID, account number
quasi_identifier	May identify someone when combined with other fields	name, birth date, ZIP, demographic fields
free_text	Narrative or unstructured text	notes, comments, descriptions
date	Date or timestamp that should be shifted or generalized	order_date, created_at
numeric	Quantity or measurement	amount, score, quantity
categorical	Label, enum, or code with limited values	status, country, type
boolean	True/false value	active, is_paid
structural	Keys or structure needed for joins and shape	primary key, foreign key, row index

Classification is deterministic. The classifier considers, in order:

1. Config overrides.
2. Primary keys, foreign keys, and structural schema facts.
3. Declared field type or format.
4. Built-in name patterns.
5. Sampled values.
6. Type fallback.

If a role is wrong, edit plan.yaml directly or use a config file.

# config.yaml
roles:
  customer.membership_code: categorical
  order.tracking_ref: identifier

thresholds:
  categorical_max_distinct: 50

content_roles: [free_text, numeric, categorical, quasi_identifier]

processors:
  deidentify:
    date_strategy: shift
    date_max_days: 365

Use the config while building the plan:

schemaforge plan --schema schema.sql --source ./data --config config.yaml --out plan.yaml

Processors

Profile

profile measures corpus size and field shape.

schemaforge profile --plan plan.yaml --out-json profile.json

It reports:

• Record counts by entity.
• Total token counts.
• Content-only token counts based on content_roles.
• Structural token counts.
• Per-field coverage, distinct counts, date ranges, numeric min/max/mean, and top values.
• Token counts using o200k_base, with cl100k_base as a cross-check.

De-Identify

deidentify transforms records based on field role.

schemaforge init-secret --out secret.key
schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key

Role behavior:

Role	De-identification behavior
identifier	Replaced with deterministic keyed surrogates
Primary and foreign keys	Replaced with deterministic keyed surrogates that preserve joins
quasi_identifier	Generalized, such as names to synthetic tokens, ZIPs to prefixes, ages to bands, dates to years
free_text	Scrubbed for known identifiers and identifier-shaped values
date	Shifted by group, preserving intervals, or generalized to year
numeric, categorical, boolean	Passed through
structural	Passed through unless it is a primary or foreign key

The write path is fail-safe. schemaforge writes to a staging directory, runs QA, and only promotes the output if the gate passes. If a leak is detected, nothing is written to the requested output directory.

Private artifacts are written to a sibling *-PRIVATE directory. Keep that directory and the secret out of any delivered dataset.

Command Reference

Command	Purpose	Example
inspect-schema	Parse and display a normalized schema	schemaforge inspect-schema --schema schema.sql --classify
plan	Build plan.yaml from schema, source, and optional config	schemaforge plan --schema schema.sql --source ./data --out plan.yaml
profile	Produce token counts and per-field metrics	schemaforge profile --plan plan.yaml --out-json profile.json
deidentify	Write de-identified JSONL output	schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key
validate	Validate an output contract	schemaforge validate --json profile.json --kind profile
run	Run a registered custom processor	schemaforge run --plan plan.yaml --processor rowcount
init-secret	Generate a private HMAC secret	schemaforge init-secret --out secret.key

Most processor commands can either use an existing plan:

schemaforge profile --plan plan.yaml --out-json profile.json

Or build a plan on the fly from --schema and --source:

schemaforge profile --schema schema.sql --source ./data --out-json profile.json

Prefer a saved plan for reviewed or repeatable work.

Python API

import schemaforge as sf

schema = sf.ingest("schema.sql")
plan = sf.build_plan(schema, "./data")

profile = sf.profile(plan)
assert not sf.errors(sf.run_qa(profile))
sf.write_json(profile.output, "profile.json")

secret = sf.generate_secret()
sf.deidentify(plan, out="clean", secret=secret)

Useful API entry points:

sf.ingest(source, fmt="auto")
sf.build_plan(schema, source, config=None)
sf.profile(plan)
sf.deidentify(plan, out="clean", secret_file="secret.key")
sf.run(plan, "processor_name")
sf.run_qa(result)
sf.errors(issues)
sf.validate(obj, "profile")
sf.write_json(obj, path)
sf.write_yaml(obj, path)
sf.generate_secret()

Custom Processors

Processors can be registered without changing core files.

import schemaforge as sf

@sf.register
class RowCount(sf.Processor):
    name = "rowcount"

    def run(self, reader, plan):
        rows = sum(1 for entity in reader.entities() for _ in reader.iter_records(entity))
        return sf.ProcessorResult(self.name, {"rows": rows})

sf.run(plan, "rowcount")

Run a registered processor from the CLI:

schemaforge run --plan plan.yaml --processor rowcount

Quality Gates

schemaforge uses hard QA gates. An error exits with code 2; usage or runtime errors exit with code 1; success exits with code 0.

Gates include:

• Schema has at least one entity and uniquely named fields.
• Every field has exactly one valid role.
• Plans round-trip and validate against the bundled plan contract.
• Profile output has internally consistent token, count, coverage, percentile, and distribution metrics.
• De-identification has matching input/output counts, injective mappings, and no known identifier or identifier-shaped value surviving verbatim in output.

For de-identification, QA runs before output promotion, so a failed gate writes nothing to the requested output directory.

Troubleshooting

schemaforge: command not found

Install the package first with python3 -m pip install . or run from a checkout with PYTHONPATH=src python3 -m schemaforge ....

No module named schemaforge

The package is not installed in the interpreter you are using. Run python3 -m pip install . from the repository root or activate the environment where schemaforge was installed.

An entity has zero records

For file sources, confirm each filename stem matches an entity name in the schema, such as customer.csv for entity customer. For a single JSON, YAML, or XML container file, confirm entity arrays are keyed by entity name.

A field has the wrong role

Edit plan.yaml and re-run the processor, or add a role override in config.yaml and rebuild the plan.

De-identification fails with a leak gate

Review the reported field/value shape, mark the source field as identifier or quasi_identifier, and re-run. No requested output directory is written when the gate fails.

Requirements

• Python 3.9+
• PyYAML
• jsonschema
• tiktoken
• SQLAlchemy
• Optional database drivers for non-SQLite SQL sources

License

Proprietary - Meridian Intelligence, engagement-scoped. This software is provided solely for the specific engagement for which it is supplied and must be deleted and cleared after that exercise. See LICENSE and LICENSE.pdf.