schema-sanitizer

Version 0.2.0: this project is still in a testing phase. Expect the core behavior to be exercised heavily before treating it as a stable production dependency.

The extension is currently being tuned and tested for generating Parquet files and schemas used by BigQuery external tables.

schema-sanitizer turns extremely messy semistructured data into stable, consistent tables. It is built for CSV, JSON, JSON Lines, XML, Parquet, and Python rows whose real-world values do not agree on one neat schema: fields appear late, arrays and objects change shape, timestamps arrive in several formats, scalars collide with nested values, and malformed records still need a place to go.

The library's main purpose is to make ingestion predictable before data reaches analytics engines, warehouses, or incremental pipelines. It scans source data, infers a reconciled Arrow schema, converts compatible values into that schema, and isolates rows that cannot be represented cleanly. The result is a table that downstream tools can consume without rediscovering schema drift on every run.

The hard parts are handled explicitly:

• Turning messy semistructured data into tables: mixed scalar, list, struct, null, date/time, and string values are reconciled into stable columns.
• Schema reconciliation for incremental pipelines: schema_registry carries the canonical schema and version routing between runs. The registry is the single public source of truth for incremental schema state.
• Field-name sanitization: output field names default to lowercase a-z only, with deterministic suffixes when dirty source keys collide after cleaning.
• Memory safety: readers and converters use bounded batches, streaming writers, spill-to-disk paths where needed, depth limits, row-size budgets, and row-level error policies so large or malformed inputs do not require loading the whole cleaned dataset into memory.
• Max depth enforcement: Arrow and Parquet depth budgets can cap deeply nested records before they exceed downstream limits such as warehouse nesting constraints.

Every public reader and converter returns a Result object with clean data and stats.

It has two public workflows:

• In-Memory Analytics: read_* functions return a Result whose clean_data is PyArrow, pandas, Polars, or DuckDB data.
• File-To-File Converters: to_* functions stream sanitized files to CSV, JSON Lines, or Parquet and return a Result whose clean_data is None.

import schema_sanitizer as ss

events = ss.read_jsonl("raw/events.jsonl")
customers = ss.read_csv("raw/customers.csv", output_format="pandas")

table = events.clean_data
df = customers.clean_data

ss.to_parquet("raw/events.jsonl", "clean/events.parquet")

Index

• Install
• In-Memory Analytics
• File-To-File Converters
• Result Object
• Error Handling
• Schema Control
• Field Name Sanitization
• Timestamp Precision
• Custom Tokens and Date/Time Patterns
• In-Memory Analytics Options
  • read_csv(path, ...)
  • read_json(path, ...)
  • read_json_folder(path, ...)
  • read_jsonl(path, ...)
  • read_xml(path, ...)
  • read_xml_folder(path, ...)
  • read_parquet(path, ...)
  • read_python(rows, ...)
• File-To-File Converter Options
  • to_csv(input_path, output_path, ...)
  • to_jsonl(input_path, output_path, ...)
  • to_parquet(input_path, output_path, ...)
• Schema Inference Heuristics
• Embedded Schema Registry Columns
• Max Depth Enforcement
• Memory Safety Measures
• Large File Tuning
• PyArrow Filesystem Integration
• Supported Inputs
• Unsupported Inputs
• Examples
• Platform Notes
• Development
• License

Install

schema-sanitizer supports Python >=3.11.

For Arrow reads and file-to-file converters:

pip install 'schema-sanitizer[pyarrow]'

Install adapter extras for the in-memory analytics tools you use:

pip install 'schema-sanitizer[pyarrow,pandas]'
pip install 'schema-sanitizer[pyarrow,polars]'
pip install 'schema-sanitizer[pyarrow,duckdb]'
pip install 'schema-sanitizer[all]'

Import with an underscore:

import schema_sanitizer as ss

In-Memory Analytics

Use read_* when you want clean data back in Python with stats.

Readers always return a Result. By default, result.clean_data is a PyArrow table.

result = ss.read_jsonl("data/events.jsonl")

print(result.clean_data.schema)
print(result.clean_data.num_rows)
print(result.stats)

Choose another in-memory analytics target with output_format.

pandas_result = ss.read_csv("data/customers.csv", output_format="pandas")
polars_result = ss.read_csv("data/customers.csv", output_format="polars")
duckdb_result = ss.read_csv("data/customers.csv", output_format="duckdb")

pandas_df = pandas_result.clean_data
polars_df = polars_result.clean_data
duckdb_rel = duckdb_result.clean_data

Accepted output_format values are pyarrow, pandas, polars, and duckdb.

Use read_python for rows that are already in memory.

rows = [
    {"id": 1, "active": "yes", "score": "10.5"},
    {"id": 2, "active": "no", "score": 8},
]

result = ss.read_python(
    rows,
    true_tokens=("yes",),
    false_tokens=("no",),
)

table = result.clean_data

File-To-File Converters

Use to_* when you want a sanitized output file and do not need clean data in memory. These functions stream sanitized output and return a Result with clean_data set to None, plus stats.

result = ss.to_parquet("raw/orders.csv", "clean/orders.parquet")

assert result.clean_data is None
print(result.stats)

ss.to_csv("raw/events.jsonl", "clean/events.csv")
ss.to_jsonl("raw/orders.parquet", "clean/orders.jsonl")

Converters infer the input format from the input file extension. If the input path has no useful extension, pass input_format.

ss.to_parquet("raw/events", "clean/events.parquet", input_format="jsonl")

Accepted input_format values are auto, csv, json, jsonl, ndjson, xml, and parquet.

Result Object

All public read_* and to_* functions return schema_sanitizer.Result.

For readers, result.clean_data contains the requested clean in-memory output. For converters, clean data is written to output_path, so result.clean_data is always None.

result = ss.read_csv("data/customers.csv", output_format="pandas")

df = result.clean_data
stats = result.stats

Result Stats

result.stats is a plain dict. All properties are integers and default to 0 when the runtime did not report that counter.

Error Handling

By default, rows that fail materialization are kept as null rows. Choose a policy with on_error.

result = ss.read_jsonl(
    "data/events.jsonl",
    on_error="emit_null_row",
)

print(result.stats)

Converters return the same Result shape as readers. Because the clean data is written to output_path, converter results always have clean_data is None.

result = ss.to_parquet(
    "raw/events.jsonl",
    "clean/events.parquet",
    on_error="emit_null_row",
)

print(result.stats)

Schema Control

For one-off reads, schema-sanitizer infers the output schema from the current input. File-to-file converters always use the embedded schema registry path: pass the previous schema_registry value fetched from the existing output table when continuing an incremental pipeline. The registry carries the canonical schema, field versions, and drift history between runs.

column_order defaults to alphabetically, which orders output fields lexicographically at every struct depth. Registry-backed strict writes may use column_order="schema_contract_first" to preserve existing registry field order and append new fields alphabetically.

Field Name Sanitization

field_name_policy defaults to lower_alpha. In this mode, every output field name is lowercased and stripped to characters a-z only. The rule is applied recursively to inferred schemas and to registry-derived schema contracts before materialization, so dirty source keys such as User-ID, user_id, @id, and #text become BigQuery-friendly column names.

result = ss.read_jsonl(
    "raw/events.jsonl",
    field_name_policy="lower_alpha",  # default
)

When two sibling source keys clean to the same base name, all members of that collision group receive a deterministic lowercase suffix derived from the original dirty key. This keeps the dirty-key to clean-key mapping stable even if the input observes the colliding keys in a different order.

Use field_name_policy="preserve" only when you want the output schema to keep source field names exactly as observed.

Use field_name_policy="lower_snake" when you need lowercase letters, digits, and underscores in output names. This is useful for BigQuery-oriented metadata and schema-variant columns such as schema_registry, schema_drifts, and sentences_v2.

Timestamp Precision

Timestamp strings are parsed internally with nanosecond precision, then written to the output Arrow schema using timestamp_precision.

result = ss.read_jsonl(
    "data/events.jsonl",
    timestamp_precision="TIMESTAMP_MICROS",
)

ss.to_parquet(
    "raw/events.jsonl",
    "clean/events.parquet",
    timestamp_precision="TIMESTAMP_MICROS",
)

Accepted values are TIMESTAMP_MILLIS, TIMESTAMP_MICROS, and TIMESTAMP_NANOS. The default is TIMESTAMP_MICROS because it is compatible with BigQuery Parquet external tables. Selecting TIMESTAMP_NANOS preserves nanosecond Arrow/Parquet timestamps, but some downstream engines, including BigQuery, do not support Parquet TIMESTAMP_NANOS.

When parsed timestamp strings contain finer precision than the selected output unit, the value is truncated to that unit. Integer values coerced into timestamp fields are interpreted as already being in the selected output unit.

Custom Tokens and Date/Time Patterns

Use true_tokens and false_tokens when boolean values use domain-specific strings. Use temporal regex options when dates or times do not match the built-in parsers.

result = ss.read_csv(
    "data/events.csv",
    true_tokens=("yes", "enabled", "1"),
    false_tokens=("no", "disabled", "0"),
    timestamp_patterns=(
        r"^(\d{4})/(\d{2})/(\d{2})[ T](\d{2}):(\d{2}):(\d{2})$",
    ),
    date_patterns=(
        r"^(\d{4})\.(\d{2})\.(\d{2})$",
    ),
    time_patterns=(
        r"^(\d{2})h(\d{2})m(\d{2})s$",
    ),
)

table = result.clean_data

For timestamp_patterns, capture groups 1-6 are year, month, day, hour, minute, and second. Optional group 7 may contain fractions, and group 8 may contain a timezone. For date_patterns, groups 1-3 are year, month, and day. For time_patterns, groups 1-3 are hour, minute, and second.

In-Memory Analytics Options

Each reader accepts the parameters listed in its section.

read_csv(path, ...)

read_json(path, ...)

read_json_folder(path, ...)

read_json_folder reads the direct .json children of a local folder or PyArrow filesystem folder URI in deterministic filename order. Folder exploration is not recursive. Each source file must contain one JSON document; the reader compacts those documents into a temporary JSON Lines stream and then runs the same sanitizer path used by read_json.

read_jsonl(path, ...)

read_xml(path, ...)

read_xml parses a local XML document in the native C++ frontend and sends the resulting rows through the same schema inference, cleaning, and output adapter pipeline as the JSON and CSV readers.

By default, the root element is treated as one row, like a single JSON object. Pass xml_row_tag="row" when a file contains repeated direct child elements that should become separate rows; the XML scanner then streams each matching row element. Internally, attributes are exposed as fields prefixed with @, repeated child tags become lists, and mixed element text is stored under #text. With the default field_name_policy="lower_alpha", those XML helper names are emitted as sanitized columns such as id and text; use field_name_policy="preserve" to keep @id and #text.

result = ss.read_xml(
    "raw/orders.xml",
    xml_row_tag="order",
    read_chunk_bytes=1024 * 1024,
    batch_memory_limit_bytes=256 * 1024 * 1024,
)

read_xml_folder(path, ...)

read_xml_folder reads the direct .xml children of a local folder or PyArrow filesystem folder URI in deterministic filename order. Folder exploration is not recursive. Each source file must contain one XML document, and all documents must use the same root tag unless you pass that tag explicitly as xml_row_tag. The reader wraps those documents in a temporary XML stream and then runs the same sanitizer path used by read_xml.

result = ss.read_xml_folder(
    "raw/order-events",
    xml_row_tag="order",
    batch_memory_limit_bytes=256 * 1024 * 1024,
)

read_parquet(path, ...)

read_python(rows, ...)

File-To-File Converter Options

Converters accept local or PyArrow FS URI output paths. Inputs can be local paths or PyArrow FS URI strings. They infer input format from the input extension unless you pass input_format.

to_csv(input_path, output_path, ...)

to_jsonl(input_path, output_path, ...)

to_parquet(input_path, output_path, ...)

Schema Inference Heuristics

Schema inference scans the full source before materialization whenever inference runs. It is not a sample-based inference step: in inferred mode and additive schema_registry mode, every source row is consumed during inference and counted in Result.stats["inferred_rows"].

For each inferred row, the sanitizer applies two internal passes:

1. The shape pass discovers structural paths: field names, objects, arrays, and fields that must be flattened by depth limits.
2. The statistics pass collects scalar type evidence for the discovered shape: booleans, integers, floats, timestamps, dates, times, strings, nulls, and mixed-type conflicts.

When a file converter runs, the previous registry is merged with the current inferred schema and the final write is materialized against the resulting registry-derived contract. Public in-memory readers do not accept a separate schema contract.

Separating shape discovery from scalar statistics keeps list and struct decisions stable across messy inputs. If one row has an object and another row has a scalar at the same field, the structural shape wins and the scalar is wrapped under scalar_object_key (default_key by default). If one row has a list and another row has a scalar at the same field, the list shape wins and the scalar is wrapped as a single list element.

The same wrapping heuristics are also applied during registry merge before a file converter creates a versioned field. The merge is conservative: it only makes the current source fit an existing canonical shape when that shape can still read already-written files. An existing list can absorb a singleton value by wrapping it as one list element, and an existing struct can absorb a scalar by placing it under scalar_object_key. An existing struct is not promoted to a list during incremental merge; that would change the Parquet repeated-field layout and still creates a versioned field.

Scalar inference is conservative:

• Nulls do not choose a type by themselves.
• Empty objects ({}) are treated as nulls, preventing unsupported childless Parquet structs while preserving typed structs when other rows contain fields.
• Boolean JSON values infer bool.
• Numeric JSON values infer int64 or float64.
• Strings can infer booleans, integers, floats, timestamps, dates, or times when the configured token and parser options match.
• Mixed scalar kinds fall back to string.
• Objects or arrays observed where a scalar is required are stringified.

Lists of scalars and lists of structs are supported, including repeated fields inside list-of-struct elements such as authors: list<struct<image_auth: list<string>>>. Scalar ambiguity inside a list-of-struct resolves at the nested field; for example, mixed string and integer-looking authors[].id values infer authors[].id: string while keeping authors as a typed list of structs. Arrays whose direct element is another array, such as list<list<int64>>, fall back to list<string>.

Embedded Schema Registry Columns

The streaming file converters can append repeated scalar metadata columns with constant_columns. This is useful when the output file itself must carry metadata instead of writing a parallel table or sidecar object.

result = ss.to_parquet(
    "raw/events.jsonl",
    "silver/events.parquet",
    constant_columns={
        "job_id": "daily-2026-01-09",
        "environment": "silver",
    },
)

The same option is available on to_csv, to_jsonl, and to_parquet. Values must be scalar values such as strings, numbers, booleans, or None; nested objects and arrays are rejected because the columns are meant to be stable per-file metadata.

For schema-drift handling, file converters always run the registry merge and generated file-metadata column planning, append schema_registry, schema_drifts, source_file, and ingestion_timestamp columns, and writes the final file with the merged strict schema. The feature is sink-independent: the same native planning path is used for CSV, JSON Lines, and Parquet outputs.

To keep large schemas and repeated metadata from inflating file size, schema_registry, schema_drifts, source_file, and ingestion_timestamp are written only on the first output row of each file; the same columns are null on the remaining rows. ingestion_timestamp is generated by the native metadata stream when that first output batch is materialized, so each file in a date-range loop records its own processing timestamp. The BigQuery example fetches the latest registry with WHERE schema_registry IS NOT NULL, so one non-null registry row per file is enough for incremental state.

result = ss.to_parquet(
    "raw/events.jsonl",
    "silver/events.parquet",
    field_name_policy="lower_snake",
    schema_registry=latest_registry_from_previous_file,
    schema_drift_date="2026-01-09",
)

The returned Result carries the registry metadata generated by the converter, so a pipeline can pass it directly into the next date or shard:

next_registry = result.schema_registry
registry_json = result.schema_registry_json
drifts = result.schema_drifts
drifts_json = result.schema_drifts_json

The BigQuery range-prefix example, examples/example_07/07_gcs_jsonl_to_silver_parquet_range_prefix.py, uses this result metadata for a single-writer daily incremental pipeline:

• schema_registry stores the latest known clean output schema as canonical_schema and the source-shape variants that should route to fields such as sentences, sentences_v2, or sentences_v3. Each entry in a source path's versions list includes is_most_compatible_current_version; exactly one version is marked true and represents the preferred target for newly processed values.
• schema_drifts stores the drift events observed for the current output file, including newly added fields and new versioned fields generated for incompatible shapes.

On each day, the example fetches the latest schema_registry from the existing BigQuery external table through Arrow ADBC, then calls ss.to_parquet. The fetch orders by the first-row ingestion_timestamp, then by schema_generation, so random-date reprocessing can make a newly written older partition become the latest schema state without rewriting later partitions. The converter handles schema inference, native registry merge, metadata-column planning, and strict final writing under the hood. This keeps each daily file compatible with the table without scanning or rewriting a whole month.

The registry is the source of truth for incremental schema state. BigQuery schema probing is not used as schema state in the registry-backed pipeline. If an existing table has no embedded registry with canonical_schema, run additive mode once to bootstrap a fresh registry, or rebuild the table from a known registry-backed output.

Before creating a new versioned field, the native registry merge first tries the same scalar/container reconciliation used inside one inferred batch:

• If the current canonical field is sentences: list<struct<...>> and a later file sends a single sentences: struct<...> value, no sentences_v2 is created. The singleton value is wrapped as one list element and written to sentences.
• If the current canonical field is details: struct<...> and a later file sends a scalar details value, no details_v2 is created. The scalar is written under details.default_key by default, adding that nullable child if needed.
• If the current canonical field is sentences: struct<...> and a later file sends sentences: list<struct<...>>, the shapes are not safely reconcilable without changing the existing Parquet repeated-field layout. The merged schema keeps both columns:

sentences: struct<...>
sentences_v2: list<struct<...>>

With field_name_policy="lower_snake", the native materializer can route the same dirty source key to the most compatible versioned sibling. List-shaped versions are preferred over scalar or struct versions because a single value can be wrapped as one list element, while a scalar or struct version cannot represent an array. With the schema above, array-shaped sentences values fill sentences_v2. If sentences_v2 is marked as the most compatible current version, later singleton object values can also be wrapped into sentences_v2; sentences remains null for those rows.

Max Depth Enforcement

Depth enforcement uses two independent limits because Arrow and Parquet/BigQuery count nested data differently:

• arrow_max_depth defaults to 32. It counts Arrow container depth: struct and list containers count, while scalar leaves and top-level field wrappers do not.
• parquet_max_depth defaults to 15. It counts Parquet/BigQuery RECORD depth: struct containers count, while list containers, scalar leaves, and top-level field wrappers do not.

The sanitizer flattens a named field when keeping that field's full nested value would exceed either limit. With the default field_name_policy="lower_alpha", <name>_flattened is emitted without the underscore, for example payloadflattened. With field_name_policy="preserve", the output name keeps the _flattened suffix. The flattened value is stored as a string.

Depth examples:

Use arrow_max_depth as a defensive complexity limit for Arrow/Parquet container nesting. Use parquet_max_depth=15 when the output Parquet will be read by BigQuery external tables, where the practical limit is nested RECORD depth rather than physical list wrapper depth.

The reported Result.stats["arrow_schema_depth"] and Result.stats["parquet_schema_depth"] use the same counting rules as the enforcement options.

Memory Safety Measures

The sanitizer is designed to process large local files and PyArrow filesystem URI inputs without requiring the whole clean dataset to live in Python memory.

• File-to-file converters stream sanitized batches directly to the output file. Result.clean_data is None for converters, so the clean table is not materialized in memory.
• PyArrow filesystem file inputs are opened as seekable streams. CSV, JSON, JSON Lines, NDJSON, and XML URI inputs are not copied to a temporary file; their bytes are read by the same chunked native scanner used for local files.
• PyArrow filesystem outputs are opened with pyarrow.fs.open_output_stream. CSV, JSON Lines, and Parquet converters write incrementally to that stream instead of staging the full output in a local temporary file.
• CSV, JSON, JSON Lines, and NDJSON readers use read_chunk_bytes to bound input chunks while scanning.
• XML without xml_row_tag is parsed into a native document tree before row emission, so batch_memory_limit_bytes limits the accumulated document size before the tree is built.
• XML with xml_row_tag streams matching direct child elements. The scanner reads bounded chunks, discards completed row slices, and raises SchemaSanitizerResourceError if the active XML buffer exceeds batch_memory_limit_bytes.
• Local and PyArrow filesystem folder readers (read_json_folder and read_xml_folder) list direct child files only, then compact one source document at a time into a local temporary JSON Lines or XML stream. The temp file is the bridge that lets many single-document files reuse the normal streaming sanitizer pipeline without building one large Python object.
• Folder temp streams contain only the compacted input representation, not the final clean dataset. With batch_memory_limit_bytes, each source document is checked before it is decoded and added to that stream. If a PyArrow filesystem does not report a child file size, the child is read in bounded chunks and the reader stops at batch_memory_limit_bytes + 1 bytes before raising SchemaSanitizerResourceError.
• Folder temp files are deleted when the read finishes, and partially written temp files are deleted if compaction raises an exception. If the Python process is killed externally, for example with SIGKILL, the operating system may not give schema-sanitizer a chance to run that cleanup.
• Supported Parquet inputs are decoded by PyArrow into record batches and fed to the native Arrow C Stream direct path. If a Parquet schema contains an unsupported Arrow shape, schema-sanitizer falls back to an incremental Parquet-to-JSONL bridge instead of staging a full conversion file.
• XML DTD and entity declarations are rejected. The XML frontend does not load external entities or expand document-defined entities.
• batch_memory_limit_bytes maps to the native per-batch memory_limit_bytes budget. It reduces inference and output batch sizes instead of changing the final schema.
• For already-resident Python inputs, batch_memory_limit_bytes is enforced as a preflight resource guard. If the Python payload is already larger than the configured limit, the call raises SchemaSanitizerResourceError before native ingestion starts.
• arrow_max_depth and parquet_max_depth cap nested expansion. Values beyond those limits are flattened to strings, preventing unbounded container nesting from creating very wide or deeply nested Arrow/Parquet schemas.
• Native parsing and materialization use owned streams, arenas, and Arrow C Data resources that are closed when the Result, stream, or sink is closed or dropped. Table-producing readers force stream materialization and close native resources before returning.

Configured resource-limit failures raise SchemaSanitizerResourceError and include limit_name="memory_limit_bytes" in their detail payload when available. True allocator failures are reported separately as SchemaSanitizerOutOfMemoryError.

Large File Tuning

Large file processing is a tradeoff between peak memory and throughput. Smaller batches and read chunks reduce the chance of an operating system or container OOM kill, but they increase overhead and can make conversion slower.

For multi-GB JSON Lines or NDJSON inputs, start with conservative settings:

import schema_sanitizer as ss

result = ss.to_parquet(
    "gs://raw-bucket/events/date=2026-01-01/events.jsonl",
    "gs://silver-bucket/events/date=2026-01-01/events.parquet",
    input_format="jsonl",
    schema_mode="strict",
    schema_registry=latest_registry,
    on_error="emit_null_row",
    batch_memory_limit_bytes=64 * 1024 * 1024,
    read_chunk_bytes=256 * 1024,
)

For non-local PyArrow filesystem input URIs such as gs://..., file converters first stage the source object to a temporary local file using read_chunk_bytes, then run registry inference and strict writing against that local staged copy. This keeps the required two-pass schema workflow intact while avoiding a second remote download. The staged copy is disk-backed, not memory-backed, and is deleted when conversion finishes or fails.

The same settings map naturally to CLI/example scripts:

python examples/example_07/07_gcs_jsonl_to_silver_parquet_range_prefix.py \
  --schema-mode strict \
  --on-error emit_null_row \
  --batch-memory-limit-bytes 67108864 \
  --read-chunk-bytes 262144

Use this tuning guide:

For Parquet inputs, check result.stats["direct_arrow_input"]. A value of 1 means the file used native Arrow C Stream ingestion. A value of 0 means the input was not Parquet or the Parquet schema used the fallback bridge. The fallback is still incremental, but it is usually slower because rows pass through JSON Lines serialization before native cleaning.

For first-run schema discovery, avoid inferring from the largest file when possible. Infer from a smaller representative file or date, write the first registry-bearing Parquet file, then fetch that embedded registry with ADBC and pass it into later converter runs. Additive registry merging must still scan the current source for new fields and can use more memory when the data has many dynamic keys or shape variants.

If a process is reported simply as Killed, that usually means the operating system or container stopped it for memory pressure before Python could raise a typed exception. Measure the real peak resident set size with:

/usr/bin/time -v python your_script.py ...

batch_memory_limit_bytes is a best-effort native batch budget, not a hard cap on total process RSS. Leave headroom for Arrow arrays, Parquet encoding, GCS buffers, Python objects, and allocator fragmentation.

When one file is still too large for the available machine, split work by date, hour, or file shard and write multiple Parquet files under the same partition prefix. BigQuery external tables can read a wildcard or prefix of Parquet files, so several smaller part-*.parquet files are usually safer than one very large conversion.

PyArrow Filesystem Integration

When PyArrow is installed, every file reader and file-to-file converter can use pyarrow.fs URI strings. This covers read_csv, read_json, read_json_folder, read_jsonl, read_xml, read_xml_folder, read_parquet, to_csv, to_jsonl, and to_parquet. Supported URI input extensions include csv, json, jsonl, ndjson, xml, parquet, and pq. Supported URI converter output extensions include csv, jsonl, and parquet.

For normal local files, prefer a regular path:

events = ss.read_jsonl("/home/user/data/events.jsonl")

Regular local paths are the simplest and usually best choice for local disk access. They avoid PyArrow URI parsing and filesystem dispatch.

file:// is PyArrow's local-filesystem URI scheme. On Linux and WSL, absolute local paths use three slashes: file:///home/user/data/events.jsonl. That URI points to the same file as /home/user/data/events.jsonl, but it is opened through pyarrow.fs.LocalFileSystem. Use it when you specifically want to test the PyArrow filesystem route or when your code passes filesystem URIs consistently across local and cloud storage. Do not write file://home/user/...; that form has home in the URI host position instead of being an absolute local path.

Common URI forms:

Cloud URI support depends on the installed PyArrow build and the normal provider credentials/configuration available to PyArrow.

import schema_sanitizer as ss

events = ss.read_jsonl("s3://raw-bucket/events/2026-06-12.jsonl")
assets = ss.read_parquet("gs://raw-bucket/assets/2026-06-12.parquet")
daily_events = ss.read_json_folder("s3://raw-bucket/events/2026-06-12/")

ss.to_parquet(
    "s3://raw-bucket/events/2026-06-12.jsonl",
    "gs://clean-bucket/events/2026-06-12.parquet",
)

URI file inputs are opened as seekable PyArrow files. CSV, JSON, JSON Lines, NDJSON, and XML bytes are fed directly to the native chunk scanner. Supported Parquet files are decoded with pyarrow.parquet into Arrow batches and fed to the native direct Arrow path; unsupported Parquet schemas fall back to incremental JSON Lines conversion. No single-file URI input is copied to a temporary file by schema-sanitizer.

Folder URI inputs are listed with non-recursive pyarrow.fs.FileSelector. read_json_folder filters direct .json child files and read_xml_folder filters direct .xml child files. The matching children are sorted by filename, then compacted one document at a time into a local temporary stream before the normal sanitizer pipeline reads that stream.

URI outputs are opened with pyarrow.fs.open_output_stream. CSV and Parquet writers stream Arrow batches to that output stream, and JSON Lines writes UTF-8 bytes incrementally. The output URI is not staged through a local temporary file.

Supported Inputs

Supported inputs are intentionally file-oriented:

• Normal local file paths for read_csv, read_json, read_jsonl, read_xml, read_parquet, to_csv, to_jsonl, and to_parquet.
• PyArrow filesystem file URI strings for the same single-file readers and converters when PyArrow is installed and can open the URI.
• Normal local folders for read_json_folder and read_xml_folder.
• PyArrow filesystem folder URI strings for read_json_folder and read_xml_folder; folder exploration is non-recursive.
• Already-resident list[dict] rows through read_python.

Unsupported Inputs

Unsupported inputs include raw JSON or XML strings, bytes payloads, opened files, io.BytesIO, io.StringIO, custom reader objects, URLs that PyArrow cannot open as files, and recursive folder scans. Write those inputs to a local file first, or use read_python for in-memory list[dict] rows.

Examples

The examples/ directory contains tutorial notebooks and one cloud pipeline CLI example:

• 01_ingestion_and_core_api.ipynb
• 02_options_and_stats.ipynb
• 03_adapters_and_converters.ipynb
• 04_streaming_large_csv_to_parquet.ipynb
• 05_full_options_catalog_sweep.ipynb
• 06_xml_reading_and_memory.ipynb
• example_07/07_gcs_jsonl_to_silver_parquet_range_prefix.py: GCS date-range JSONL to registry-backed silver Parquet, then creating or replacing the Hive-partitioned external table. In date-range mode it lists source folders before conversion and skips missing logical dates.

Platform Notes

Published PyPI wheels target glibc-based Linux environments (manylinux_2_28). Alpine Linux uses musl, so Alpine users should use a glibc-based Python environment or build from source.

Development

Install the project for local development:

pip install -e .[dev]

Run the tests:

pytest

Build the native core directly with CMake:

cmake -S . -B build/dev -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build/dev

License

schema-sanitizer is licensed under the Apache License 2.0. See LICENSE.