dsjframe

dsjframe is a Python library for accurately reading, writing, and validating Dataset-JSON with Apache Arrow. It supports plain JSON, NDJSON, and DSJC (.json, .ndjson, and .dsjc).

Installation

Install the base package:

pip install dsjframe

Install the optional dependencies to support additional file formats:

pip install dsjframe[file-support]

Python 3.10 or newer is required.

Quick Start

import dsjframe
# Read
table = dsjframe.read_dataset("adsl.json")

# Write
metadata = {
    "datasetJSONVersion": "1.1.0",
    "label": "Subject Level Analysis Dataset",
}

dsjframe.write_dataset(table, "adsl.ndjson", metadata)

# validate
report = dsjframe.validate_dataset("adsl.dsjc")

See example.ipynb for a longer walkthrough.

Metadata for Writing

Dataset-JSON allows extensions, but dsjframe targets the standard structure by default. Unexpected metadata fields are treated as errors.

When writing, metadata is merged in this priority order:

1. Explicit metadata
2. define.xml
3. Embedded Arrow schema metadata
4. readstat_meta
5. Library defaults such as datasetJSONVersion and itemGroupOID

datasetJSONCreationDateTime is filled automatically. In many cases, you can omit most or all of metadata if define.xml, embedded schema metadata, or readstat_meta already provide the required fields. When define.xml is used, metaDataRef is set to define.xml automatically. If you need a different path or reference value, set it explicitly in metadata.

When writing from a pyarrow.Table or pandas DataFrame, column dataType is derived from the actual column type in the frame except for text-backed temporal columns declared as date, time, or datetime. Provided column metadata is still used for fields such as label, length, displayFormat, and keySequence, but it does not override the real data type. Compatible targetDataType values are preserved where allowed, decimal exports are normalized to "targetDataType": "decimal", and targetDataType: "integer" is ignored for text-backed temporal columns so partial dates and other ISO 8601 text values remain text-backed.

Missing Values

dsjframe follows Arrow conventions and represents missing values as null. In practice, especially when Dataset-JSON is used as an XPT replacement, character missing values are often written as "" instead of null. Because Arrow is better aligned with nulls for missing data, empty strings in string-like columns are converted to null by default when reading. If you need to preserve the distinction between null and an empty string, you can disable that conversion with an option.

More Than Metadata Validation

Because Dataset-JSON is a text format, its readability and editability are often treated as advantages. Those same properties can also make type drift, malformed values, or accidental edits harder to catch.

To address that, dsjframe includes strict validation for both metadata and row data.

In addition to JSON Schema validation for metadata, dsjframe also checks combinations of dataType and targetDataType, row structure, value conversion, record counts, and consistency between file content and file extension. String-backed temporal values are also checked as ISO 8601 text. Reduced-precision partial dates such as YYYY and YYYY-MM are accepted for temporal text values.

Because datetime columns can contain date-only values in practice, the time component of a complete YYYY-MM-DD value with targetDataType: "integer" is treated as 00:00:00.

The bundled JSON schema is used without modification. The Dataset-JSON v1.1 NDJSON schema could not be used as-is, so dsjframe uses a slightly modified version.

DSJC Support

DSJC is treated as gzip-compressed NDJSON.

The current implementation follows the available examples rather than the still-evolving compressed Dataset-JSON v1.1 wording. The specification may change, so DSJC behavior may need to change with it.

API Reference

read_dataset(source, *, as_pandas=False, out_metadata=False, empty_to_null=True, partial_as_null=False)

Read Dataset-JSON, NDJSON, or DSJC from a path, bytes object, or file-like object. It returns a pyarrow.Table by default, or a pandas DataFrame when as_pandas=True. Set out_metadata=True to also receive pyreadstat-compatible metadata. By default, empty strings in string-like columns are converted to null on read; pass empty_to_null=False to keep them as empty strings. By default, reduced-precision values in date or datetime columns with targetDataType: "integer" raise a conversion error because Arrow temporal types cannot preserve their precision. Pass partial_as_null=True to replace valid YYYY and YYYY-MM values with null instead. Malformed dates still raise an error, and validate_dataset remains strict.

write_dataset(frame, destination, metadata=None, *, output_format=None, define_xml=None, readstat_meta=None, compression_level=6, json_indent=2)

Write a pyarrow.Table or pandas DataFrame as JSON, NDJSON, or DSJC. The output format is inferred from the file suffix unless you pass output_format explicitly. Metadata can come from metadata, define.xml, Arrow schema metadata, or readstat_meta.

detect_format(source)

Inspect a source and return a lightweight report describing the detected format. This is useful when you want to check the input before reading or validating it. The ok value reflects only a lightweight check of the suffix, compression header, and first one or two non-empty lines. It does not mean that the complete dataset is valid; use validate_dataset for metadata and row validation.

validate_dataset(source, *, validate_metadata=True, validate_data=True)

Validate a dataset and return a diagnostic report. You can validate only metadata, or validate both metadata and row data.

build_metadata(*, frame=None, metadata=None, define_xml=None, readstat_meta=None)

Merge metadata from the available sources and return a validated Dataset-JSON metadata dictionary. Use this when you want to inspect or prepare export metadata before writing rows.

build_schema(source_or_dataset)

Build a pyarrow.Schema from a Dataset-JSON source or from a Dataset-JSON payload dictionary. Use this when you need the inferred Arrow schema without loading the dataset rows.

Errors

The tables below list the stable, library-specific validation messages exposed by the public API. Operating-system, filesystem, and third-party exceptions are outside this list. <...> denotes a placeholder filled with the actual name, path, or value. Validation failures from read_dataset, write_dataset, build_metadata, and build_schema raise an exception. detect_format and validate_dataset normally return the corresponding text in the report message field.

Library-specific exception types are DataValidationError, UnsupportedTypeError, DefineXmlError, and MissingDependencyError. build_schema raises TypeError when passed a columns list instead of a Dataset-JSON document.

Input and format detection

Metadata and row validation on read and validate

Export metadata validation in build_metadata and write_dataset

The <field> must be a string pattern applies to itemGroupOID, name, label, fileOID, studyOID, metaDataVersionOID, metaDataRef, and originator.

Row conversion, frame handling, and output on write

Utility API and optional dependency errors

Define-XML

Packaging and internal schema loading

These indicate a broken installation rather than bad user input.

Development

See AGENTS.md and CHANGELOG.md.

License

This project is licensed under the AGPL.

Code from this repository that is provided to an AI system, and code produced from that input, is treated as a derivative work. Redistributing an AI-based reimplementation without preserving this license is considered copyright infringement.

This project is developed and maintained independently.
To help keep it maintained, consider supporting it through sponsorship or by engaging me for contract work.
Contact: info@knworx.com.

tests/data/official_example and dsjframe/schema are derived from https://github.com/cdisc-org/DataExchange-DatasetJson (Copyright (c) 2022 cdisc) under the MIT license.