jsonl-diff

A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and top-level JSON arrays by record identity instead of line position, without loading the complete inputs into memory.

jsonl-diff matches records using one or more top-level fields and reports: equal, added, deleted, modified, and tolerated duplicates in each source.

It is useful for snapshots, ETL validation, migrations, exports, and CI checks.

For line-level, field-level, JSON Patch, or visual diffs, use a tool designed for those purposes instead.

Installation

pip install jsonl-diff

Optional speedups

For large datasets, install the optional msgspec accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding and record canonicalization, speeding up large diffs by roughly 2x:

pip install "jsonl-diff[speedups]"

It is used automatically when available — no configuration or code changes. When msgspec is absent, the pure-Python path runs instead and produces identical results.

Quick start

Given:

{"id":1,"name":"Ada"}
{"id":2,"name":"Grace"}

and:

{"id":2,"name":"Grace Hopper"}
{"name":"Ada","id":1}
{"id":3,"name":"Linus"}

compare them by id:

jsonl-diff old.jsonl new.jsonl --key id

Records:
  equal:     1
  added:     1
  deleted:   0
  modified:  1
  OLD duplicates:  0
  NEW duplicates:  0

Record order does not matter.

Features

• Single or composite top-level identities.
• Disk-backed comparison for large datasets (architecture).
• Optional JMESPath filtering with --where.
• Configurable duplicate handling with counts and diagnostics: error, first, or last.
• Exact RFC 6901 JSON Pointer ignores.
• Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
• Semantic number comparison using Decimal.
• Optional msgspec accelerator for ~2x faster large diffs (pip install "jsonl-diff[speedups]").
• Deterministic summaries and change iteration.
• Original OLD/NEW physical line numbers.
• Machine-readable JSONL change log with --details (format).
• Local, HTTP/HTTPS, file-like, and supported compressed sources.
• Incremental top-level JSON array parsing with --format json.
• CLI and Python API using the same comparison engine.

CLI

jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
           [--duplicates {error,first,last}] [--missing-key {error,null}]
           [--details FILE] [--quiet]
           [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
           [--format {jsonl,json}]
           old new

Option	Description
old, new	Local path, HTTP/HTTPS source, or - for stdin
--key KEY	Required top-level identity field; repeat or comma-separate for composite keys
--ignore POINTER	RFC 6901 pointer to exclude from content comparison
--where EXPRESSION	JMESPath filter applied to each record
--duplicates POLICY	error (default), or select and report duplicates with first/last
--missing-key POLICY	error (default), or null to treat an absent identity field as null
--details FILE	Write deterministic machine-readable JSONL changes
--schema-diff	Compare observed fields, types, nullability, and requiredness
--schema-ignore	RFC 6901 pointer to exclude from observed-schema profiling
--quiet	Suppress the normal summary
--max-temp BYTES	Best-effort budget for jsonl-diff workspace temporary storage
--format FORMAT	Input format: jsonl (default) or a top-level JSON array with json

Examples:

# Composite identity
jsonl-diff old.jsonl new.jsonl --key country,customerId,type

# Ignore volatile fields
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --ignore /updated_at \
  --ignore /metadata/request_id

# Filter records
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --where 'deleted_at == `null`'

# Tolerate an identity field that is absent from some records
jsonl-diff old.jsonl new.jsonl \
  --key country,customerId \
  --missing-key null

# Detect observed schema drift
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --schema-diff \
  --schema-ignore /metadata

# Compressed input
jsonl-diff old.jsonl.xz new.jsonl.gz --key id

# Top-level JSON arrays (parsed incrementally)
jsonl-diff old.json new.json --key id --format json

Exit codes

Code	Meaning
0	No record, duplicate, or requested observed-schema issues
1	Record differences, tolerated duplicates, or schema changes
2	Input, resource, output, or runtime error
3	Invalid CLI configuration or usage

Python API

from jsonl_diff import ChangeOperation, diff

with diff(
    "old.jsonl.gz",
    "new.jsonl.gz",
    key=("country", "customerId"),
    ignore=("/updated_at",),
    where='country == `"ES"`',
    schema_diff=True,
) as result:
    print(result.summary)

    for change in result.changes(ChangeOperation.MODIFIED):
        print(change.key, change.old_line, change.new_line)

    for change in result.schema_changes():
        print(change.operation, change.path)

diff() returns a disk-backed DiffResult, used as a context manager. Results are streamed lazily through changes() rather than materialized in memory.

The main result models are:

Summary(equal=10, added=2, deleted=1, modified=3)

and immutable Change objects containing the operation, typed identity, and OLD/NEW source locations.

The public exception hierarchy is rooted at JsonlDiffError:

• ConfigurationError
• InputError
• DuplicateKeyError
• ResourceError

See the Python API reference for the full callable signature, result models, Decimal key semantics, and error hierarchy.

Comparison semantics

jsonl-diff is intentionally strict:

• In the default jsonl format, every input line must contain exactly one valid top-level JSON object.
• With --format json, each input must be one top-level JSON array whose elements are the records.
• Blank lines, malformed JSON, NaN, and infinities are rejected.
• Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
• Identity fields must be top-level scalar values (null allowed only as one component of a composite key).
• Identity types remain significant: "1" ≠ 1, true ≠ 1.
• Duplicate identities fail by default; --duplicates first/last select one occurrence, report every discarded occurrence, and return exit code 1.
• Object property order is ignored; array order is significant.
• Numbers are compared by mathematical value: 1, 1.0, and 1e0 are equal.
• Unicode strings are compared without normalization.
• Changes are reported in deterministic identity order.

--where selects which records participate; --key defines identity; --ignore removes fields from content comparison. With --schema-diff, --schema-ignore independently removes fields from observed-schema profiling. See Comparison semantics for the full rules, including duplicate handling, ignore-pointer edge cases, --where evaluation order, and canonical number formatting.

Sources & compression

Supported sources include local paths and HTTP/HTTPS URLs. The Python API also accepts file-like objects. JSON arrays use py-jsonl.open_stream() for the source and ijson for incremental parsing, so compression and remote source handling are shared with JSONL input.

Supported compression:

• gzip
• bzip2
• xz
• Zstandard on Python 3.14

See architecture for the full source/compression support matrix and how sources are delegated to py-jsonl.

Limitations

jsonl-diff does not provide:

• line/position-based diffs
• field-level diffs or JSON Patch
• move/rename detection
• nested identities or automatic key detection
• fuzzy matching or numeric tolerances
• unordered-array comparison
• validation against a declared schema or input repair
• ZIP, database, or cloud-provider inputs
• GUI or HTML reports

It is a dataset reconciliation tool, not a general-purpose visual JSON diff.

Further reading

• Comparison semantics — identity, duplicates, ignores, --where, canonical numbers, determinism.
• Disk-backed architecture — SQLite index, max_temp, cleanup, sources and compression.
• Details JSONL format — machine-readable --details output schema.
• Python API reference — full signature, result models, and error hierarchy.

Development

uv sync --group test --group lint

uv run pytest
uv run ruff check --quiet --output-format=concise .

The test suite covers CLI behavior, identity/canonicalization, validation, details output, sources, compression, temporary limits, and cleanup.

License

See LICENSE.