DataSemver

datasemver on the web — what it does and why, with the output shown rather than described.

Your data changed. DataSemver tells you whether that is a patch, a minor or a breaking release.

DataSemver compares two versions of a CSV, JSON or Parquet dataset, classifies every difference it finds against a configurable rule set, and returns the semantic version bump plus a ready-to-commit changelog entry. It is a CLI first and a Python library second, and it needs no schema registry, no database and no service running.

Install

pip install datasemver              # library and CLI
pip install "datasemver[web]"       # adds the dashboard
pipx install datasemver             # standalone command

Python 3.10 or newer. The package ships typed, so py.typed annotations reach type checkers.

Use it

datasemver diff old.csv new.csv --current-version 1.4.2

╭───────────── DataSemver ──────────────╮
│ Suggested bump: MAJOR                 │
│ 0.0.0 -> 1.0.0                        │
│                                       │
│ old: old.csv (8 rows)                 │
│ new: new.csv (10 rows)                │
╰───────────────────────────────────────╯
                                    Columns
┏━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ column      ┃ status    ┃ type old ┃ type new ┃ nulls         ┃ cardinality ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ country     │ added     │ -        │ string   │ - -> 0.0%     │ - -> 4      │
│ phone       │ modified  │ int64    │ string   │ 0.0% -> 0.0%  │ 8 -> 10     │
│ email       │ modified  │ string   │ string   │ 25.0% -> 0.0% │ 6 -> 10     │
│ legacy_code │ removed   │ string   │ -        │ 0.0% -> -     │ 8 -> -      │
└─────────────┴───────────┴──────────┴──────────┴───────────────┴─────────────┘
                                        Changes
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ severity ┃ rule                      ┃ description                                   ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ MAJOR    │ column_removed            │ Column 'legacy_code' was removed              │
│ MAJOR    │ type_changed_incompatible │ Column 'phone' changed type from int64 to     │
│          │                           │ string                                        │
│ MINOR    │ row_count_increased       │ Row count grew from 8 to 10 (+25.00%)         │
│ MINOR    │ column_added              │ Column 'country' was added                    │
│ PATCH    │ nulls_fixed               │ Column 'email' nulls dropped from 25.0% to    │
│          │                           │ 0.0%                                          │
└──────────┴───────────────────────────┴───────────────────────────────────────────────┘

A removed column and an int64 that became a string make this a breaking release. Without --output, the changelog entry is printed at the end; with it, the entry is prepended to the file you name.

datasemver diff old.csv new.csv --output CHANGELOG.md
datasemver diff old.csv new.csv --json | jq -r '.bump'
datasemver rules examples/lenient_rules.yaml

What it looks at

• Schema — columns added, removed and renamed, dtype changes, nullability.
• Content — row counts, cardinality, mean and standard deviation of numeric columns, mode and category sets of categorical ones.
• Semantics — renames inferred from the similarity of both the column name and its values, so user_name becoming username is one rename rather than a removal plus an addition.

The bump is the strongest severity across every classified change. Changes no rule covers are reported as unclassified and never inflate it.

Formats

Detected by extension: .csv, .csv.gz, .tsv, .tsv.gz, .json, .jsonl, .ndjson, .parquet, .pq, .feather, .arrow, .xlsx and .xlsm.

The delimiter of a .csv is detected from its first lines — comma, semicolon, tab and pipe are recognised, and a character that only appears inside quoted values does not win — while .tsv always uses the tab. Set DATASEMVER_CSV_DELIMITER to skip detection and force one character, the tab written as \t.

Nested JSON objects and the structs of the Arrow-backed formats are flattened with a ., so {"user": {"name": "..."}} is profiled as user.name. Types are inferred for the text formats; Parquet and Feather carry their own schema and are trusted as they stand. A workbook names its sheet after #, as quarterly.xlsx#Q3.

Engines

Every statistic in a profile is an aggregate, and an aggregate does not need the dataset in memory. With pip install "datasemver[duckdb]", --engine duckdb computes them over the file instead of over a dataframe — the same numbers for about half the memory — and --engine duckdb-sketch estimates the quantile grid for a third of that again. Measured on 16M rows: 21.4 s and 3.2 GB becomes 16.1 s and 1.7 GB, or 3.1 s and 1.2 GB.

Databases

A source can be a table instead of a file, with the connection URL naming the database and the fragment naming the table:

pip install "datasemver[sql]"

datasemver diff "sqlite:///snapshots.db#customers_v1" "sqlite:///snapshots.db#customers_v2"
datasemver diff "postgresql://reader:secret@warehouse:5432/analytics#customers" new.csv

SQLite needs no driver; PostgreSQL and MySQL use the ones the sql extra installs. Quote the argument, since # opens a comment in most shells. Passwords are removed before the source reaches a report. Whole tables only for now: no views, no queries, no schema qualification.

Rules

Every severity is a list of rules, evaluated major, then minor, then patch. The first rule that matches a change assigns its severity.

major:
  - column_removed
  - type_changed_incompatible
  - row_count_decrease_greater_than: 20

minor:
  - column_added
  - row_count_decreased

patch:
  - nulls_fixed
  - minor_stat_change

Pass it with --rules custom.yaml, and check how it was parsed with datasemver rules custom.yaml. Threshold rules pair with their plain counterpart in a lower severity, which then acts as the fallback. Unknown rule names and severities are errors, not silent no-ops.

Python API

from datasemver import analyze

report = analyze("old.csv", "new.csv", current_version="1.4.2")

print(report.bump)          # Severity.MAJOR
print(report.next_version)  # 2.0.0

for item in report.classified:
    print(item.severity, item.rule, item.change.description)

analyze_schemas() takes two already loaded profiles, so dataframes from anywhere can be compared without touching the filesystem:

import pandas as pd
from datasemver.core.analyzer import analyze_schemas
from datasemver.formats.loader import schema_from_frame

report = analyze_schemas(
    schema_from_frame(pd.read_sql(query, engine), "warehouse@yesterday"),
    schema_from_frame(pd.read_sql(query, engine), "warehouse@today"),
)

Also in the box

• A web dashboard — FastAPI backend, no-build frontend — under the web extra, run with uvicorn datasemver_web.backend.main:app. It is a local tool with no authentication: keep it on the loopback interface.
• A GitHub Action that analyses the datasets a pull request touches and posts the suggested bump as a comment, rewritten on each push.
• Two ready-made rule profiles, strict and lenient, and a full catalogue of rules, metrics and thresholds.

Those, the source, the changelog and a Spanish edition of this page live in the project repository, linked from this page's sidebar.

Security

Reading a dataset parses it. CSV and JSON go through pandas and the standard library, which do not execute file content; Parquet goes through pyarrow, and the dependency floor is pyarrow>=23.0.1 because earlier versions carried a critical code-execution flaw (CVE-2023-47248) triggered by a malicious Parquet file. Do not lower that floor. Rule files are YAML loaded with yaml.safe_load and cannot execute code. The library and the CLI never open a socket, and write nothing unless you pass --output.

License

Apache License 2.0. The licence grants no right to the DataSemver name or marks. Releases up to and including 0.7.0 were published under the MIT licence and remain available under it.