undatum

A command-line tool for data processing and analysis

undatum (pronounced un-da-tum) is a CLI for converting, analyzing, validating, and transforming datasets across many formats. Conversion, statistics, selection, validation and database loads stream records, so file size is not limited by memory; each command page states how much memory the command needs.

Features

• 140+ formats via iterabledata: CSV, JSON, JSON Lines, BSON, XML, XLS/XLSX, Parquet, AVRO, ORC, plus geospatial, lakehouse (Delta/Iceberg/Lance/DuckLake/Paimon), scientific, RDF, log, config, graph, and feed formats. Run undatum formats list to see every supported format and its read/write capabilities.
• Compression support: GZ, XZ, BZ2, ZIP, ZSTD, LZ4, 7Z, Brotli, Snappy, LZO
• Verified repacking: Recompress local streams and native Parquet, ORC and Avro files, or compact offline DuckDB and SQLite databases (undatum repack). Outputs are staged, independently verified and published atomically; optimization keeps the original unless the candidate saves space. Measure savings with --dry-run --json and set optional time, temporary-space and memory budgets
• Multi-cloud I/O: Read and write s3://, gs:///gcs://, and az:///abfs:///abfss:// URIs natively via iterabledata (pip install "undatum[cloud]")
• Databases: Query and dump PostgreSQL, MySQL/MariaDB, SQLite, SQL Server, ClickHouse, MongoDB, and Elasticsearch/OpenSearch (undatum db query, undatum db dump); load files into DuckDB, SQLite, PostgreSQL, MySQL, ClickHouse, SQL Server, MongoDB, and Elasticsearch/OpenSearch (undatum db load, with create-table, replace and upsert where the database supports them)
• Large files: record commands stream with bounded memory; sort, dedup and reverse spill to disk; DuckDB runs SQL and many transforms out of core. Each command page states its memory use (for example uniq and frequency grow with the number of distinct values)
• Unix pipelines: - reads standard input and results go to standard output in the input's format: cat data.csv | undatum sort - --by amount | undatum head - -n 10
• SQL expressions: --where "amount > 100 AND city = 'Berlin'" filters and --add "total = price * qty" adds computed fields on select, convert and other commands, evaluated by DuckDB on typed values
• Partitioned output: undatum convert data.csv out/ --partition-by year,month -O parquet writes Hive-style field=value/ directories that DuckDB, Spark, Athena and BigQuery read
• Automatic detection: Encoding, delimiters (comma, semicolon, tab, pipe), headers, compression and file types
• Data validation: rule files with a built-in library — emails, URLs, dates, E.164 phones, ISO country/currency/language codes, IBAN, UUID, patterns, ranges, unique and references to another file (undatum validate --rules; --list-rules prints the catalogue)
• Quality reports: undatum quality profiles a dataset (empty values, distinct and top values, type conformance), checks it against an expected schema and rules, and fails CI when a threshold is not met (Markdown, HTML or JSON)
• Schema drift: undatum diff --schema and undatum schema-drift report added, removed, retyped and renamed fields across deliveries; --fail-on turns drift into a non-zero exit code
• Machine-readable output: --json on informational commands (count, headers, sniff, stats, schema, diff, validate, ...) prints versioned JSON documents; errors become JSON too, with documented exit codes
• Frictionless Data Packaging: Create, extend, and validate datapackage.json descriptors (undatum package)
• Ad-hoc SQL on files: Run DuckDB SQL over CSV, JSONL, Parquet, and other formats (undatum sql)
• Python SDK: a lazy Dataset API — chained steps build a plan that runs as one DuckDB query when every step has a SQL form
• AI-powered tooling: undatum ai and --autodoc on analyze, schema and doc share iterabledata's providers (OpenAI, Anthropic, Gemini, Azure, OpenRouter, Ollama, LM Studio, Perplexity, OpenAI-compatible endpoints); values in personal-data columns are masked in samples sent to remote providers
• Agent tools & MCP server: one tool per operation for MCP and LangChain; undatum mcp serve also exposes dataset listings, schemas and samples as resources and ships ready-made prompts (mcp 1.x and 2.x)
• Optional TUI and web UI: Explore a bounded sample in the terminal (undatum tui) or a local browser (undatum web)
• Optional Data API: Serve file-backed datasets over HTTP (FastAPI + DuckDB)

Documentation

The full documentation site (Docusaurus) lives in docs/ and is published at datenoio.github.io/undatum.

Source pages: docs/docs/. Changelog: CHANGELOG.md. Contributor workflow: WORKFLOW_GUIDE.md.

Installation

Using uv or pipx (recommended for CLI use)

uv tool install undatum
# or
pipx install undatum

Using pip

pip install --upgrade pip setuptools
pip install undatum

Dependencies are declared in pyproject.toml and will be installed automatically by modern versions of pip (23+), including pyarrow for Parquet.

macOS

brew install pipx && pipx install undatum
# or
uv tool install undatum

Since 1.8.0, release tags also publish PyInstaller single-file binaries (Linux, macOS, Windows) on GitHub Releases; 1.7.0 and earlier are on PyPI only. pipx/uv remain the supported install paths for most users.

A man page ships with the package (man undatum after install, or make man to regenerate man/undatum.1).

Container

Release images are published to GitHub Container Registry:

docker run --rm -v "$PWD:/data" ghcr.io/datenoio/undatum convert data.csv data.parquet

:<version> and :latest hold the core install, :<version>-full and :latest-full add the common extras. A Homebrew tap and a conda-forge package are planned; see the installation guide.

Optional extras

pip install "undatum[extract,api]"

After installation both undatum and the shorter data command are available:

undatum --version
undatum headers data.csv
data headers data.csv   # same thing

Shell completion

undatum --install-completion   # for the shell you run it from (bash, zsh, fish, PowerShell)
undatum --show-completion      # print the script instead of installing it

Requirements

• Python 3.10 or greater (CI tests 3.10–3.13)

Install from source

python -m pip install --upgrade pip setuptools wheel
python -m pip install .

Quick start

# Inspect supported formats
undatum formats list --capabilities

# Convert
undatum convert people.csv people.parquet
undatum convert --tagname item data.xml data.jsonl

# Recompress local files with independent read-back verification
undatum repack data.csv.gz
undatum repack data.parquet --dry-run --json

# Inspect
undatum headers data.jsonl
undatum analyze data.jsonl
undatum stats data.csv
undatum table data.csv --limit 20

# Query and transform
undatum sql "SELECT city, COUNT(*) AS n FROM data GROUP BY city" cities.csv
undatum select sales.csv --where "amount > 100" --add "total = price * qty" --output big.csv
undatum convert sales.csv sales/ --partition-by year -O parquet

# Pipelines: '-' reads stdin; CSV in, CSV out (-O jsonl to change it)
cat data.csv | undatum sort - --by amount --numeric amount | undatum head - -n 5

# Validate, check quality and drift, package
undatum validate data.csv --rules rules.yml
undatum quality data.csv --rules rules.yml --thresholds quality.yml --output report.html
undatum schema-drift deliveries/ --baseline schema.json --fail-on removed,type
undatum package create data.csv --output datapackage.json

# JSON for scripts and agents
undatum count data.csv --json

# Document
undatum ai doc data.csv
undatum doc data.jsonl --format-out markdown --output dataset.md

More first-success paths: quick start and the cookbook.

Recompress and compact files

# Change the outer codec without changing the decoded bytes
undatum repack data.csv.gz data.csv.zst

# Compact offline databases into new files
undatum repack warehouse.duckdb compact.duckdb
undatum repack app.sqlite compact.sqlite

# After closing all SQLite users, compact in place
undatum repack app.sqlite --sqlite-offline

repack accepts local files and retains their data format. It preserves the source when encoding or verification fails and skips a same-representation candidate that does not save space; use --allow-grow to accept it. Replacing an existing SQLite destination requires --sqlite-offline; active database sidecars cause refusal. DuckDB's initial supported profile is limited to 256 MiB input and 1 million rows. See the repack reference for format limits, profiles, resource budgets and the undatum.repack/1 JSON report.

Commands

All commands are available as undatum <command> or via the shorter data alias.

Top-level data commands: convert, repack, extract, analyze, doc, stats, validate, quality, schema, schema-bulk, schema-drift, diff, sql, select, search, mask, plot, tui, web, migrate-script, and the other transform/inspection commands in the CLI reference.

Upgrading: the older names document, profile, ingest and scheme and the old option spellings (--filetype, --outtype, --n, ...) still work with a deprecation warning until 2.0 (use doc, stats, db load, schema --format cerberus). undatum migrate-script scripts/ --write rewrites them in shell scripts, Makefiles, Markdown and pipeline YAML; the migration guide and the changelog list the changed defaults.

Command groups:

undatum convert --help
undatum sql --help

Contributing

Contributions are welcome. See CONTRIBUTING.md and the development docs.

License

MIT License — see LICENSE. Third-party attribution and license terms are in THIRD_PARTY_NOTICES.

Links

• Documentation
• GitHub
• PyPI
• Changelog
• Issue tracker
• iterabledata (streaming I/O engine)