sortdx 0.2.0

Sort CSV, TSV, JSONL, and text files of any size, with gzip and zstandard support

sortdx

A sorting library and command line tool for Python that handles CSV, TSV, JSONL, and text files, including gzip and zstandard compressed files. Large files are sorted on disk through chunking and a k-way merge, so memory usage stays bounded. The core library has no required dependencies and ships full type information.

Installation

pip install sortdx

The core library has no required dependencies. The command line interface and extra features (encoding detection, flexible date parsing) need the optional dependencies:

pip install "sortdx[full]"

Quick start

import sortdx

# Sort a CSV by a numeric column
sortdx.sort_file("data.csv", "sorted.csv", keys=[sortdx.key("age", "num")])

# Sort a JSONL file by timestamp
sortdx.sort_file("logs.jsonl", "sorted.jsonl", keys=[sortdx.key("timestamp", "date")])

# Sort in-memory data
data = [{"name": "Bob", "age": 30}, {"name": "Alice", "age": 25}]
result = list(sortdx.sort_iter(data, keys=[sortdx.key("age", "num")]))

Sort keys

Create sort keys with sortdx.key():

# String sorting (default, case-insensitive)
sortdx.key("name")

# Numeric sorting
sortdx.key("price", "num")

# Date sorting
sortdx.key("created_at", "date")

# Natural sorting: file2 before file10
sortdx.key("filename", "nat")

# Descending order for one key
sortdx.key("salary", "num", desc=True)

# Locale-aware string collation
sortdx.key("name", "str", locale_name="fr_FR.UTF-8")

Columns are addressed by name for dict rows (CSV, JSONL), by integer index for list rows, and None sorts plain text lines as whole items.

Data types

Type	Description
str	Case-insensitive text sorting (default)
num	Numeric sorting for integers and floats
date	Date and time sorting; ISO 8601 always works, other formats need python-dateutil
nat	Natural sorting where embedded numbers compare by value

Empty and unparseable values sort first within their type.

API

sortdx.sort_file(input_path, output_path, keys, ...)

Sort a file into another file.

Parameter	Description
input_path	Input file path
output_path	Output file path
keys	List of sort keys
memory_limit	Memory budget such as "512M" or "2G"; larger inputs are sorted on disk (default threshold: 100M)
reverse	Reverse the entire sort order
unique	Column whose values must be unique in the output
stats	Return a SortStats object

stats = sortdx.sort_file(
    "huge.jsonl.gz",
    "sorted.jsonl.gz",
    keys=[sortdx.key("timestamp", "date")],
    memory_limit="512M",
    stats=True,
)
print(f"Processed {stats.lines_processed} lines in {stats.processing_time:.2f}s")

sortdx.sort_iter(data, keys, ...)

Sort an iterable in memory and return an iterator. Accepts the same reverse and unique arguments.

Multi-key sorting

sortdx.sort_file("employees.csv", "sorted.csv", keys=[
    sortdx.key("department"),
    sortdx.key("salary", "num", desc=True),
])

Command line

# Sort a CSV by price, then name; write to a file
sortdx data.csv -o sorted.csv -k price:num -k name:str

# Sort a compressed JSONL file with a memory budget
sortdx logs.jsonl.gz -o sorted.jsonl.gz -k timestamp:date --memory-limit=512M

# Natural sort of a text file to stdout
sortdx filenames.txt --natural

# Keep one row per user id, newest first
sortdx users.jsonl -o unique.jsonl -k created_at:date --reverse --unique=user_id

Key specifications use the format column:type[:option=value], for example price:num, name:str:locale=fr_FR.UTF-8, or rank:num:desc=true.

Supported formats

Format	Extensions
CSV	.csv (delimiter auto-detected)
TSV	.tsv, .tab
JSON Lines	.jsonl, .ndjson, .json
Plain text	.txt and anything else
Compression	.gz, .gzip, .zst, .zstd (zstandard requires pip install zstandard)

Development

git clone https://github.com/Okymi-X/sortdx.git
cd sortdx
pip install -e ".[full,dev]"
pytest

See CONTRIBUTING.md for the full set of checks (formatting, lint, type checking) and the contribution guidelines.

Changelog

Release history is documented in CHANGELOG.md.

License

MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome. Please open an issue or submit a pull request on GitHub.