lx

A pipe-friendly Swiss-army knife CLI for data manipulation.

Design principle: lx manipulates data structure. It does not query.

Named-key extraction (pluck, get, select) is only provided as a convenience when the target is unambiguously a Python dict. If you need traversal, indexing, conditionals, or expressions — use jq, yq, xsv, or another specialist tool. lx is meant to be used alongside them, not as a replacement.

Every command defaults to reading from stdin and writing to stdout, so lx composes naturally with other Unix tools.

Status: Early development (v0.2.0). APIs and command names may change.

---

Features

• JSON – pretty-print, minify, validate, sort keys (recursively or by array key), reverse key or array order, deduplicate, rename keys, select keys, move keys, pluck values, convert to JSON Lines
• JSON Lines – count, head, tail, validate, sort and reverse-sort by key, deduplicate, rename keys, select keys, move keys, pluck values, shuffle, sample, convert to a JSON array
• CSV – sort and reverse-sort by column name or index, select or remove columns, deduplicate, rename columns, move columns, validate structure, count rows, head, tail, shuffle, sample
• Encoding – detect (with confidence scores), check against an expected encoding, recode, add or strip BOM

Built on orjson for fast JSON handling and charset-normalizer for encoding detection.

---

Installation

lx is designed to be installed and run via uv.

Run without installing

uvx lx --help
uvx lx json pretty data.json

Install as a tool

uv tool install lx-tools
lx --help

uv handles the virtual environment, Python version, and PATH automatically.

From source (development)

git clone https://github.com/enzo-agosta/lx-tools.git
cd lx-tools
uv sync

---

Usage

lx is organized into sub-commands by data format. Every command accepts file arguments or reads from stdin and writes to stdout by default.

JSON

# Pretty-print (2-space indent)
lx json pretty data.json

# Minify
lx json minify data.json > data.min.json

# Validate and pass through unchanged
lx json validate data.json | lx json pretty

# Sort top-level keys (or array elements)
lx json sort messy.json

# Sort keys recursively in every nested object
lx json sort --recurse messy.json

# Sort an array of objects by a top-level key
lx json sort --key age users.json

# Reverse top-level keys or array order
lx json reverse '{"a":1,"b":2,"c":3}'

# Reverse-sort an array of objects by key
lx json reverse --key age users.json

# Deduplicate a JSON array (keeps first occurrence)
lx json dedup '[1,2,1,3]'

# Rename a top-level key
lx json rename --old name --new full_name data.json

# Select only specific keys (in given order)
lx json select --keys name,age user.json

# Reorder keys to front or back
lx json move --keys b,a --front '{"a":1,"b":2,"c":3}'

# Extract a single value
lx json pluck --key name user.json

# Convert a JSON array to JSON Lines
cat array.json | lx json to-jsonl

JSON Lines

# Count valid, non-empty lines
lx jsonl count data.jsonl

# First/last N lines (defaults to 10)
lx jsonl head -n 5 data.jsonl
lx jsonl tail -n 20 data.jsonl

# Validate every non-empty line
lx jsonl validate data.jsonl | lx jsonl sort --key name

# Sort lines by a top-level key
lx jsonl sort --key timestamp data.jsonl

# Reverse-sort by key (missing keys sort last)
lx jsonl reverse --key timestamp data.jsonl

# Deduplicate lines (keeps first occurrence, normalizes whitespace)
lx jsonl dedup data.jsonl

# Rename a key in every object
lx jsonl rename --old name --new full_name data.jsonl

# Select only specific keys from each line
lx jsonl select --keys name,age users.jsonl

# Reorder keys to front or back on each line
lx jsonl move --keys b,a --front data.jsonl

# Extract a single value from each object (skips missing by default)
lx jsonl pluck --key user_id data.jsonl

# Convert JSON Lines to a JSON array
lx jsonl to-json data.jsonl

# Shuffle randomly (use --seed for reproducibility)
lx jsonl shuffle --seed 42 data.jsonl

# Sample N lines without replacement
lx jsonl sample -n 100 --seed 42 data.jsonl

CSV

# Sort by column name (requires --header)
lx csv sort --header --name Age data.csv

# Sort by zero-based column index
lx csv sort --index 2 data.csv

# Reverse-sort by column name
lx csv reverse --header --name Age data.csv

# Deduplicate data rows (--header preserves header row)
lx csv dedup --header data.csv

# Validate structure (checks consistent column counts)
lx csv validate data.csv

# Validate with header checks and no empty cells
lx csv validate --strict data.csv

# Rename a column by header name (first row is always header)
lx csv rename --old Name --new FullName data.csv

# Reorder columns by name (first row is always header)
lx csv move --names Age,Name --front data.csv

# Reorder columns by zero-based index
lx csv move --indices 2,0 --back data.csv

# Keep only specific columns (first row treated as header)
lx csv select --names Name,Email data.csv

# Keep columns by index
lx csv select --indices 0,2,4 data.csv

# Drop columns by name
lx csv remove --names Password,InternalID data.csv

# Count rows (--header excludes the first row)
lx csv count --header data.csv

# First/last N data rows (preserves header)
lx csv head -n 5 --header data.csv
lx csv tail -n 5 --header data.csv

# Shuffle or sample rows (preserves header)
lx csv shuffle --seed 42 --header data.csv
lx csv sample -n 100 --seed 42 --header data.csv

Encoding

# Detect encoding
lx encoding detect file.txt

# See all candidates with confidence scores
lx encoding detect --all --long file.txt

# Check against an expected encoding (passes through on match)
lx encoding check --expected utf-8 file.txt

# Recode from one encoding to another
lx encoding recode --from latin1 --to utf-8 file.txt

# Add a BOM (strips any existing BOM first)
lx encoding add-bom --encoding utf-16-le file.txt

# Strip BOM
lx encoding strip-bom file.txt

Flags you will reach for often

• --strict – on sort, reverse, select, move, and pluck commands: raise an error instead of silently skipping rows or objects that lack the requested key. On csv validate: equivalent to --header --no-empty.
• --seed – on shuffle and sample commands: make random output reproducible.
• --raw-lines – on jsonl head, tail, shuffle, and sample: skip JSON validation and treat the input as plain text lines.
• --recurse – on json sort: sort keys recursively inside nested objects.
• --header – on CSV commands: treat the first row as a header and preserve it through the operation.
• --no-empty – on csv validate: error if any cell is empty.

Piping examples

# Stable diff between two JSON files
lx json sort --recurse a.json a.sorted.json
lx json sort --recurse b.json b.sorted.json
diff a.sorted.json b.sorted.json

# Chain multiple operations
curl -s https://api.example.com/data.json \
  | lx jsonl sort --key id \
  | lx jsonl to-json \
  | lx json pretty \
  | lx encoding add-bom --encoding utf-8 --output sorted.json

# Sample CSV, then select columns
lx csv sample huge.csv --header -n 1000 | lx csv select --names colA,colB --output subset.csv

# Query YAML with yq, then format with lx
cat config.yml | yq '.services.web' | lx json pretty --output config.web.json

# Detect and recode in one shot
lx encoding detect --all --long legacy.txt
lx encoding recode --from windows-1252 --to utf-8 legacy.txt legacy.utf8.txt

---

Architecture

The project separates CLI concerns from reusable library logic:

src/lx_tools/
├── cli/        # cyclopts-based command-line interface
│   ├── csv.py
│   ├── encoding.py
│   ├── json.py
│   └── jsonl.py
└── lib/        # Pure functions and core business logic
    ├── csv.py
    ├── encoding.py
    ├── json.py
    └── jsonl.py

• CLI modules handle argument parsing, I/O wiring, and error formatting.
• Library modules are importable and can be used independently in other Python projects.

---

Development

# Sync dependencies
uv sync

# Run the CLI in the project environment
uv run lx --help

# Build and publish (CI handles releases via Trusted Publishing)
uv build

---

Contributing

Contributions are welcome! Because the project is still in its early stages, opening an issue to discuss larger changes before submitting a PR is appreciated.

---

License

MIT