Installation • Quick start • Commands • Formats • Options • Examples • Piping

nail - Lightning-Fast Data Analysis CLI

nail is a high-performance command-line tool for analyzing, transforming, and exploring Parquet, CSV, JSON, JSON Lines, Arrow IPC, and Excel files. Built with Rust, Apache Arrow, and DataFusion.

• Fast — gigabyte-scale datasets in seconds, parallel across all CPU cores.
• SQL-powered — DataFusion under the hood, familiar filter and expression syntax.
• Zero configuration — formats are detected from the file extension, on input and output.
• Composable — every command reads stdin and writes stdout, so nail pipes into itself and other tools.
• Self-contained — a single binary that works offline.

Installation

Prebuilt binary (macOS/Linux — auto-detects your OS and architecture):

curl -fsSL https://raw.githubusercontent.com/Vitruves/nail-parquet/main/install.sh | sh

It installs to /usr/local/bin when writable, otherwise to ~/.local/bin. Set BINDIR to choose the location explicitly, and the installer tells you if the target dir is not on your PATH:

curl -fsSL https://raw.githubusercontent.com/Vitruves/nail-parquet/main/install.sh | BINDIR="$HOME/bin" sh

With Cargo:

cargo install nail-parquet

From source:

git clone https://github.com/Vitruves/nail-parquet
cd nail-parquet
cargo build --release
sudo cp target/release/nail /usr/local/bin/
nail --help

With nix:

nix shell nixpkgs#nail-parquet

Dependencies: macOS — none. Linux — pkg-config and openssl.

Quick Start

nail describe sales.parquet                                    # what is in this file?
nail head sales.parquet -n 5                                   # look at a few rows
nail filter sales.parquet -c "revenue > 1000" -o big.parquet   # keep what matters
nail convert big.parquet -o big.csv                            # hand it to another tool

The package is nail-parquet on crates.io; the executable is nail.

Commands

append        Concatenate multiple datasets
binning       Bin continuous variables into categories
convert       Convert between file formats
correlations  Calculate correlation matrices
count         Count total rows
create        Create new columns with expressions
dedup         Remove duplicate rows or columns
describe      Show global file overview and metadata
diff          Compare two datasets and show differences
drop          Remove columns or rows
fill          Fill missing values
filter        Filter rows by conditions
frequency     Calculate frequency distributions
head          Display first N rows
headers       Display column headers
id            Add unique identifier column
merge         Join two datasets
metadata      Show Parquet file metadata
optimize      Optimize Parquet files for better performance
outliers      Detect outliers in data
pivot         Create pivot tables with aggregations
preview       Preview random N rows
rename        Rename columns
sample        Extract data samples
schema        Display schema information
search        Search for values in data
select        Select specific columns or rows
shuffle       Randomly shuffle rows
size          Show data size information
sort          Sort data by columns with various strategies
split         Split data into multiple files
stats         Calculate descriptive statistics
tail          Display last N rows
transpose     Transpose rows and columns
unique        List distinct rows or per-column value counts
update        Check for newer versions
help          Print this message or the help of the given subcommand(s)

Run nail <command> --help for full usage.

Supported Formats

Formats are detected from the file extension on both input and output, and can be forced with -f/--format.

Both Arrow IPC flavours are read: the file format (ARROW1 magic + footer) and the stream format written by e.g. HuggingFace datasets.save_to_disk. Writes always produce the self-contained file format.

nail head dummy_dataset/data-00000-of-00001.arrow
nail convert dummy_dataset/data-00000-of-00001.arrow -o dummy.jsonl
nail filter events.jsonl -c "status == 'error'" -o errors.arrow

Global Options

Available on all commands:

Examples

Explore a dataset:

nail describe sales.parquet
nail stats sales.parquet -c "revenue,profit" --percentiles "0.5,0.9,0.99"
nail correlations sales.parquet -c "price,volume,discount" --tests t_test
nail frequency sales.parquet -c "category,region"

Clean and enrich:

nail dedup raw.parquet --row-wise -c "id" -o unique.parquet
nail outliers unique.parquet -c "price" --method iqr --remove -o cleaned.parquet
nail create cleaned.parquet --column "margin=(price-cost)/price" -o enriched.parquet

Build an analysis pipeline:

nail optimize raw.parquet -o opt.parquet --compression zstd --sort-by "ts,customer_id" --dictionary
nail binning opt.parquet -c "age" -b "18,25,35,50,65" --method custom --labels "18-24,25-34,35-49,50-64,65+" -o binned.parquet
nail pivot binned.parquet -i "age_binned" -c "category" -l "revenue" --agg sum -o summary.parquet
nail stats summary.parquet --stats-type exhaustive -o summary_stats.json

Reshape and summarize:

nail unique sales.parquet -c "category"                 # distinct values of a column
nail unique sales.parquet -c "category,region" --count  # value counts, most frequent first
nail transpose metrics.parquet --header-column metric -o wide.parquet

Compare versions:

nail diff yesterday.parquet --compare today.parquet --keys "id" --changes-only

Piping (stdin/stdout)

Every command can read from stdin and write to stdout, so you can chain nail with itself or other tools. Use -o - on the producer to stream out, and - as the input on the consumer to read in. The input format is auto-detected.

Streaming to stdout defaults to Parquet, which preserves the full schema (types and nested List/Struct/Map columns) losslessly across the pipe. Use -f csv or -f json when you want text output for other tools.

# chain nail commands (Parquet by default — keeps all types)
nail filter sales.parquet -c "revenue > 1000" -o - | nail sort - -c revenue -o - | nail head - -n 10

# stream as CSV/JSON for other tools
nail select sales.parquet -c "id,revenue" -o - -f csv | grep -v '^0,'
nail sample sales.parquet -n 100 -o - -f json | jq '.revenue'

# read from stdin produced elsewhere
cat data.parquet | nail count -

Pipes that close early (| head, | less) are handled cleanly — no broken-pipe errors.

Performance Tips

• Prefer Parquet over CSV for analytical workloads.
• Scope operations with -c regex patterns.
• Use intermediate files for multi-step transforms.
• Tune -j to match your machine.
• Add --verbose to monitor long runs.

License

MIT — see LICENSE.

Contributing

Fork, branch, add tests, ensure cargo test and cargo clippy pass, open a PR.

Support

Issues and questions: https://github.com/Vitruves/nail-parquet/issues