dftk — DataFrame analysis and manipulation toolkit

dftk is a command-line toolkit for exploratory data analysis on TSV files. Each subcommand reads tabular data, performs one operation, and writes TSV to stdout, making it easy to chain commands in pipelines.

dftk was developed during computational genomics research as a fast, composable alternative to writing one-off pandas scripts. It is designed for analysts who live in the terminal and want a consistent, pipeable toolkit for the full data analysis pipeline — from initial exploration through statistical modelling and publication-quality figures.

Installation

uv tool install dataframe-toolkit
dftk --help

Or with pip:

pip install dataframe-toolkit

To install the latest main branch directly from GitHub:

uv tool install git+https://github.com/hemingur/dataframe-toolkit.git

To install from a local clone:

git clone https://github.com/hemingur/dataframe-toolkit.git
cd dataframe-toolkit
uv tool install .

Quick example

dftk stat data.tsv -c height weight -g sex
dftk stat data.tsv -c value -g group -o \
  | dftk pivot -i group -v value_mean -o \
  | dftk print

Subcommands

Data transformation

Statistics

Plots

All plot commands write a PNG/PDF when -f FILE is given, or display interactively otherwise. They support --groupcol for colour grouping, --subgraphcol for subplot grids, and figure/font presets for publication-quality output.

Utilities

Common patterns

Chaining commands

# z-score within group, then fit a model
dftk scale data.tsv -c expr -g condition -o \
  | dftk fit - -f "expr_z ~ time + batch" -g condition

Group summary then plot

dftk stat results.tsv -c value -g group -o \
  | dftk line - -x group -y value_mean --yerr value_sem -f fig.png

Wide-to-long then plot overlaid histograms

dftk melt data.tsv -i sample -d gene -v expression \
  | dftk hist - -x expression -g gene -k -f dist.png

Interpolation (standard curve lookup)

dftk interp samples.tsv --ref stdcurve.tsv \
    -x fluorescence -v concentration -d conc_ng_ul

Bootstrap confidence intervals

dftk stat data.tsv -c value -g group --bootstrap 1000 --randomseed 42 -o \
  | dftk pivot -i group -v value_mean -f mean -o \
  | dftk stat - -c group_A group_B

Input/output

All commands accept:

• A TSV filename as a positional argument
• - to read TSV from stdin
• ... to receive a parquet path from stdin (written by a previous -o command)
• A .parquet filename to read a named parquet file directly

Standard output options (available on all tabular commands):

• -o / --output — write a temp parquet for the next piped command instead of TSV to stdout
• --select col1 col2 … — keep only these columns
• --drop col1 col2 … — remove these columns
• --round N — round numeric output
• --postquery EXPR — filter output rows after processing
• --meta KEY=VALUE — embed provenance metadata in parquet output (repeatable)

Provenance annotations

Metadata embedded with --meta is stored in the parquet file schema and propagates automatically through the pipe: every subsequent -o write re-embeds it alongside any new --meta values.

# Tag a file at creation
dftk eval raw.tsv -f "z = x + y" -o results.parquet \
    --meta genome=hg38 --meta source=gwas_2024

# Inspect annotations
dftk annotate results.parquet
# genome   hg38
# source   gwas_2024

# Add or update an annotation in-place
dftk annotate results.parquet --set step=qc_filtered

# Annotations survive piping
dftk eval results.parquet -f "z_scaled = z / 2" -o | dftk scale ... -c z -o scaled.parquet
dftk annotate scaled.parquet   # genome and source still present

Figure options (plot commands)

--size single|double|full|WxH    figure size (single ≈ 3.5", double ≈ 7.2")
--fontsize screen|publication|presentation
-f FILE                          save to file (PNG/PDF/SVG); omit to display
--groupcol COL                   colour-code by this column
--subgraphcol COL                split into subplot grid by this column
--ncols N                        columns in subplot grid (default: auto)
--legend TEXT                    legend label for ungrouped series

Dependencies

• pandas, numpy, scipy — core data handling and statistics
• statsmodels — regression (fit, scale --resid, wstat)
• duckdb — SQL queries (query --sql)
• matplotlib, seaborn — plots
• pyarrow — parquet I/O backend