dftk — DataFrame analysis and manipulation toolkit
dftk is a command-line toolkit for exploratory data analysis on TSV files.
Each subcommand reads tabular data, performs one operation, and writes TSV to
stdout, making it easy to chain commands in pipelines.
dftk was developed during computational genomics research as a fast,
composable alternative to writing one-off pandas scripts. It is designed
for analysts who live in the terminal and want a consistent, pipeable toolkit
for the full data analysis pipeline — from initial exploration through
statistical modelling and publication-quality figures.
No data file handy? Every example below uses dftk dataset to pull in a
small, real dataset, so you can copy-paste any of them and run them right
now. Run dftk dataset --list to browse everything available (dozens of
datasets from seaborn, statsmodels, and pydataset). dftk dataset NAME
searches all three sources in that order and returns the first match, so
--source is optional — pass it only to disambiguate a name that exists in
more than one source, or to restrict the search.
Installation
uv tool install dataframe-toolkit
dftk --help
Or with pip:
pip install dataframe-toolkit
To install the latest main branch directly from GitHub:
uv tool install git+https://github.com/hemingur/dataframe-toolkit.git
To install from a local clone:
git clone https://github.com/hemingur/dataframe-toolkit.git
cd dataframe-toolkit
uv tool install .
Skill for coding agents
skills/dftk/ is an agent skill: a short guide that coding agents such as
Codex and Claude Code load when a task involves dftk. It covers the piping
conventions, which command to use for what, and patterns for Snakemake and
Make. Copy it into your agent's skills directory, for example:
cp -r skills/dftk ~/.codex/skills/ # Codex
cp -r skills/dftk ~/.claude/skills/ # Claude Code
Copy it again after upgrading dftk so it matches the installed version.
Quick example
dftk dataset tips | dftk stat - -c tip total_bill -g day
Chain commands with -o (write to a temp parquet, print its path) and ...
(read a parquet path from stdin) instead of piping TSV text directly — faster
for large data, and the recommended pattern once a pipeline has more than one
step:
dftk dataset tips -o \
| dftk pivot ... -i day -v tip -f mean -o \
| dftk print ...
Subcommands
Data transformation
| Command | Description |
|---|---|
eval |
Add or modify columns: eval expressions, string/path functions, statistical ops |
query |
Filter rows with pandas query expressions or SQL (--sql via DuckDB) |
merge |
Join two tables on key columns (inner/left/right/outer) |
concat |
Concatenate two or more tables row-wise |
melt |
Reshape wide-to-long (pd.melt) |
explode |
Split separated values in a cell (a,b) into one row per value |
implode |
Join a column's values within groups into one cell, the inverse of explode |
pivot |
Reshape long-to-wide with per-cell aggregation |
func |
Column transforms: cumsum, group mean/sum/min/max/count/median/std, rank, qcut:N |
scale |
Normalise: z-score, min-shift, sum/max/mean scaling, Blom rank, regression residuals |
interp |
Interpolate values from a reference curve into a table (1-D lookup) |
binx |
Assign bin indices to a column based on explicit or generated edges |
transpose |
Flip rows and columns; original column names land in a new --keycol column |
segid |
Assign a segment ID that increments on each value change in a column |
cleannames |
Rename columns to lowercase identifiers (Sample ID → sample_id), e.g. for SQL |
Statistics
| Command | Description |
|---|---|
stat |
Descriptive statistics (count/mean/std/CI/SEM/skew/kurtosis) with grouping and bootstrap |
wstat |
Weighted descriptive statistics (wmean, wstd, weighted quantile CI) |
fit |
OLS/robust/weighted regression via R-style formulas; tidy table, --summary, --anova |
test |
P-values between column pairs: t-test, Mann-Whitney, Wilcoxon, KS, correlations, bootstrap; groups |
corr |
Pairwise column correlations (Pearson/Spearman/Kendall) with optional BCa bootstrap CI |
describe |
Quick column-level summary (dtype, n, n_unique, n_null, sample values) |
info |
Per-column dtype, null counts, and memory usage; --summary for dataset-level totals |
randvar |
Sample from a distribution and append as a new column (norm, alpha, beta, …) |
Plots
All plot commands write a PNG/PDF when -f FILE is given, or display
interactively otherwise. They support --groupcol for colour grouping,
--subgraphcol for subplot grids, and figure/font presets for
publication-quality output.
| Command | Description |
|---|---|
scat |
Scatter plot; OLS/robust or LOWESS fit (--fit lowess, one per group with -g), hexagonal binning for large tables (--hexbin), bubble size (--sizecol), colour (--colorcol) |
line |
Line plot; optional error bars (--yerr) or CI bands (--yci lo,hi) and fit overlay |
hist |
Histogram or KDE (--kde); normalisation, cumulative mode, mean±σ annotation (--stats) |
chi |
Corner-resolved chi-plot: whether tail dependence sits in the lower or upper corner |
Utilities
| Command | Description |
|---|---|
dataset |
Load a curated example dataset from seaborn, statsmodels, or pydataset |
sample |
Random row sampling (with or without replacement, by count or fraction) |
split |
Split a dataframe into one file per group |
annotate |
Read and write provenance metadata (genome, source, …) in parquet files |
print |
Read any dftk input (TSV, stdin, ... parquet pipe) and write TSV |
clean |
Remove leftover temp parquet pipe files from interrupted pipelines |
help |
List all subcommands or show full help for one: dftk help stat |
Common patterns
Chaining commands
# z-score sepal_length within species, then fit against petal_length
dftk dataset iris -o \
| dftk scale ... -c sepal_length -g species -o \
| dftk fit ... -f "sepal_length_scaled ~ petal_length" -g species
Clean up spreadsheet headers
Headers such as Country Name or 1960 need quoting in SQL. cleannames turns them into lowercase identifiers (country_name, x1960) and keeps the original names in the parquet metadata (dftk annotate FILE):
dftk dataset fertility -o \
| dftk cleannames ... -o \
| dftk query ... --sql "SELECT country_name, x1960, x2000 FROM data LIMIT 3"
For your own export, dftk cleannames data.tsv --backend duckdb -o data.parquet also keeps integer columns with empty cells as integers.
Large tables: hexbin instead of points
With tens of thousands of points a scatter plot is a solid blob. --hexbin counts points per hexagon instead (--hexlog for a log colour scale), and --fit lowess adds a smooth curve:
dftk dataset diamonds -o \
| dftk scat ... -x carat -y price --hexbin --hexlog --logx --logy -f diamonds.png
--colorcol COL colours each hexagon by the mean of COL instead of the count; with --subgraphcol all panels share one colour scale.
Group summary then plot
dftk dataset tips -o \
| dftk stat ... -c tip -g day -o \
| dftk line ... -x day -y mean --yerr sem -f fig.png
Wide-to-long then plot overlaid histograms
dftk dataset iris \
| dftk melt - -i species -d measurement -v value \
| dftk hist - -x value -g measurement -k -f dist.png
Interpolation (standard curve lookup)
printf "conc\tfluor\n0\t5\n10\t52\n20\t98\n30\t151\n" > stdcurve.tsv
printf "sample\tfluor\ns1\t60\ns2\t110\n" \
| dftk interp - --ref stdcurve.tsv -x fluor --refx fluor -v conc -d conc_ng_ul
Filling missing values
eval -f takes pandas column methods (COL.method(...)), and fillna,
ffill, bfill and interpolate fill empty cells. Defaults from another
table, after a merge (a column in both files gets the suffix _r):
dftk merge data.tsv -r defaults.tsv -k type -t left -o \
| dftk eval ... -f "depth = depth.fillna(depth_r)" --drop depth_r
The group mean, a fixed value, or the last value above (ffill) within
each group:
dftk dataset penguins -o | dftk eval ... \
-f "mean = body_mass_g.groupby(species).transform('mean')" \
-f "body_mass_g = body_mass_g.fillna(mean)" --drop mean \
-f "sex = sex.fillna('unknown')"
dftk dataset healthexp -o \
| dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" -o \
| dftk eval ... -f "le_filled = le.groupby(Country).ffill()"
ffill, bfill and interpolate use the rows in the order they are in:
sort in a step of its own first (dftk eval data.tsv --sortasc site day -o | dftk eval ... -f ...), and use groupby(...) when the table holds
several series, or values are taken from the neighbouring group.
interpolate() has no grouped form and works by row position. dftk eval -h covers these cases, with examples.
Bootstrap confidence intervals
Bootstrap mode repeats the full stat computation N times on resampled data
and emits one row per resample (tagged with samplenum) — pipe into pivot
with percentile aggfuncs (cilo/cihi) to collapse that into an empirical
confidence interval:
dftk dataset tips -o \
| dftk stat ... -c tip -g day --bootstrap 1000 --randomseed 42 -o \
| dftk pivot ... -i day -v mean -f mean cilo cihi -o \
| dftk print ...
Input/output
All commands accept:
- A TSV filename as a positional argument
-to read TSV from stdin...to receive a parquet path from stdin (written by a previous-ocommand)- A
.parquetfilename to read a named parquet file directly
-vs...— these are not interchangeable. Use-when the previous command in the pipe wrote plain TSV to stdout (the default). Use...when the previous command used bare-o(pipe mode), which writes a temp parquet and prints its path to stdout —-would try to parse that path as TSV and fail.
Comment lines and preambles
dftk has no comment or skip-rows option: # is ordinary data, so headers
like Count # and values like A #1 are read as they are. Drop comment
lines or a preamble in the shell instead:
grep -v '^#' data.tsv | dftk print # drop every line starting with #
tail -n +3 data.tsv | dftk print # skip a 2-line preamble
dftk print <(grep -v '^#' data.tsv) # the same, as a filename
With --backend duckdb, a whole-line # comment below the header is an
error, and # lines above the header are skipped.
Standard output options (available on all tabular commands):
-o/--output— controls where output goes; takes an optional value:-oalone — write a temp parquet to pipe into the next command, printing its path to stdout (auto-deleted once read)-o FILE.parquet— write a named, reusable parquet, printing its path to stdout-o FILE(no.parquetextension) — write TSV toFILE, with no stdout output- omit
-oentirely — write TSV to stdout (the default)
--select col1 col2 …— keep only these columns--drop col1 col2 …— remove these columns--round N— round numeric output--postquery EXPR— filter output rows after processing--meta KEY=VALUE— embed provenance metadata in parquet output (repeatable)
Provenance annotations
Metadata embedded with --meta is stored in the parquet file schema and propagates
automatically through the pipe: every subsequent -o write re-embeds it alongside
any new --meta values.
# Tag a file at creation
dftk dataset iris -o iris.parquet --meta source=seaborn --meta species_col=species
# Inspect annotations
dftk annotate iris.parquet
# source seaborn
# species_col species
# Add or update an annotation in-place
dftk annotate iris.parquet --set step=raw
# Annotations survive piping
dftk eval iris.parquet -f "petal_area = petal_length * petal_width" -o iris2.parquet
dftk annotate iris2.parquet # source, species_col, and step all still present
Figure options (plot commands)
--size single|double|full|WxH figure size (single ≈ 3.5", double ≈ 7.2")
--fontsize screen|publication|presentation
-f FILE save to file (PNG/PDF/SVG); omit to display
--groupcol COL colour-code by this column
--subgraphcol COL split into subplot grid by this column
--ncols N columns in subplot grid (default: auto)
--legend TEXT legend label for ungrouped series
Dependencies
pandas,numpy,scipy— core data handling and statisticsstatsmodels— regression (fit,scale --resid,wstat)duckdb— SQL queries (query --sql)matplotlib,seaborn— plotspyarrow— parquet I/O backend
Metadata
Release files for dataframe-toolkit 0.11.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dataframe_toolkit-0.11.0.tar.gz | 104.4 kB | With provenance | Details for dataframe_toolkit-0.11.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataframe_toolkit-0.11.0-py3-none-any.whl | Python 3 | none | any | Details for dataframe_toolkit-0.11.0-py3-none-any.whl |
Total release size: 232.2 kB
Release files / dataframe_toolkit-0.11.0.tar.gz
| Download URL | dataframe_toolkit-0.11.0.tar.gz |
|---|---|
| Size | 104.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
de01fc6fb3f8a25ade69c97a6bee7b199cd34e34e8ee43042c00de7b9db26781
|
|
BLAKE2b-256 checksum How to use checksums |
51933e412c7101075fccf50ca763bd1e730de24de6b479b224c0ed1aac48145c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 24, 2026.
Transparency logRelease files / dataframe_toolkit-0.11.0-py3-none-any.whl
| Download URL | dataframe_toolkit-0.11.0-py3-none-any.whl |
|---|---|
| Size | 127.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2eb1208ffc89ccdbf71f53c625afb51d8f933bc8cfdb73c4dbbabe5141882d55
|
|
BLAKE2b-256 checksum How to use checksums |
b8c87bef30dce96c7491bc74ac5892c8f9421aa63c258042184830577809a8e5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 24, 2026.
Transparency logSecurity for dataframe-toolkit 0.11.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for dataframe_toolkit-0.11.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 24, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.