dftk — DataFrame analysis and manipulation toolkit
dftk is a command-line toolkit for exploratory data analysis on TSV files.
Each subcommand reads tabular data, performs one operation, and writes TSV to
stdout, making it easy to chain commands in pipelines.
dftk was developed during computational genomics research as a fast,
composable alternative to writing one-off pandas scripts. It is designed
for analysts who live in the terminal and want a consistent, pipeable toolkit
for the full data analysis pipeline — from initial exploration through
statistical modelling and publication-quality figures.
No data file handy? Every example below uses dftk dataset to pull in a
small, real dataset, so you can copy-paste any of them and run them right
now. Run dftk dataset --list to browse everything available (dozens of
datasets from seaborn, statsmodels, and pydataset, plus gwas, a synthetic
association scan made by dftk). dftk dataset NAME searches dftk, seaborn,
statsmodels and pydataset in that order and returns the first match, so
--source is optional — pass it only to disambiguate a name that exists in
more than one source, or to restrict the search.
Cheatsheet: two landscape pages with the commands, options, SQL idioms,
plots and pipeline recipes at a glance —
dftk-cheatsheet.pdf.
Every example on it is run against dftk when it is built
(python docs/cheatsheet/make_cheatsheet.py).
Installation
uv tool install dataframe-toolkit
dftk --help
Or with pip:
pip install dataframe-toolkit
To install the latest main branch directly from GitHub:
uv tool install git+https://github.com/hemingur/dataframe-toolkit.git
To install from a local clone:
git clone https://github.com/hemingur/dataframe-toolkit.git
cd dataframe-toolkit
uv tool install .
Skill for coding agents
dftk ships an agent skill: a short guide that coding agents such as Codex and Claude Code load when a task involves dftk. It covers the piping conventions, which command to use for what, and patterns for Snakemake and Make. Install it with:
dftk skill install # every agent found: ~/.claude and/or ~/.codex
dftk skill install --codex # only one (--claude or --codex; creates the folder)
dftk skill install --dir DIR # another agent's skills folder
It prints where it installed the skill. Run it again after upgrading dftk
so the skill matches the installed version; it replaces only the earlier
dftk skill. The source is in src/dftk/skills/dftk/.
Quick example
dftk dataset tips | dftk stat - -c tip total_bill -g day
Chain commands with a bare -o (write a temp parquet, print its path)
instead of piping TSV text: the next command sees the path on stdin and reads
that file. It's faster for large data, keeps column types exact, and is the
recommended pattern once a pipeline has more than one step:
dftk dataset tips -o \
| dftk pivot -i day -v tip -f mean -o \
| dftk print
Subcommands
Data transformation
| Command | Description |
|---|---|
eval |
Add or modify columns: eval expressions, string/path functions, statistical ops |
query |
Filter rows with pandas query expressions or SQL (--sql via DuckDB) |
merge |
Join two tables on key columns (inner/left/right/outer) |
concat |
Concatenate two or more tables row-wise |
append |
Add one row from -c COL=VALUE; --inplace adds it to a file, e.g. a status log |
melt |
Reshape wide-to-long (pd.melt) |
explode |
Split separated values in a cell (a,b) into one row per value |
implode |
Join a column's values within groups into one cell, the inverse of explode |
pivot |
Reshape long-to-wide with per-cell aggregation |
func |
Column transforms: cumsum, group mean/sum/min/max/count/median/std, rank, qcut:N; interpolate/ffill/bfill within groups by an x column |
scale |
Normalise: z-score, min-shift, sum/max/mean scaling, Blom rank, regression residuals |
interp |
Interpolate values from a reference curve into a table (1-D lookup) |
binx |
Assign bin indices to a column based on explicit or generated edges |
transpose |
Flip rows and columns; original column names land in a new --keycol column |
segid |
Assign a segment ID that increments on each value change in a column |
cleannames |
Rename columns to lowercase identifiers (Sample ID → sample_id), e.g. for SQL |
Statistics
| Command | Description |
|---|---|
stat |
Descriptive statistics (count/mean/std/CI/SEM/skew/kurtosis) with grouping and bootstrap |
wstat |
Weighted descriptive statistics (wmean, wstd, weighted quantile CI) |
fit |
OLS/robust/weighted regression via R-style formulas; tidy table, --summary, --anova |
test |
P-values between column pairs: t-test, Mann-Whitney, Wilcoxon, KS, correlations, bootstrap; groups |
corr |
Pairwise column correlations (Pearson/Spearman/Kendall) with optional BCa bootstrap CI; globs (-c '*:*') or a full matrix (--all) |
describe |
Quick column-level summary (dtype, n, n_unique, n_null, sample values) |
info |
Per-column dtype, null counts, and memory usage; --summary for dataset-level totals |
randvar |
Sample from a distribution and append as a new column (norm, alpha, beta, …) |
Plots
All plot commands write a PNG/PDF when -f FILE is given, or display
interactively otherwise. They support --groupcol for colour grouping,
--subgraphcol for subplot grids, and figure/font presets for
publication-quality output.
| Command | Description |
|---|---|
scat |
Scatter plot; OLS/robust or LOWESS fit (--fit lowess, one per group with -g), hexagonal binning for large tables (--hexbin), bubble size (--sizecol), colour (--colorcol) |
line |
Line plot; optional error bars (--yerr) or CI bands (--yci lo,hi) and fit overlay |
hist |
Histogram or KDE (--kde); normalisation, cumulative mode, mean±σ annotation (--stats) |
chi |
Corner-resolved chi-plot: whether tail dependence sits in the lower or upper corner |
heatmap |
Coloured grid of values from a long (-x -y -v) or wide (-y) table; correlograms from corr --all, --annot, --lower |
pairs |
Scatter-plot matrix with KDEs (or histograms) on the diagonal; -g colours groups, --upper corr puts r in the upper triangle |
qq |
QQ plot of p-values: observed against expected -log10(p), confidence band, genomic inflation λ; -g one curve and λ per group |
locus |
Regional association plot: -log10(p) in one region coloured by r² with the lead variant (--ldcol), lead as a diamond, --lead/--label by ID or chr:pos |
manhattan |
Genome-wide -log10(p) by position (Chr, pos, Pval, or --mlogpcol/--chi2col); significance lines, thinning for millions of variants, -g/--subgraphcol for several scans |
Utilities
| Command | Description |
|---|---|
dataset |
Load a curated example dataset from dftk, seaborn, statsmodels, or pydataset |
sample |
Random row sampling (with or without replacement, by count or fraction) |
split |
Split a dataframe into one file per group |
annotate |
Read and write provenance metadata (genome, source, …) in parquet files |
print |
Read any dftk input (TSV, parquet, stdin) and write TSV, JSON, Markdown or LaTeX |
clean |
Remove leftover temp parquet pipe files from interrupted pipelines |
help |
List all subcommands or show full help for one: dftk help stat |
Common patterns
Chaining commands
# z-score sepal_length within species, then fit against petal_length
dftk dataset iris -o \
| dftk scale -c sepal_length -g species -o \
| dftk fit -f "sepal_length_scaled ~ petal_length" -g species
Clean up spreadsheet headers
Headers such as Country Name or 1960 need quoting in SQL. cleannames turns them into lowercase identifiers (country_name, x1960) and keeps the original names in the parquet metadata (dftk annotate FILE):
dftk dataset fertility -o \
| dftk cleannames -o \
| dftk query --sql "SELECT country_name, x1960, x2000 FROM data LIMIT 3"
For your own export, dftk cleannames data.tsv --backend duckdb -o data.parquet also keeps integer columns with empty cells as integers.
JSON files
DuckDB reads JSON itself, so query --sql takes a JSON file in FROM (a file of records, [{...}, {...}], or JSON Lines, one object per line; 'logs/*.jsonl' reads several). When the SQL doesn't use the data table, dftk reads no input. In TSV a nested object or list is shown as one line of JSON (["x","y"]), which later commands can't compute on, so flatten it in the same query: pos.* makes a column of each field, len() counts a list and unnest() gives one row per item:
printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
'{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
> records.jsonl
dftk query --sql \
"SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
# id chr start ntags
# 1 1 100 2
# 2 2 200 0
dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
| dftk pivot -i tag -v id -f count
A field that is a string in some records and a list in others (a variant catalog's ReferenceRegion, say) gets DuckDB type JSON; TSV shows its strings without quotes and its lists as JSON. json_type() tells them apart, so a list can become one row per item:
printf '%s\n' '{"locus": "AFF2", "region": "chrX:1-60"}' \
'{"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}' \
> catalog.jsonl
dftk query --sql "
SELECT locus,
unnest(CASE WHEN json_type(region) = 'ARRAY' THEN region->>'\$[*]'
ELSE [region->>'\$'] END) AS region
FROM 'catalog.jsonl'"
# locus region
# AFF2 chrX:1-60
# ATXN7 chr3:1-30
# ATXN7 chr3:30-42
To get JSON back, write -o FILE.json or .jsonl, or --format json for stdout (see Output formats). Nested values and mixed fields are written as JSON again; a key missing from a record is written as null:
dftk query --sql "SELECT * FROM 'catalog.jsonl' WHERE locus LIKE 'ATXN%'" --format json
# [
# {"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}
# ]
One row: status logs and small records
append adds one row from -c COL=VALUE values; columns not given stay empty. Numbers and true/false are typed, a quoted value ('"007"') stays text and note= is empty. Without input it makes a one-row table, a quick way to write a small JSON record:
dftk append -c sample=S12 -c reads=48213 -c ok=true --format json
# [
# {"sample": "S12", "reads": 48213, "ok": true}
# ]
--inplace adds the row to a file, creating it if missing: a status log that parallel jobs (Snakemake rules, array jobs) can write to, since the file's directory is locked while each row is written. --timestamp adds the local time as ISO 8601 text (2026-11-09T19:13:57, no space, so awk and cut see one field; a real timestamp in .parquet), and --strict refuses columns the log doesn't have, so a typo can't start a new one:
dftk append runs.tsv --inplace -c step=align -c sample=S12 -c status=ok --timestamp
dftk append runs.jsonl --inplace -c step=align -c reads=48213 --timestamp
A row whose columns are all in a TSV header, and any .jsonl row, is written as one more line; existing rows are never changed. A new column, or a .parquet log, rewrites the file (to a temp file that then replaces it). In a .jsonl log records need not share keys; read it back with dftk query --sql "SELECT * FROM 'runs.jsonl'".
Large tables: hexbin instead of points
With tens of thousands of points a scatter plot is a solid blob. --hexbin counts points per hexagon instead (--hexlog for a log colour scale), and --fit lowess adds a smooth curve:
dftk dataset diamonds -o \
| dftk scat -x carat -y price --hexbin --hexlog --logx --logy -f diamonds.png
--colorcol COL colours each hexagon by the mean of COL instead of the count; with --subgraphcol all panels share one colour scale.
Group summary then plot
dftk dataset tips -o \
| dftk stat -c tip -g day -o \
| dftk line -x day -y mean --yerr sem -f fig.png
Wide-to-long then plot overlaid histograms
dftk dataset iris \
| dftk melt - -i species -d measurement -v value \
| dftk hist - -x value -g measurement -k -f dist.png
Interpolation (standard curve lookup)
printf "conc\tfluor\n0\t5\n10\t52\n20\t98\n30\t151\n" > stdcurve.tsv
printf "sample\tfluor\ns1\t60\ns2\t110\n" \
| dftk interp - --ref stdcurve.tsv -x fluor --refx fluor -v conc -d conc_ng_ul
Filling missing values
eval -f takes pandas column methods (COL.method(...)); fillna,
ffill, bfill and interpolate fill empty cells and keep the others.
| Formula | Fills with |
|---|---|
x = x.fillna(y) |
the value in column y |
x = x.fillna(0), sex = sex.fillna('unknown') |
a fixed value |
x = x.fillna(y).fillna(0) |
y first, then 0 |
x = x.fillna(x.groupby(g).transform('mean')) |
the mean of the row's group (or 'median', ...) |
x = x.ffill() / x = x.bfill() |
the value above / below (ffill(limit=1): at most one row) |
x = x.groupby(g).ffill() |
the value above, within each group of g |
x = x.interpolate() |
a straight line between the values above and below, by row position |
Write to a new column (x_filled = x.fillna(y)) to keep the original, and
mark the gaps first (missing = x.isna()) to tell filled values apart.
Only empty cells are filled: a missing row (a year with no row at all) is
not added.
Defaults from another table. A column that is in both files gets the
suffix _r from merge; fill from it, then drop it. Here the defaults are
each species' median body mass:
dftk dataset penguins -o | dftk pivot -i species -v body_mass_g \
-f median --move median_body_mass_g:body_mass_g -o defaults.parquet
dftk dataset penguins -o \
| dftk merge -r defaults.parquet -k species -t left -o \
| dftk eval -f "body_mass_g = body_mass_g.fillna(body_mass_g_r)" \
--drop body_mass_g_r
From neighbouring rows (co2 is a weekly series with gaps):
dftk dataset co2 -o | dftk eval -f "ffill = co2.ffill()" \
-f "bfill = co2.bfill()" -f "linear = co2.interpolate()"
ffill leaves the first rows empty and bfill the last ones;
x.ffill().fillna(x.bfill()) fills both. interpolate() counts one step
per row, so the rows must be evenly spaced; for unevenly spaced rows use
func -t interpolate --xcol X (below), and for a lookup from a reference
curve, dftk interp.
Row order and groups. These methods use the rows in the order they are
in. Sort first, in a step of its own: --sortasc applies to a command's
output, so in the same command it would come after the fill. When a table
holds several series (sites, countries, samples), fill within each group,
or values are taken from the neighbouring group. Rows of a group don't need
to be next to each other, but within each group they must be in order:
dftk dataset healthexp -o \
| dftk eval -f "le = Life_Expectancy.where(Year % 10 != 5)" \
--sortasc Country Year -o \
| dftk eval -f "le_filled = le.groupby(Country).ffill()"
For interpolation within groups, func -t interpolate does it in one
step. It draws the line against an x column (--xcol, required), so
unevenly spaced rows (missing years, genome positions) get the right
values; it takes each group's rows in X order itself, so no sort step is
needed, and it leaves gaps before a group's first value or after its last
empty. -t ffill and -t bfill work the same way; chained after
interpolate on the same column (-d), they fill those ends too:
dftk dataset healthexp -o \
| dftk eval -f "le = Life_Expectancy.where(Year % 10 != 5)" -o \
| dftk func -c le -t interpolate -g Country --xcol Year -d le_filled -o \
| dftk func -c le_filled -t ffill -g Country --xcol Year -d le_filled -o \
| dftk func -c le_filled -t bfill -g Country --xcol Year -d le_filled
eval's interpolate() counts one step per row: with 10 at x=2 and 14 at
x=6, it gives the gap at x=5 12.0, where func --xcol gives 13.0.
Differences between rows
For the change in a column from one row to the next, eval -f takes
pandas diff() and pct_change(), also within groups. They use the rows
in their order, so sort first, in a step of its own:
dftk dataset healthexp -o | dftk eval --sortasc Country Year -o \
| dftk eval -f "change = Spending_USD.groupby(Country).diff().round(1)" \
-f "pct = (Spending_USD.groupby(Country).pct_change() * 100).round(1)" \
-f "years = Year.groupby(Country).diff()"
Year Country Spending_USD Life_Expectancy change pct years
1971 Canada 313.391 72.8
1976 Canada 543.337 73.8 229.9 73.4 5.0
1979 Canada 692.269 75.1 148.9 27.4 3.0
1980 Canada 791.812 75.2 99.5 14.4 1.0
The first row of each country gets an empty cell. Without
groupby(Country), it would be compared with the last row of the
previous country.
To compare a value with one in another column of the previous or next row, for example the
gap between segments given by Chr, Begin, End, use SQL's LAG() /
LEAD() in query --sql:
dftk query segments.tsv --sql \
'SELECT *, "Begin" - LAG("End") OVER (PARTITION BY Chr ORDER BY "Begin")
AS gap
FROM data ORDER BY Chr, "Begin"'
Chr Begin End Id gap
chr1 100 200 a
chr1 250 400 b 50
chr1 390 500 c -10
chr2 50 80 d
chr2 100 300 e 20
A negative gap is an overlap, and the first segment on each chromosome
gets an empty cell. lead("Begin") OVER (...) - "End" gives the gap to
the next segment instead.
BeginandEndare SQL keywords: quote such column names with double quotes and the whole statement with single quotes.- No sorting is needed first;
ORDER BYinsideOVERsets the order. - Integers stay integers.
eval -f can do the same with pandas shift, but it goes by row order, so
sort by Chr and Begin first (results are floats):
dftk eval segments.tsv -f 'gap = Begin - End.groupby(Chr).shift()'
Without groupby(Chr) the first segment of a chromosome is compared with
the last one of the previous chromosome.
Correlograms and heatmaps
corr --all correlates every pair of numeric columns both ways round, plus the diagonal, which is the square table heatmap draws. Values of both signs get diverging colours centred on 0; --vmin -1 --vmax 1 keeps the full scale for r, so a weak correlation doesn't look strong. --annot writes the values in the cells and --lower keeps the lower triangle:
dftk dataset iris -o | dftk corr --all -o \
| dftk heatmap -x col1 -y col2 -v correlation --annot --vmin -1 --vmax 1 -f iris_corr.png
dftk dataset diamonds -o | dftk corr --all --method spearman -o \
| dftk heatmap -x col1 -y col2 -v correlation --annot --lower --square \
--vmin -1 --vmax 1 -f diamonds_corr.png
Missing values are dropped pair by pair, so nobs can differ between cells; dftk print --dropna -o | dftk corr --all uses the complete rows throughout. p-values are not adjusted for multiple testing.
For the shape of each relationship, not just r, pairs draws every pair of columns as a scatter plot with each column's KDE on the diagonal; -g colours groups (each group's KDE scaled by its share of the rows) and --upper corr writes r in the upper triangle, with -g also r within each group, which can differ even in sign from r over all rows:
dftk dataset iris -o | dftk pairs -g species -f iris_pairs.png
dftk dataset iris -o | dftk pairs -g species --upper corr --diag hist -f iris_pairs_r.png
heatmap draws any table with one row per cell (-x -y -v), or a wide one with -y naming the row labels and the other numeric columns as heatmap columns, as pivot makes it:
dftk dataset penguins -o | dftk pivot -i species -g island -v body_mass_g -f count -o \
| dftk heatmap -y species --annot --cmap Blues -f penguin_counts.png
Manhattan plots
manhattan draws -log10(p) of every variant along the genome, chromosomes end to end in order (1–22, X, Y, MT; a chr prefix is ignored). The columns are Chr, pos and Pval unless named with --chrcol, --poscol and --pcol; a missing default is looked for under common spellings (CHR, #CHROM, BP, P, …). dataset gwas is a synthetic scan to try it on:
dftk dataset gwas -o | dftk manhattan -f manhattan.png
dftk dataset gwas -o | dftk manhattan --ylim 0 40 --sig 1e-6 --suggestive none -f manhattan.png
A p-value below about 1e-308 is stored as 0, which plots as -log10(p) = 323 with a warning. A chi-squared column gives the true value however small p is (--chi2df, default 1, a number or a column), and --mlogpcol takes a column that already holds -log10(p), such as regenie's LOG10P:
dftk dataset gwas -o | dftk manhattan --chi2col Chisq -f manhattan.png
Points with p > 0.01 are thinned to one per cell of a grid finer than the figure (--no-thin draws all), and points are rasterized: ten million variants take a few seconds and a PDF stays small. --ylim LO HI caps the axis, with points above it drawn as triangles on the top edge. --logy instead keeps every peak: the axis is linear up to -log10(p) = 10 and logarithmic above it (--logy 5 moves the switch), so a peak at 380 and one at 8 both show. --sigcolor red colours the points past the --sig line.
Several scans in one table: --subgraphcol study draws a panel each, -g study a colour each, and --miami study draws two of them face to face (a Miami plot): the first on top, the second upside down below it, with the chromosome numbers between them. The column must hold two values, or name the two to draw (--miami study a b, top first). Here the second scan is the example one with every chi-squared halved:
dftk dataset gwas -o \
| dftk eval -f "half = Chisq / 2" --select Chr pos Chisq half -o \
| dftk melt -i Chr pos -d study -v chi2 -o \
| dftk manhattan --chi2col chi2 --miami study --sigcolor red -f miami.png
qq plots the same p-values (same column options, no chromosome or position needed) against what null tests would give, with a 95 % band (--ci) and the genomic inflation factor λ, the median chi-squared over its null median. A curve lifted off the diagonal from the start, λ above about 1.05, points to stratification or other inflation; a few points leaving it at the top are associations. With -g, each group gets its own curve and λ:
dftk dataset gwas -o | dftk qq -f qq.png
dftk dataset gwas -o | dftk eval -f "Chisq = Chisq * 1.15" -o | dftk qq --chi2col Chisq -f qq.png
qq --logy makes its y axis linear up to 10 and logarithmic above, as in manhattan.
locus zooms in on one region: -log10(p) against position in Mb, the points coloured by r² with the lead variant (--ldcol, five bins, grey where missing) and the lead drawn as a diamond. It plots the rows it gets, so cut the region out with query first (input on several chromosomes is an error that suggests a query around the top hit). The lead is the smallest p-value unless --lead names one, and --label names more variants to write on the plot; both take an ID (SNP column, or --idcol) or a position like chr6:32600000. dftk doesn't compute LD: merge in r² with the lead, e.g. from plink --r2 --ld-snp. dataset gwas has an r2 column around each peak:
dftk dataset gwas -o | dftk query -q "Chr == '6' and pos > 31.6e6 and pos < 33.6e6" -o \
| dftk locus --ldcol r2 -f locus.png
Bootstrap confidence intervals
Bootstrap mode repeats the full stat computation N times on resampled data
and emits one row per resample (tagged with samplenum) — pipe into pivot
with percentile aggfuncs (cilo/cihi) to collapse that into an empirical
confidence interval:
dftk dataset tips -o \
| dftk stat -c tip -g day --bootstrap 1000 --randomseed 42 -o \
| dftk pivot -i day -v mean -f mean cilo cihi -o \
| dftk print
Input/output
All commands accept:
- A TSV filename as a positional argument
- A
.parquetfilename to read a named parquet file directly - Nothing, or
-, to read stdin: either TSV, or the parquet path printed by a previous command's bare-o(that temp file is then read and deleted) - A pipe as the filename, e.g.
<(grep -v '^#' data.tsv)or<(dftk … -o), read the same way as stdin - JSON through
query --sql "SELECT … FROM 'f.jsonl'", which needs no other input (see JSON files)
dftk dataset tips -o | dftk stat -c tip -g day # parquet path on stdin
dftk dataset tips | dftk stat -c tip -g day # TSV on stdin
dftk merge -l tips.tsv -r <(dftk pivot tips.tsv -i day -v tip -f mean -o) -k day
Older scripts give ... as the file to read a parquet path from stdin; that
still works. In zsh with Oh My Zsh an unquoted ... becomes ../.., so
leave it out (or use -) in scripts you share.
Output formats
Without -o, output is TSV on stdout. -o alone, or -o FILE.parquet, writes parquet and prints the path for the next command; -o FILE writes TSV, except for these extensions:
-o |
Writes |
|---|---|
FILE.json |
JSON array of records |
FILE.jsonl |
JSON Lines, one record per line |
FILE.md |
Markdown table (numeric columns right-aligned) |
FILE.tex |
bare LaTeX tabular with booktabs rules |
--format tsv|json|jsonl|md|tex writes the same formats to stdout, or overrides the extension. --digits N applies to all of them. The .tex file is only the tabular, with special characters (_ & % $ # { } ~ ^ \ | < >) escaped; it needs \usepackage{booktabs}, and goes inside your own table environment with \input{tips_by_day.tex}. Add \usepackage[T1]{fontenc} (pdfLaTeX's default font has no straight ", which JSON lists contain):
dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 --format md
# | day | mean_tip | count_tip |
# | --- | ---: | ---: |
# | Thur | 2.77 | 62 |
# ...
dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 -o tips_by_day.tex
Comment lines and preambles
dftk has no comment or skip-rows option: # is ordinary data, so headers
like Count # and values like A #1 are read as they are. Drop comment
lines or a preamble in the shell instead:
grep -v '^#' data.tsv | dftk print # drop every line starting with #
tail -n +3 data.tsv | dftk print # skip a 2-line preamble
dftk print <(grep -v '^#' data.tsv) # the same, as a filename
With --backend duckdb, a whole-line # comment below the header is an
error, and # lines above the header are skipped.
Files without a header
--noheader reads the first line as data and names the columns V1,
V2, …. It only describes the input: the output gets a header (V1 V2 …)
unless you add --removeheader. Name the columns with --move:
dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader # no header out
In merge, --noheader applies to both files, so both sides are named
V1 V2 …. The key appears once and the right side's other columns get the
_r suffix. --move runs after the merge, so it renames the merged names:
dftk merge -l left.txt -r right.txt -k V1 --noheader
# V1 V2 V3 V2_r V3_r
dftk merge -l left.txt -r right.txt -k V1 --noheader \
--move V1:id V2:n V3:s V2_r:n2 V3_r:s2
# id n s n2 s2
Parquet always has column names, so --removeheader with -o and
--noheader on parquet input do nothing (dftk warns).
Standard output options (available on all tabular commands):
-o/--output— controls where output goes; takes an optional value:-oalone — write a temp parquet to pipe into the next command, printing its path to stdout (auto-deleted once read)-o FILE.parquet— write a named, reusable parquet, printing its path to stdout-o FILE(no.parquetextension) — write TSV toFILE, with no stdout output- omit
-oentirely — write TSV to stdout (the default)
--select col1 col2 …— keep only these columns--drop col1 col2 …— remove these columns--round N— round numeric output--postquery EXPR— filter output rows after processing--move OLD:NEW …— rename columns--removeheader— leave out the header row in TSV output--meta KEY=VALUE— embed provenance metadata in parquet output (repeatable)
Provenance annotations
Metadata embedded with --meta is stored in the parquet file schema and propagates
automatically through the pipe: every subsequent -o write re-embeds it alongside
any new --meta values.
# Tag a file at creation
dftk dataset iris -o iris.parquet --meta source=seaborn --meta species_col=species
# Inspect annotations
dftk annotate iris.parquet
# source seaborn
# species_col species
# Add or update an annotation in-place
dftk annotate iris.parquet --set step=raw
# Annotations survive piping
dftk eval iris.parquet -f "petal_area = petal_length * petal_width" -o iris2.parquet
dftk annotate iris2.parquet # source, species_col, and step all still present
Figure options (plot commands)
--size single|double|full|WxH figure size (single ≈ 3.5", double ≈ 7.2")
--fontsize screen|publication|presentation
-f FILE save to file (PNG/PDF/SVG); omit to display
--groupcol COL colour-code by this column
--subgraphcol COL split into subplot grid by this column
--ncols N columns in subplot grid (default: auto)
--legend TEXT legend label for ungrouped series
Dependencies
pandas,numpy,scipy— core data handling and statisticsstatsmodels— regression (fit,scale --resid,wstat)duckdb— SQL queries (query --sql)matplotlib,seaborn— plotspyarrow— parquet I/O backend
Metadata
Release files for dataframe-toolkit 0.22.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dataframe_toolkit-0.22.0.tar.gz | 178.8 kB | With provenance | Details for dataframe_toolkit-0.22.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataframe_toolkit-0.22.0-py3-none-any.whl | Python 3 | none | any | Details for dataframe_toolkit-0.22.0-py3-none-any.whl |
Total release size: 378.1 kB
Release files / dataframe_toolkit-0.22.0.tar.gz
| Download URL | dataframe_toolkit-0.22.0.tar.gz |
|---|---|
| Size | 178.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bd67d09c6e6dc2d1633b8daa18dec438f6d97a50c6d540e816bc6a56b367e7fa
|
|
BLAKE2b-256 checksum How to use checksums |
b2626c0c1e2d71aaff4f1de4bac235fb86b3c7e5055f9ff32a8b492bd202a605
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / dataframe_toolkit-0.22.0-py3-none-any.whl
| Download URL | dataframe_toolkit-0.22.0-py3-none-any.whl |
|---|---|
| Size | 199.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
025d49ce2478b9ecfa3c0cb1793cf98fc9a9376c55d5be0cb73365164f31096b
|
|
BLAKE2b-256 checksum How to use checksums |
bf439909a4f83de16b4190008312479a4871f986950d6cc5374fcfaa2212c64d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logSecurity for dataframe-toolkit 0.22.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for dataframe_toolkit-0.22.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.