dtflow

English | 中文

A terminal browser for LLM training data, plus a CLI toolbox to filter, clean, convert and export it.

dt view opens an SFT / DPO / agent JSONL file as a table-plus-detail browser: conversations rendered as colored chat turns, tool calls formatted with bad JSON flagged, full-file search across every field, mouse support, and windowed loading that keeps a 900k-line file under 100 MB of RAM. The rest of dt is a pipeable Unix-style toolkit for the same data, where every condition is plain Python.

_{dt view data.jsonl — one command turns a training set into a searchable, filterable terminal browser}

Try it in 30 seconds

pip install dtflow                      # or, without installing: uvx --from dtflow dt view data.jsonl

dt view data.jsonl                                              # browse (press ? for keys)
dt stats data.jsonl --schema                                    # nested schema: types, null rates, values
dt filter data.jsonl "turns(x) >= 2 and search(x, 'refund')" \
  | dt select - "id,turns=turns(x),last=x.messages[-1].content" \
  | dt sort - --by x.turns --desc | dt head - 5
dt transform data.jsonl --preset=openai_chat -o train.jsonl     # sharegpt / alpaca / dpo → OpenAI messages

Reads JSONL/NDJSON (also .gz), JSON, CSV/TSV, Parquet, Arrow and Excel. Every data command accepts - for stdin and writes to stdout when no -o is given, so everything composes. The interface is in English; dt lang zh switches it to Chinese. No data at hand? examples/ has small synthetic chat / ShareGPT / Alpaca / DPO files to try these on.

Why dtflow

• It knows what a training sample is. Generic table tools show messages as {3} or a truncated string. dt view detects openai_chat / sharegpt / dpo / alpaca and renders one complete sample per screen: turns colored by role, code highlighted, tool_calls and reasoning_content unpacked, malformed tool arguments flagged, images of VLM samples one key away (i) with <image> count mismatches flagged.
• Conditions are Python, not a DSL. x.score > 0.8 and 'wiki' in x.meta.source, any('refund' in m.content for m in x.messages), plus row helpers that know what a conversation is: turns(x), roles(x), calls(x), search(x, 'refund'). One expression language across filter, select, map, sort, group, join, the viewer and YAML pipelines.
• Built for agents as much as humans. stdout carries only data, stderr carries messages, exit codes are contractual, dt schema prints the machine-readable command tree, and dt install-skill teaches Claude Code or Codex the whole tool in one command.
• Streams by default. filter / select / map / clean / dedupe / transform never load the file; the viewer opens a 910k-row JSONL at ~90 MB RSS with parallel full-file scans.

dt view: browse training data in the terminal

dt view <file> opens a master-detail browser. The table summarizes each sample (derived columns turns/roles/first_user/chars/calls plus your metadata); the detail pane renders the current row by format. JSONL, CSV and Parquet all open, and files with hundreds of thousands of lines open instantly.

Main screen: pick a row, detail renders by role with code highlighting	/ full-file search: every value of every record (assistant replies included), hits in yellow
F value picker: Excel AutoFilter style, unique values + counts, keep what you tick	dpo comparison: chosen / rejected in different colors, format auto-detected

dt view data.jsonl                          # open, press ? for keys
dt view app.jsonl -100 -f                   # follow the last 100 lines (log mode)
dt view data.jsonl -w "turns(x)>=6" -s error   # start filtered + searched
dt sample data.jsonl 500 | dt view -        # pipe: inspect sampled / processed output

Full key list, filter syntax, large-file and follow-mode details: docs/view.md.

How it compares to other terminal viewers

VisiData and tabiew are general table tools, jless and fx are JSON tree viewers, csvlens only reads CSV. None of them has the concept of "this row is a training sample": messages shows up as {3} or a string, and reading a conversation means expanding it level by level, one record at a time. dt view goes the other way: detect the format first, then draw the screen, so one screen is one complete sample.

When not to use dt view: for joins, pivots or per-column plots use VisiData; to run SQL over a table use tabiew; to inspect the structure of a single JSON document use jless. dt view only wins at "this is a batch of training samples and I need to scan, find, filter and export", which is exactly what you do with training data every day.

_{Memory is peak RSS after opening the file on the same Linux machine (2026-09). Other rows were checked against each tool's README and source; versions as in the header.}

The CLI: a Swiss army knife for training data

Every data command follows one contract: FILE may be - (NDJSON from stdin), a directory, or a quoted glob; without -o data goes to stdout while progress and summaries go to stderr. Conditions, derived fields, sort keys, group keys and join keys are Python expressions with the current row as x.

# primitives
dt filter  data.jsonl "x.score > 0.5 and 'wiki' in x.meta.source"
dt filter  data.jsonl "turns(x) >= 6 and 'get_weather' in calls(x)"    # row helpers: turns/roles/first_user/chars/calls/search
dt select  data.jsonl "id,text,n=len(x.messages),src=x.meta.source"     # project / rename / derive
dt map     data.jsonl "x.text = x.text.strip(); del x.debug"            # edit in place
dt explode data.jsonl --field messages --index-as turn                  # one row per list element
dt sort    data.jsonl --by "len(x.messages)" --desc
dt group   data.jsonl --by "roles(x)" --top 10                         # {"key","count","pct"}, ten largest groups
dt group   data.jsonl --by x.label --agg "avg=mean(len(r.text) for r in g)"
dt describe data.jsonl "turns(x)" "chars(x)" "x.score"                  # quantiles + histogram per expression
dt join    data.jsonl meta.jsonl --on x.id --prefix m_                  # left join, right side in memory
dt join    train.jsonl test.jsonl --on "first_user(x)" --anti           # drop rows that overlap the test set
dt dedupe  data.jsonl --key=messages[0].content -i                      # exact dedupe, in place
dt clean   data.jsonl --drop-empty=text --min-len=messages.#:2 -o clean.jsonl
dt split   data.jsonl --ratio=0.9 --seed=42                             # train/test files
dt concat  data.jsonl -o data.parquet                                   # one file in = format conversion
dt filter  shards/ "x.lang == 'zh'" -o zh.jsonl                         # a directory or 'shards/*.jsonl' reads every file
dt stats   data.jsonl --schema                                          # look before you write expressions

# pipes
dt filter d.jsonl "x.score>0.5" | dt select - "id,n=len(x.messages)" | dt sort - --by x.n --desc | dt head - 5
cat big.jsonl.gz | dt filter - "x.lang=='zh'" | dt transform - --preset=openai_chat | dt view -
dt group d.jsonl --by x.label | dt sort - --by x.count --desc

# and the rest
dt sample data.jsonl 1000 --by=meta.source      # stratified sampling
dt token-stats data.jsonl --model=gpt-4         # token distribution per role
dt validate data.jsonl --preset=openai_chat     # schema check, --filter to keep valid rows
dt diff a.jsonl b.jsonl --key=id                # what changed between two versions
dt export data.jsonl -f llama-factory           # data + config for LLaMA-Factory / ms-swift / Axolotl
dt run pipeline.yaml                            # reproducible pipeline, steps = CLI commands

Rows whose expression fails (missing field, None > 0.5) don't abort the run: filter drops them, select sets the item to null, map keeps the row, and stderr prints one summary at the end. --strict turns the first failure into exit code 1. A bare field name without x. is rejected at compile time instead of silently matching nothing. In a terminal, data commands preview the first 50 rows as a table with the same columns as dt view; piped, they emit NDJSON.

Command reference: docs/cli.md · expressions and field paths: docs/expressions.md · pipelines: docs/pipeline.md.

Python API

The same operations as a chainable class; x.field attribute access works on nested dicts and lists.

from dtflow import DataTransformer, load_stream

(DataTransformer.load("data.jsonl")
    .filter(lambda x: x.score > 0.8)
    .transform(preset="openai_chat", user_field="q", assistant_field="a")
    .dedupe("messages[0].content")
    .save("train.jsonl"))

(load_stream("huge.jsonl")                 # constant memory for files that don't fit
    .filter(lambda x: x["score"] > 0.5)
    .save("filtered.parquet"))

Presets (openai_chat, alpaca, sharegpt, dpo_pair), schema validation, token counting with tiktoken or HuggingFace tokenizers, converters for LLaMA-Factory / ms-swift / Axolotl / HuggingFace datasets / OpenAI Batch, lineage tracking: docs/python-api.md.

Agent skill

dt install-skill                         # Claude Code (default)
dt install-skill --target codex          # Codex

After installing, /dtflow in Claude Code or $dtflow in Codex gives the agent the full reference. The CLI itself is designed to be driven by an agent: dt schema prints the command tree as JSON, dt <cmd> --help carries examples, --dry-run previews every side-effecting command with exit code 10, and errors are structured JSON on stderr when stdout is not a terminal.

Documentation

• Quickstart
• CLI reference
• dt view
• Expressions and field paths (includes the 0.9 migration table)
• Pipelines
• Python API
• Changelog

Design

dtflow is built on a few decisions that everything else follows from.

• The unit is a training sample, not a row. A sample is a conversation, a preference pair or an instruction; formats are detected, not declared. That one piece of knowledge lives in a single place, the row helpers turns(x) / roles(x) / calls(x) / search(x, …), and the viewer's columns, the CLI's filters and the pipeline's steps all read from it.
• One language. Conditions, keys and derived fields are Python with the current row as x. There is no DSL to learn and no second dialect: the viewer compiles the same string as dt filter, so what you typed while browsing is what you paste into a script.
• Looking and processing are one loop. Every viewer state translates into a command (C reproduces the view, P reproduces it as dt filter … | dt sort …), and every command's output can be viewed again (dt … | dt view -, or | inside the viewer). Exports carry lineage, so a file can always say where it came from.
• The Unix contract, kept strictly. stdout is data, stderr is messages, exit codes mean things, - is stdin, no -o means stdout, output format follows the extension. That is what lets 30 small commands compose, and what lets an agent drive them: dt schema, structured JSON errors, --dry-run.
• Stream unless the operation cannot. filter / select / map / clean / dedupe / transform never load the file; the viewer parses one window and scans the rest in parallel. Only sort, shuffle, group --agg and the right side of join materialize, and the docs say so.
• No silent wrong answers. A bare field name, a bad regex or an unknown helper is a compile-time error, not an empty result with exit code 0. Rows that fail at runtime are counted and summarized; --strict turns the first one into a failure.
• Functions over class hierarchies. dt.to(lambda x: {...}) instead of class MyFormatter(BaseFormatter). Presets are conveniences, not the core abstraction.

License

MIT