Perdix

Perdix is a desktop workbench for feature engineering on tabular and time-series data. You build a recipe, an ordered list of steps, and see the data after every step. The result is checked for leakage and exported as a recipe file, a standalone pandas script or a scikit-learn pipeline.

Install and run

Requires Python 3.10 or newer.

pip install 'perdix[io]'      # io: Parquet/Feather (pyarrow) and Excel (openpyxl)
perdix                        # GUI; reopens your last session
perdix data.csv               # GUI on a table
perdix gui --demo             # GUI on the synthetic machine-sensor dataset

Optional extras: ai (Ask Claude), ts (catch22), calendar (public holidays).

To work on Perdix itself, install a checkout with dev (pytest and ruff):

python3 -m venv .venv
.venv/bin/pip install -e '.[dev,io]'
./perdix.sh                   # the GUI from the checkout

perdix.sh looks for an installed perdix ($PERDIX_VENV, the active venv, ./.venv). If there is none, it runs the checkout with the current python3.

New to Perdix? Click Try the demo dataset on the start screen: it opens the demo with an empty recipe and Getting started beside it.

Documentation and examples

The documentation is bundled with the package and opens inside the app (F1, or the book icon in the app bar). Each step's editor has a Theory link to its section. The same pages can be read on GitLab.

• Getting started and Concepts
• Theory per step family: cleaning, imputation, scaling, transforms, encoding, binning, date/time, generation, time series, text, selection, joins, geo
• Analysis, Leakage, Evaluation and Ask Claude
• Reference: CLI, file formats, exports, writing a step
• What changed between versions: CHANGELOG

Seven example cases come with data, a recipe and a tutorial (overview): six on synthetic data, which ship with Perdix, and one on NASA's simulated turbofan engines (C-MAPSS), whose data Perdix downloads from NASA when you ask (Settings → Datasets, or perdix examples download turbofan-engines). Open one from the start screen, from Help → Open example, or from the command line.

./perdix.sh gui --example machine-failure
perdix examples                          # list them
perdix examples copy customer-churn ./   # copy data, recipe and tutorial to a folder

The window

Errors and confirmations appear as toasts in the bottom-right corner. With no data open, the centre shows a start screen with your recent files. A table or a recipe can also be dropped anywhere on the window.

Selectors such as @numeric and @categorical pick columns by type when the step is fitted. They never pick the target, time or group columns. Explicit names are used as given.

Steps run in the background and are cached by everything upstream. Editing step 5 recomputes steps 5 and later only. The session (data path and recipe) is autosaved and restored on the next start.

Steps

69 steps in 13 categories.

All time-series steps sort by time within each group and put the results back in the original row order. Window steps exclude the current row by default, so a feature uses only the past.

perdix steps -v lists every step with its description and parameters.

Leakage report

Analysis → Leakage checks the data and the recipe for the following.

• features that nearly equal the target (|Spearman| ≥ 0.9, or one feature that predicts it alone);
• columns whose name contains the target name;
• identifiers (unique, or monotonic row counters);
• window steps on the target that include the current row;
• target encoding, and feature selection, fitted on all rows;
• any fitted statistics when Fit on fraction is 1;
• duplicate (group, time) pairs and target missing values.

On the demo dataset it finds the two planted traps, service_ticket, which is recorded after a failure, and record_id. It cannot see a leak that leaves no trace in the data, such as a weak column recorded after the outcome. That still needs somebody who knows how the data was made.

Exports

For every step, the test suite checks that the generated script reproduces the recipe's output exactly. The same is checked for the pipeline wherever the step can go in one.

Command line

perdix steps [--category Encode] [-v]
perdix demo demo.csv
perdix fit recipe.toml train.csv -o fitted.json      # fit and keep the learned values
perdix apply fitted.json new.csv -o features.parquet # transform with them
perdix apply recipe.toml data.csv -o out.csv --refit
perdix script fitted.json -o fe.py
perdix pipeline fitted.json -o pipe.pkl
perdix profile data.csv [--csv profile.csv]
perdix leakage data.csv --target y [--time t --group id --recipe r.json]
perdix evaluate fitted.json data.csv                 # held-out score, per-step ablation
perdix stability recipe.toml data.csv                # drift between training and held-out rows
perdix suggest data.csv --target y                   # rule-based step suggestions
perdix suggest data.csv --target y -o recipe.toml    # ... written as a recipe to start from
perdix search recipe.toml data.csv --top 10          # candidate features, screened on training rows
perdix search recipe.toml data.csv --generator gp    # ... or formulas bred by genetic programming
perdix propose recipe.toml data.csv --describe "..." # features proposed by Claude (needs ANTHROPIC_API_KEY)
perdix validate fitted.json new.csv                  # check new data against the recipe's data contract
perdix apply recipe.toml big.csv -o out.csv --chunksize 100000   # row-wise recipes on files larger than memory
perdix export-sql fitted.json -o features.sql        # one DuckDB query
perdix export-sklearn fitted.json data.csv -o plain.pkl          # plain scikit-learn, no Perdix needed
perdix apply r.toml customers.csv --table purchases=tx.csv -o out.csv  # second tables for point-in-time joins
perdix plugins                                       # steps loaded from ~/.perdix/plugins
perdix webapp -o perdix.html                         # the app as one HTML file that runs in a browser

Layout

perdix/core/            no Qt: params, registry, recipe, codegen, exports (script, sklearn, SQL), analysis,
                        evaluate, stability, suggest, search, gp, ai, contract, tables, chunked, io, demo
perdix/core/steps/      one module per category; each step = fit() -> state, transform(), code()
perdix/gui/             PySide6 + pyqtgraph: session (state, cache, undo), panels, editor, pages, theme, docs viewer
perdix/web/             the browser app: api.py (JSON calls into the core), app.html, worker.js, build.py
perdix/docs/            documentation (Markdown, shown in the app and on GitLab)
perdix/examples/        example cases: data.csv + recipe.toml each, generate.py rebuilds the data
                        (cmapss.py downloads and converts NASA's engine data, which are not shipped)
benchmarks/             speed benchmark that also checks every output is bit-identical to a recorded run
tests/                  step contract (stepcheck.py), analysis, CLI, GUI (offscreen)

A step is a class with parameter specs, fit(df) -> state (JSON-serialisable), transform(df, state) and code(state). Its transform is built from module-level helper functions. The script generator copies their source, so the script and the GUI share the code. To add a step, register a class in one of the steps/ modules and add a check_step call in the tests.

.venv/bin/python -m pytest -q
.venv/bin/ruff check perdix tests

License

Perdix is free software under the GNU Lesser General Public License, version 3 or later (LGPL-3.0-or-later); see LICENSE, COPYING.LESSER and COPYING. We may use it from code under any license, and changes to Perdix itself stay open.

What Perdix writes for us is ours: exported scripts, SQL, recipe files, pipelines and transformed data may be used under any terms, including the helper code copied into an exported script. LICENSE has the exact wording of this exception.