elessar 1.2.0

Automated exploratory data analysis and data-quality profiling.

1.2: its conditional guarantees and release gates are in the scientific contract and the release review record.

One function call (elessar.see(df), or elessar.see("sales.parquet")) or one command (elessar data.csv) gives you semantic types, descriptive statistics, the best-ranked candidate distribution, supported associations with conditional or approximate significance where statistically justified, target associations and leakage-provenance warnings, feature selection, PCA and factor analysis, outliers, time series, data-quality alerts and a preparation plan with the code to apply it, and report-ready charts.

In Tolkien's legend, the Elessar is the green Elfstone through which whatever is seen is made whole again.

Elessar reports an explicit decision path for semantic column typing. Other analyses use statistical tests, optimisation, fitted models and documented diagnostic rules; not all their results are yes/no decision trees. Their methods and limits are explained in How elessar decides.

Contents: What you get · Install · Quickstart · Any input, any size · API · How elessar decides · What it computes · Sample terminal output · Plots · Development

What you get

Pass a DataFrame, a file or almost anything tabular to see(), or a file to the elessar command, and get in your terminal and in one HTML page:

• the semantic type of every column (not just the dtype): id, boolean in disguise, numbers stored as text, dates stored as text in any format, emails, URLs, phones, postal codes, IPs, JSON, free text, ordinal scales (low/medium/high, months, weekdays, 1-5 ratings), counts, integers that are really category codes (region 11, store 7, MCC 5411), mixed types… with the certainty of each decision
• dates of birth told apart from event dates, and turned into ages
• full descriptive statistics adapted to each type, with confidence intervals
• likelihood-based numerical distribution candidates, ranked by AIC, and four normality diagnostics
• supported associations with inference under documented assumptions: Pearson, Spearman, Kendall, Cramér's V, correlation ratio, mutual information, AUC, with p-values, Benjamini-Hochberg correction and confidence intervals; in a table indexed by time, non-stationary columns are tested on their differences and numeric Pearson/Spearman tests use a lag-1 effective sample size; serial Kendall/categorical/mixed tests use conservative truncated time shifts with explicit stationarity conditions
• target analysis: feature ranking, leakage-suspect flags requiring provenance review (association alone cannot prove leakage), informative missingness
• feature selection by embedded methods (decision tree, random forest, lasso, ridge) and a filter, with tree depth and linear penalties tuned by cross-validation, model scores on held-out rows, and a vote per feature against shuffled shadows
• multivariate statistics: whether PCA is worth it (Bartlett, KMO, parallel analysis), PCA-based component extraction with varimax rotation (not a fitted common-factor model), exploratory composite scores and internal-consistency diagnostics (Cronbach's alpha / Spearman-Brown), redundant groups of columns and multivariate outliers
• outliers flagged three ways (3 sigma, boxplot, modified z); optional cleaning recommendations require domain review because an extreme measurement is not necessarily erroneous
• time series recognised on their own (single series, panel, event log): trend, stationarity, seasonality, decomposition, windows, validation folds on the real dates, augmentation recipes
• a preparation plan: every cleaning step (invalid values, wrong formats, misspellings, duplicates, consistency, missing data, outliers) and transformation (bucketing, decomposition, aggregation, combination, rescaling, normalisation, transformation, encoding, dimensionality reduction), each with its reason and its code
• data quality checks and actionable alerts (with the code to fix them): placeholders, duplicates, multicollinearity, missingness associated with observed columns (evidence against MCAR), sorted rows…
• a set of plots that actually carry information, saved as PNG files, and one HTML page with everything
• many tabular input formats, with resource-dependent limits: CSV, Parquet, Excel, JSON, SQLite, SAS, Stata… or polars, pyarrow, dask, NumPy objects; files bigger than memory are typed progressively and streamed in parallel, with exact row and missing-value counts over the full table (high-cardinality distinct counts are approximate)

It is built for scripts and the terminal: plain-text output that fits your terminal width, files on disk, no display needed (except to click the target, if you ask for it).

Install

pip install elessar

The development version: pip install git+https://github.com/thibautdqb/elessar.

One install brings everything: pandas, NumPy, SciPy, Matplotlib, scikit-learn, dask, pyarrow (Parquet) and openpyxl / python-calamine (Excel). Requires Python ≥ 3.10.

Big files and dask. dask is installed with elessar and used automatically for files too big to load whole: it splits the file into blocks that are read and summarised on every core. Nothing to configure. engine="pandas" / --engine pandas streams in a single process instead.

Quickstart

import pandas as pd
import elessar

df = pd.read_csv("titanic.csv")

report = elessar.see(df, target="Survived")  # prints the report, writes the charts and the report page,
                                             # then opens the page in your browser
report.alerts                                # [{severity, column, message}, ...]
report.summary                               # one row per column, one column per metric
report.correlations["target"]                # features ranked, with tests, p-values, AUC, MI, leakage flag
report.multivariate["factor"]["steps"]       # the factor analysis, step by step
report.out_dir                               # elessar_report/df_target-Survived_2026-10-29/

elessar.see("data/*.parquet", target="ask")  # any file, glob or folder; pick the target in a window
elessar.see(polars_df), elessar.see(arrow_table), elessar.see(dask_df), elessar.see(numpy_array)
src = elessar.load("export.xlsx", sheet="2024")  # just the loader: src.df, src.full (exact figures), src.origin

Or straight from the shell (python -m elessar works too):

elessar titanic.csv --target Survived      # with a target
elessar titanic.csv --no-target            # without one

Run python examples/demo.py to see it on a deliberately messy synthetic dataset.

The report page

When the analysis finishes, elessar writes report.html next to the charts (and the same content as report.md, which renders on GitHub, in VS Code, Obsidian…), and opens it in your browser. The page is a dark "data observatory": a sidebar with every section numbered (the one you are reading lights up), key figures at the top (rows, columns, missing cells, alerts, target), each chart in a card that opens full size on click, code blocks with a copy button, and collapsible sections per column; it works offline, in one file with --single-file, and on a phone. In order:

• key findings first: the features most associated with the target, then every high and medium alert
• overall statistics: the dataset overview, numeric and categorical columns at a glance, how each column was typed (its path down the decision tree and the certainty of the decision) and, for a sampled big table, the exact figures over every row
• target analysis: ranking with tests, AUC and mutual information, informative missingness, target charts
• feature selection: each method's importance and votes, model hyperparameters selected by training-only cross-validation, model scores on separate voting rows, and (when available) the independently held-out score of a refitted model on the frozen selected subset; plus the L1 path
• time series (when the table is one): structure, trend, stationarity, seasonality, decomposition and autocorrelation charts, the per-entity table of a panel, the treatment and the validation folds
• distributions: for every numeric column the normality tests, the best fit with its parameters, Akaike weight, ΔAIC to the runner-up and the size of the misfit (KS D), next to histogram + fit + Q-Q charts
• outliers: the three methods per column, a chart of every flagged point, the most extreme points by row, the rows that are outliers on several columns (outliers.csv lists every flagged point)
• data quality, including the checks on dates of birth
• multivariate statistics: PCA; the factor analysis as a checklist (each step, its question, its answer), the unrotated and rotated component matrices with communalities, Cronbach's alpha per factor with the alpha if each item is deleted, the new factors and their descriptives, and the code; groups of redundant columns; then the associations (heatmaps with significance, strongest pairs, linear dependencies, VIF)
• the preparation plan: cleaning then transformation steps, each with why and the code
• attribute statistics: one collapsible section per column with every statistic laid out in cards (center, spread, quantiles, shape, outliers, row order, labels, time, text), its normality tests, its top fits, its top values, its alerts with the fix, and its charts
• run details: source, how it was read, processes used, time per step, versions

StoneConfig(single_file_report=True) / --single-file also writes report_single.html with every chart inlined: one file to send or attach.

Where it is written. Each run gets its own folder, elessar_report/<name>_<target>_<YYYY-MM-DD>/, named after the file or the variable holding the DataFrame, and the target or its absence: elessar trans.asc -t type → elessar_report/trans_target-type_2026-10-29/, elessar.see(sales) → elessar_report/sales_no-target_2026-10-29/. Reports of different tables or targets never overwrite each other; running the same analysis again on the same day refreshes that day's report. out_dir=... / -o FOLDER writes to exactly that folder instead.

It opens automatically only when you run from a terminal, never in CI, tests or pipelines. Use open_report="md" / --open md to open the Markdown file instead, or open_report=False / --open none to never open it. report.to_markdown(path) and report.to_html(path) write it anywhere.

Choosing the target

The target is the column the others should explain (a label, a price, churn…). elessar never guesses it silently. Give it with target="Survived" / --target Survived, or let a person pick: target="ask" opens a small window listing every column, with the likely targets first (★, ranked by suggest_targets(df): names such as target, label, churn, price, few classes, last column; identifiers and free text last). Double-click a row or press Enter. Without a display it asks in the terminal. From the command line the picker opens by itself when you run elessar data.csv interactively without --target (--no-target skips it).

Any input, any size

see() and elessar accept a pandas DataFrame / Series, a dict, a list of records, a NumPy array, a polars DataFrame or LazyFrame, a pyarrow Table, a dask DataFrame, any object with the DataFrame interchange protocol, and paths: a file, a URL, a glob pattern (logs/2024-*.csv) or a folder.

Compressed files (.gz .bz2 .zip .xz .zst) are read directly; a file without a known extension is recognised by its content.

Big tables

A table with more than max_rows rows (default 1,000,000), or larger than half of the free memory, is never loaded whole. It goes through three steps:

1. Progressive typing. The types are decided before the full read, on as few rows as the evidence allows. Rows are reached directly (byte offsets in a text file, row groups in Parquet) and never read twice; the lines drawn from a text file are parsed together, so types come out as in a full read. When the first look finds a time axis, the extra rows are contiguous blocks, so series are never cut into scattered rows.
2. One full pass guided by those types, in chunks, on every core; every chunk is reduced to a small summary:
   • exact over every row: row count, missing values per column, min / max / mean / standard deviation (Chan's parallel algorithm) on the values the types say they are (numbers written as text parsed, dates read with their format), zeros, negatives, infinities, frequencies of low-cardinality columns, duplicate rows (64-bit row hashes), also ignoring the identifier columns;
   • type checks over every row: values that do not fit their column's type, with their row numbers, and the uniqueness of every identifier, which no sample can prove. A decision that does not survive is corrected (an "identifier" that repeats becomes a grouping key) and reported;
   • for a time-indexed table, the series aggregated per period over every row (counts, sums, means), and for a panel the active entities per period and whole entities drawn at random (Bernoulli sampling on a hash of the entity key), so temporal statistics never rest on a thinned sample;
   • distinct counts: exact below 4,096 values, estimated above (k-minimum-values sketch, ~1.6 % error);
   • a uniform random sample of max_rows rows without replacement (the rows with the smallest random keys), in the original order and indexed by original row number, on which the row-level profile runs.
3. The profile, on the sample for row-level statistics and on the exact series for time.

Chunks are independent, so they are reduced on every core: CSV files are split into byte ranges by dask, Parquet files by row group, in-memory frames by slice. The report says the profile is sampled and shows the exact table.

Parallelism. The heavy steps (per-column statistics and distribution fits, every association test, the advanced analyses, every chart) run as independent tasks on forked processes, one per core: the DataFrame is inherited, never copied or pickled, and BLAS threads are limited to one per process. Results are identical to a single-process run (a test checks it). Small tables stay in one process; StoneConfig(n_jobs=1) / -j 1 disables it, n_jobs=k caps it.

On macOS and Windows, where a process cannot be forked safely (Windows has no fork; on macOS it can crash the numerical libraries), the chunks of a big file are read and summarised by fresh processes on every core instead (they are self-contained: a byte range or a row group), even from a script without an if __name__ == "__main__": guard; the other steps run in one process. Results are the same everywhere.

Historical 1.1 timings, not remeasured for the preliminary changes. They are not performance claims for the new tuning pipeline.

API

Every function accepts a StoneConfig to change thresholds (alpha, missing %, correlation and VIF thresholds, rare-category %, sample sizes, max_rows, probe_max_rows, candidate distributions, …) or to set a column's type by hand: StoneConfig(column_types={"region": "categorical", "cylinders": "numeric_discrete"}) (numeric_continuous, numeric_discrete, categorical, ordinal-candidate, id or text) replaces the decision tree for those columns. StoneConfig(descriptions={"A11": "average salary"}) shows what each column means in every report and in the data dictionary; reference_date="1999-01-01" computes ages at that date for an old extract; theme="light" draws light charts and a light page (print, PDF); report="summary" keeps the column charts only for columns with a high alert or among the features most associated with the target.

Command line

elessar SOURCE [-t TARGET ... | --no-target] [-o FOLDER] [--type COL=TYPE ...] [--describe FILE.csv]
               [--dictionary] [--summary] [--light] [--reference-date YYYY-MM-DD] [--no-plots] [--single-file]
               [--sheet S] [--table T] [--sep SEP] [--encoding E] [--columns a,b] [--nrows N]
               [--max-rows N] [--engine {auto,dask,pandas}] [-j JOBS] [--sample N] [--alpha A]
               [--width W] [--json] [--markdown] [--csv] [-q] [--open {auto,html,md,none}]

SOURCE is a file, URL, glob pattern or folder in any format above. -t ask opens the target picker (it opens anyway when you run interactively without -t). --type COL=TYPE (repeatable) sets a column's type by hand. -V / --version prints the version. Without -o, the report goes to its dated folder in elessar_report/ and the command prints where. -t repeated (-t loan_1997 -t card_1997) runs one analysis per target, each in <out>/target-<name>. --describe reads a two-column CSV (column, description); --dictionary writes dictionary.csv and dictionary.md.

Vision

Several tables

elessar.see_relations(
    {"account": "account.csv", "loan": "loan.csv", "trans": "trans.csv"},
    keys=["loan.account_id -> account.account_id", "trans.account_id -> account.account_id"],
    dates={"account": "date", "loan": "date", "trans": "date"},
)

Every declared key is checked over every row: whether the parent column is a key, the orphans (child values with no parent), children without a key, parents without a child, children per parent (1:1 or 1:n), and, with a date on both sides, the children dated before their parent. Children are read block by block (CSV, Parquet): a 106-million-row child is checked in about 30 seconds with 0.6 GB of memory.

Keys that are not declared are suggested among integer label columns, when chance cannot explain the match: the same values shifted circularly within the parent's range (by 1 to 199) are looked up too. Identifiers of unrelated tables overlap a lot in the same low range; a shift keeps that overlap and breaks a real key. A parent that fills its whole range (77 districts numbered 1 to 77) cannot be told from chance and is not suggested, and text keys are never guessed: declare them.

How elessar decides

Each tree below is the code's own order of questions; the first yes decides. Thresholds are constants in the code and fields of StoneConfig. How well these decisions hold against a known truth (type I errors, number of factors, trends under autocorrelation, public datasets) is measured in VALIDATION.md.

1. The type of a column

A column is first routed by how it is stored, then goes down the numeric or the text branch:

Every decision carries its certainty, shown in the report next to the path:

• certain: every value was seen, or, on a sample, each threshold test ("at least 90 % of the values are dates") holds at 95 % confidence (the Wilson interval of the observed share does not cross the threshold) and no label is likely unseen (Good-Turing estimate of the unseen probability mass, values seen once / values seen, below 1 %);
• probable: an identifier, empty or constant column decided on a sample: only every row can prove it, and the full pass does; or integers read as category codes: no number of rows can prove what an integer means;
• uncertain: a test whose interval crosses its threshold, a share of distinct values within 5 points of its threshold (not a proportion of independent draws, so no interval is claimed for it), few values, or an ambiguous day / month order. On a big file, uncertain columns are typed again on more rows (above).

Integers that are codes. A region 11, a store 7, a payment type 3, a merchant category 5411 or a postal code 75011 are labels written with digits: their mean, standard deviation, correlation or "outliers" mean nothing, and they belong with the categories (Cramér's V, chi-square, one-hot or target encoding). They are recognised:

• by name: the first or last word of the column name is a category word (region, store, dept, zip, mcc, naf, country, sex, gender, status, type, category, segment, cluster, group, mode, payment, channel, brand, origin… or code with another word: code_region, mcc_code), unless a word says the column counts or measures (n, count, total, price, amount, age, year, score, rating, level, defects, visits…: store_count stays a number);
• by values alone: at most 50 distinct fixed-width integers of 3 digits or more, not years (1800-2100) and not rounded amounts (all multiples of 5), such as merchant or activity codes.

A graded scale starting at 0 in steps of 5 (0 / 5 / 10) is a score, never a code. These decisions are marked probable, grouped in one alert, and can be reversed per column: StoneConfig(column_types={"store": "numeric_discrete"}) or elessar data.csv --type store=numeric_discrete. Checked on 1,007 columns of 65 real tables: the only columns that change are gender coded 1 / 2 (three psychology datasets), and counts (SibSp, cylinders, defect counts), years, prices, pulse rates and test scores stay numbers.

Dates, whatever the format: ISO 8601 with or without time and zone; day-first or month-first with any separator (the order that parses every value wins, and a column where both work is reported as ambiguous); month names in English, French, German, Spanish, Italian, Portuguese, Dutch and Czech; compact digits as text or numbers (YYYYMMDD, YYMMDD, DDMMYYYY, YYYYMM, YYYYMMDDHHMM(SS), Czech / Slovak birth numbers with month + 50); quarters (2020Q1) and ISO weeks (2020-W05); Unix timestamps in s / ms / µs / ns; Excel serial days when the column name says it is a date. Two-digit years never land in the future. Zero dates (0000-00-00, 00/00/0000) are placeholders for a missing date, never values. The suggested conversion uses the exact format found, and elessar.parse_dates(series) does it in one call.

2. Is a date a date of birth?

A date of birth is a date, but not a time axis: it says who a row is about, not when something happened, and what models need is the age.

Each date of birth gets a derived age column, profiled and tested against the target like any other, computed row by row at another event of the row when the table has one (age at admission, at the start of a term, at an award year), else on the day of the analysis (which the report says). It is never used as a time axis, the preparation plan models the age instead of the calendar parts, and it is checked: births after the reference, ages above 120 at an event, an excess of 1 January or of first days of the month (one-sided binomial test against the calendar share: day or month unknown), agreement with an age column (a constant gap dates the extract), the share of minors, and a reminder that a date of birth is personal data and a strong quasi-identifier.

3. Is the table a time series, and of which kind?

Candidate time axes: a datetime column, dates as text, a datetime index, a year column, or year + month columns (never a date of birth).

Then, per series:

• trend: Mann-Kendall test corrected for autocorrelation (the uncorrected p-value of an autocorrelated series is far too small; the report shows both): the variance is inflated as in Hamed-Rao, by the larger of two autocorrelation models, an AR(1) corrected for its small-sample bias and an AR(p) chosen by AIC (6% false trends at 5% under AR(1) 0.8 noise, at most about 10% on other short-memory noise, against 25% for the textbook Hamed-Rao); Sen's slope;
• seasonality: a period natural to the frequency (7 / 30 / 365 days, 24 / 168 hours, 12 months, 4 quarters…) is screened on the autocorrelation of the detrended series (a peak, a dip before it, two full cycles) and kept when its seasonal strength from a classical decomposition reaches 0.25. The decision rests on this effect size: a significance test would itself be invalidated by the autocorrelation;
• stationarity: ADF (H0 unit root) and KPSS (H0 stationary), with and without a linear trend, read together on the seasonally adjusted series: stationary, trend-stationary (detrend, do not difference), unit root (difference) or inconclusive (break, changing variance, long memory);
• for a panel, the same tests on each entity's own series (whole entities drawn at random), with the share of entities with a significant trend after a Benjamini-Hochberg correction: a mean across entities would mix their dynamics with the change in which entities are present.

Treatment: ordering, gaps, aggregation, panel-safe lags, differencing or detrending as the tests say, Fourier terms, window and lag sizes, expanding-window validation folds computed on the real dates, augmentation for the training folds (jittering, scaling, window slicing, magnitude warping). The native resolution is kept up to 200,000 periods.

4. Is PCA worth it?

On the standardised numeric columns (target, identifiers, booleans and constants left out, missing values replaced by the median):

The report also gives Kaiser's rule and the components needed for 80 / 90 / 95 % of the variance.

5. PCA-based exploratory factor extraction, step by step

PCA-based exploratory component extraction with varimax is not a fitted common-factor model or a proof of construct validity. Each step is a line of the report's checklist, with its question, its answer and a yes / no / removed mark.

1. Preparation. Items are quantitative or Likert variables: ordered labels such as disagree < neutral < agree become their ranks; identifiers, booleans, dates, free labels and the target are left out, and so are columns that describe the respondent rather than measure a construct (age, sex, education, income, year…, recognised by their whole name: housing_median_age stays). When most of the variables are integer items on one response scale (a questionnaire answered 1 to 6), that questionnaire is analysed alone. Declared missing codes (-999…) become missing values, rows are deleted listwise, and an item that is an exact combination of others (a total, a mean of sub-items: VIF above 100) is removed, since it makes the correlation matrix singular. The determinant is reported against Field's 0.00001. Same scale for every item, or items standardised for the reliability and the scores. At least 100 complete rows and about 10 per item, otherwise the result is indicative.
2. Extraction: principal components of the correlation matrix, unrotated solution, scree plot, varimax rotation with Kaiser normalisation, loadings sorted by size.
3. Pertinence: KMO above .50 and Bartlett's test below .05, on all the items and again on the items kept.
4. Communalities (extraction): an item explained below .40 (Costello & Osborne, 2005) is removed and the analysis run again, one item at a time, the weakest first.
5. Number of factors: Horn's parallel analysis decides, keeping the components whose eigenvalue beats the 95th percentile of those of random data of the same size. Eigenvalue above 1, cumulative variance of at least 60 % and the elbow of the scree plot are shown beside it: on 300 simulated structures the parallel analysis found the right number in 98 % of them, against 94 % for the rule "two of these three agree, else the scree plot" (Kaiser and the 60 % target overestimate it in most cases). Then coherence: a factor resting on a single item is not a construct, and several items at fault at once usually mean too many factors rather than bad items, so one factor fewer is tried and kept when it leaves fewer items at fault.
6. Rotated component matrix: each item goes to the factor of its highest loading; an item whose highest loading is below .40, or that loads at least .40 on a second factor within .20 of the first (cross-loading), is removed and the analysis run again. Each factor is oriented like most of its items, so the minority shows as reversed, and is named after its items.
7. Cronbach's alpha per factor, reversed items flipped, with the alpha if each item is deleted: excellent ≥ .90, very good ≥ .80, good ≥ .70, acceptable ≥ .60, poor ≥ .50, else unacceptable. A deletion that would raise alpha by .05 or more is proposed, never applied: it must also make sense for the construct. A two-item factor is judged on the Spearman-Brown coefficient 2r / (1 + r) instead, which alpha understates for two items (Eisinga, te Grotenhuis & Pelzer, 2013).
8. New factors, only for the scales with an acceptable reliability: the mean of their items, with the code for the mean of the answered items and for the version requiring every item (MEAN.k).
9. Descriptives of the component scores: mean, median, mode when uniquely meaningful, standard deviation.

The exploratory verdict is recommended when components pass the heuristics and internal-consistency checks; Reliability alone does not establish unidimensionality or construct validity. The verdict is indicative when the sample is small and not useful when KMO or Bartlett fail or no internally consistent composite can be identified.

Checked on reference data sets whose structure is known: the 25 Big Five items of psych::bfi give the five factors with the published keying (E1, E2, C4, C5, O2, O5 and A1 reversed) and alphas of .61 to .82, age and education left out; Holzinger and Swineford's tests give the verbal, speed, spatial and memory factors; the mood adjectives of psych::msq give the two arousal dimensions (energy and tension); one-factor scales (Davis's drive for thinness, the breast-cancer biopsy scores, SAT / ACT, the beauty ratings of evals) give one factor; aggregated trait scores (TIPI) and unrelated measures are found not useful.

6. What to do with an outlier

Per numeric column, three methods: 3 sigma (|x − mean| > 3 sd), the boxplot fences (Tukey, 1.5 IQR) and the modified z-score (Hampel, |0.6745 (x − median) / MAD| > 3.5). Every flagged point is listed with its row, value, z and robust z. A value is a likely entry or unit error when it lies more than 10 robust standard deviations away, measured on the log scale for a positive, right-skewed column, and the rest of the column has light tails (excess kurtosis below 1 plus three standard errors): the long natural tail of incomes or durations, or the heavy tails of a Student distribution, are then not mistaken for errors (removal advised for 0 to 0.7 % of clean columns, against 7 to 70 % before 1.1), while a value multiplied by 100 in a light-tailed column still is. In a heavy-tailed column such a value is still flagged, with the advice to check it.

Multivariate outliers: robust Mahalanobis distance (minimum covariance determinant on Yeo-Johnson-normalised continuous columns, near-duplicates and linear combinations removed) beyond the chi-square 99.9 % quantile, confirmed by an isolation forest; flagged as indicative when more than 15 % of rows exceed it.

7. Which features to keep

With a target, the features (numeric, and categorical one-hot encoded but scored as one) go through five methods, after configured provenance exclusions and observed exact target aliases are quarantined on training rows (a strong association alone does not imply leakage):

A shadow is a feature with its rows shuffled: same distribution, no link with the target (as in Boruta). The tree, the lasso and ridge are refitted with one shadow per feature, at the setting they tuned, and vote for a feature only when it beats every shadow. Selected by 3 methods or more = keep, 2 = borderline, else drop. About 80 % of rows train the selector, 10 % vote and 10 % form a separate final test. Partitions are fixed without consulting target values. Leakage exclusions, the filter and category vocabulary use training rows. During tuning, imputation and scaling are fitted inside each fold; the forest uses fixed settings. Lasso uses a fixed alpha grid from 1e-6 to 1e6; ridge and logistic regularisation also use explicit grids. These finite grids can miss an appropriate setting for unusual scales or difficult tasks.

Rows with dates are sorted in time. Dates remain indivisible in both the holdout and internal folds; the holdout fraction and number of folds can change when necessary, and selection is unavailable when a valid split cannot be formed. Automatic differencing is decided on outer training rows, then applied causally within each entity. Scores are R², ROC AUC for a binary target, or balanced accuracy.

Score scope: scores diagnose the base models on voting rows, which also determine forest importance. consensus_evaluation separately assesses the final kept subset with a predefined random forest (100 trees, leaf size 3, fixed feature-sampling rule), refitted on training + voting rows and scored once on the final test. An empty subset uses a training-prior dummy model. The final test never votes or tunes a model. n_holdout remains the total reserved from tuning, now divided into n_voting and n_final_test. The full report learns selection types on training rows; callers supplying types directly must fix them independently of final outcomes. Small tables, impossible temporal cuts, externally supplied difference policies or undefined metrics report the independent assessment as unavailable, with a reason. High target association raises a provenance warning and retains near-deterministic predictors. Known post-outcome variables can be named in StoneConfig(exclude_features=("column",)). Observed exact affine or low-cardinality bijective target aliases are quarantined by default on training rows; this explicit policy can be overridden with exclude_target_aliases=False when pre-outcome availability is established. Association alone does not establish leakage: inspect provenance and availability at prediction time.

Screening/stationarity choices are not all refitted inside every inner fold. The unused final test measures the outcome of that complete adaptive procedure; it does not make the internal CV fully nested. A chronological test estimates future-block performance without assuming its rows are IID. See the serial and consensus validation record for executed checks and limits.

What it computes

Semantic types: empty, constant, boolean (including 0/1, yes/no, true/false, y/n, oui/non), id (unique or near-unique, consecutive ints, UUIDs), numeric_continuous, numeric_discrete (counts, few levels), categorical (labels and high-cardinality codes), ordinal-candidate (low/medium/high, sizes, Likert, months, weekdays, quarters, 1-5 integer scales, ordered categoricals), datetime, datetime_string, numeric_string ("1 234,5", "12%", "$3.2"), text, multi_label, email, url, phone, postal_code, ip_address, json_like, mixed. Numbers written with units ("90 min", "60 kg") are parsed, and mixed units in one column (min and Seasons, kg and lbs) are flagged.

Statistics

• all columns: count, missing, unique, memory, modes, top-N values
• numeric: caps / floors (censoring: values piled up at the max or min, tested again on one value per group when the column is an attribute of a key, so one large district holding the maximum is not censoring), zero inflation, mean (95 % CI), median (95 % CI from order statistics), std, var, sem, CV, min/max/range/sum, quantiles 1–99 %, IQR, MAD, 10 % trimmed mean, geometric and harmonic means, skewness, excess kurtosis, zeros, negatives, infinities, outliers (1.5×IQR, |z| > 3, modified z > 3.5), decimals, heaping (excess of round values, mixed-precision aware), monotonicity, lag-1 autocorrelation in row order with its p-value, bimodality coefficient, number of KDE modes
• categorical / boolean: cardinality, entropy, normalized entropy, Gini impurity, imbalance ratio, chi-square test of equal frequencies, share of True with its Wilson CI, rare categories, labels that differ only by case / spaces / accents, string lengths
• datetime: range, span, inferred frequency, largest gaps, duplicated timestamps, future dates, trend in rows per period (Kendall tau + p), distribution by year / month / weekday / hour (only when not trivial)
• text: length and word counts, empty / whitespace-only strings, share of digits / uppercase / punctuation, most common tokens

Distributions: Shapiro-Wilk, D'Agostino K², Jarque-Bera and Anderson-Darling with an exploratory majority verdict (not a combined calibrated test); likelihood-based numerical fits of norm, lognorm, expon, gamma, weibull_min, beta, uniform, logistic, t, laplace, pareto, chi2, cauchy (continuous) or poisson, nbinom, geom, binom (discrete), with support handling (shifted / rescaled fits), ranked by AIC, with ΔAIC, Akaike weights, BIC and CDF-gap diagnostics. Calibrated fitted-model ks_pvalue is unavailable: the same-data approximation is retained only as diagnostic_pvalue, with its fitting/selection limitations. Integer data use both sides of jumps on the integer support and a grouped chi-square diagnostic. These are not refitting-aware calibrated goodness-of-fit tests. A value holding 30 % or more of a continuous column (rain: 57 % exact zeros) is reported as a point mass and the candidates are fitted to the rest; fits that collapse onto repeated values are rejected. Each candidate has a timeout. The binomial candidate profiles its likelihood over integer trial counts up to 100,000, with p=mean/n. The search domain is reported with the fit; a maximum at the upper search limit is rejected. This is not a claim of an unconstrained maximum over all integers, and a scientifically known trial count should take precedence in a modelling analysis.

Associations and significance (see_corr keys)

For serial Kendall/categorical/mixed associations, effects and tests use the same fixed central window of original levels. There is no adaptive stationarity/differencing pretest for TTS. The method tests independence of the entire processes, requires at least one to be strictly stationary and assumes a fixed statistic/type policy. Automatic typing or ADF/KPSS cannot certify these conditions. It is not a general test for two nonstationary processes or for conditional/causal dependence.

The conservative rule is min(1, max(B_forward, B_reverse)/(r+1)), including ties and zero shift. Radius r=min(serial_shift_radius, n_time//4) depends only on length. Defaults are 99 and at most 2,000 consecutive time points (serial_max_points), within the row sampling budget. No circular wrapping or random temporal thinning is used. Balanced panels shift whole entity vectors synchronously, preserving cross-entity dependence. Missing observations, irregular grids and unbalanced panels return unavailable p-values instead of an IID fallback. With radius 99 the smallest p-value is 0.01: multiplicity can substantially limit power. Choose larger fixed budgets before analysing results when necessary. See Yuan & Shou (2024). For missingness, the complete indicator is tested; gaps in its comparator still make inference unavailable. Numeric-target omnibus statistics are recomputed for every surrogate. The signed Spearman coefficient remains a descriptive effect.

Groups of columns with mean |Spearman| ≥ 0.7 are proposed for combination.

Preparation plan (exploratory; generated code is not a leak-proof model pipeline and outlier actions require source review): cleaning: invalid values (placeholders, impossible values, infinities), wrong formats (numbers or dates as text, mixed types, stray spaces, a file without a header row), misspellings (case, accents, spaces, and 1-2 typos found with a BK-tree on the edit distance), duplicates (also ignoring identifier columns, compared with the count chance gives: tables drawn to keep each column's values and its strongest link to another column, a Chow-Liu tree; only an excess is a record entered twice), consistency (mixed units, values on two scales ~100× or 1000× apart), missing data (strategy by type and by missingness mechanism), outliers to remove or reassign. Transformation: bucketing (numeric cut points from a decision tree on the target; labels merged by an adaptive binary tree whose ordering and cuts are selected on one half of the rows and confirmed on independent rows (at most seven tests with a Bonferroni familywise bound under IID assumptions), decomposition (dates into calendar parts, dates of birth into ages, lists into 0/1 columns, cells holding several data items such as "Paris, France" or "1920x1080" split), aggregation (time series, panels), combination (redundant groups, exact linear relations, factor scores), rescaling, normalisation, transformation of skewed columns (Box-Cox / log / Yeo-Johnson, with the skewness before and after), encoding (by cardinality and type), dimensionality reduction (PCA or factor scores).

Quality: duplicate rows and columns, constant and quasi-constant columns, high missing, missing-value patterns (columns that go missing together, rows with many missing, most frequent patterns), mixed types, leading/trailing whitespace, placeholders used as missing ("", "NA", "null", "?", -999, 99999, 0000-00-00…), suspicious values (negative ages or prices, default dates like 1970-01-01), dates of birth (births after the reference, impossible ages, 1 January excess), id-like columns, redundant columns, multicollinearity, sorted rows, imbalanced target, dtype optimizations with the estimated memory saving.

Sampling inside the profile: on top of max_rows (above), expensive steps (correlations, Kendall, KDE, pairwise tests, multivariate analyses) use at most sample_size rows (default 100 000); distribution fits and normality tests use at most fit_sample_size values (default 20 000). Sampling is deterministic (random_state).

Robustness: elessar runs on any DataFrame, whatever the column names (empty, None, NaN, numbers, tuples / MultiIndex, duplicates, slashes, newlines, emoji, pandas method names like count…) and whatever the content: empty frames, a single row, all-NaN columns, inf and 1e±300 values, huge cardinality, lists / dicts / bytes in cells, mixed types, sparse, nullable, categorical, period, interval, complex, Decimal, tz-aware datetimes, timedeltas, duplicated or MultiIndex row index. The target can be given by its original label (tuple, number, timestamp…). A randomized test suite generates hostile frames on every CI run. A failing step on one column is recorded as an error entry and an alert; it never stops the report.

Real data: historically exercised on 55 public datasets (not all preserved as reproducible fixtures); the versioned validation suite independently rechecks the datasets listed in VALIDATION.md (Titanic, adult census, California housing, diamonds, Pokémon, Vega movies, NYC taxis, penguins, wine quality, Netflix, Olympic athletes, Spotify, Nobel laureates, members of Congress, a 62-column MultiIndex brain-network table…), on the Berka bank data (1 million transactions) and on an 8-million-row file; the statistical checks against a known truth (simulations, Holzinger-Swineford, the Big Five questionnaire, ten seaborn datasets) are measured in VALIDATION.md. Things it finds there unprompted: Pokémon Total = 1·HP + 1·Attack + 1·Defense + 1·Sp. Atk + 1·Sp. Def + 1·Speed; housing values capped at 500,001 and ages at 52; adult capital gain top-coded at 99,999 and 92 % zeros; Titanic survival 30 % vs 67 % depending on whether Cabin is missing; taxi tips always 0 when the payment type is missing; movie titles stored as numbers (300, 1776); Nobel laureates' and congressmen's birth dates, with their ages at the award or at the start of the term; the five personality factors of the Big Five questionnaire; files sorted by price or year.

Historical sample terminal output (v1.1; not a current v1.2 validation trace)

Historical 1.1 excerpt from the Kaggle Titanic training set (elessar train.csv --target Survived). These illustrative numbers are not validation results for the preliminary changes above:

════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
 elessar report  ·  891 rows x 12 columns
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════

OVERVIEW
────────
  source          csv file · 58.9 KB · delimiter comma · read in 0.0s
  rows            891
  columns         12
  memory          118.9 KB  (suggested dtypes save ~53.7 KB)
  pandas dtypes   int64=5, str=5, float64=2
  semantic types  categorical=4, numeric_continuous=2, numeric_discrete=2, id=1, boolean=1, ordinal-candidate=1,
                  text=1
  duplicate rows  0 (0.0%)
  missing cells   8.1%  (708 rows with at least one)
  target          Survived

ALERTS (15: 1 high, 3 medium, 11 info)
──────────────────────────────────────
  HIGH    Cabin        77.1% missing -> drop, or impute and add a missing-indicator
  MEDIUM  Age          missingness is informative about 'Survived': Survived=1: 29% when missing vs 41% when present
                       (p_adj=0.0059) -> add a missing-indicator feature
  MEDIUM  Cabin        missingness is informative about 'Survived': Survived=1: 30% when missing vs 67% when present
                       (p_adj=6e-21) -> add a missing-indicator feature
  MEDIUM  Name         2 values with leading/trailing whitespace -> .str.strip()
  info    -            dtype optimizations save ~53.7 KB: PassengerId -> uint16, Survived -> uint8, Pclass -> uint8,
                       Sex -> category, Age -> float32, SibSp -> uint8, Parch -> uint8, Fare -> float32 (+1 more)
                       (see quality['dtype_optimizations'] for code)
  info    -            missing values are not random in 2 columns: Cabin (on Pclass), Age (on Embarked) -> see
                       correlations['missingness']; impute conditionally
  info    -            PCA recommended: 2 components (parallel analysis) keep 66% of the variance of 5 columns (KMO
                       0.54 miserable, Bartlett p 2e-153)
  info    -            104 rows are outliers on several columns at once (robust Mahalanobis + isolation forest) ->
                       inspect them (see Outliers)
  info    -            3 columns are heavily skewed (|skewness| > 2): SibSp, Parch, Fare -> details per column
  info    Cabin        high cardinality (147 categories) -> group rare levels or use target/frequency encoding
  info    Fare         13.0% outliers by the 1.5xIQR rule (outside [-26.72, 65.63])
  info    PassengerId  identifier-like column -> exclude from modelling and statistics
  info    SibSp        5.2% outliers by the 1.5xIQR rule (outside [-1.5, 2.5])
  info    Survived     feature selection (5 methods): drop SibSp, Parch; keep 5
  info    Ticket       high cardinality (681 categories) -> group rare levels or use target/frequency encoding

COLUMN SUMMARY
──────────────
  column       type                missing  unique            mean/top  std/top%   min    max   skew
  ───────────  ──────────────────  ───────  ──────  ──────────────────  ────────  ────  ─────  ─────
  PassengerId  id                     0.0%     891                   1      0.1%
  Survived     boolean                0.0%       2                   0     61.6%
  Pclass       ordinal-candidate      0.0%       3               2.309    0.8361     1      3  -0.63
  Name         text                   0.0%     891  Braund, Mr. Owen …      0.1%
  Sex          categorical            0.0%       2                male     64.8%
  Age          numeric_continuous    19.9%      88                29.7     14.53  0.42     80   0.39
  SibSp        numeric_discrete       0.0%       7               0.523     1.103     0      8   3.70
  Parch        numeric_discrete       0.0%       7              0.3816    0.8061     0      6   2.75
  Ticket       categorical            0.0%     681              347082      0.8%
  Fare         numeric_continuous     0.0%     248                32.2     49.69     0  512.3   4.79
  Cabin        categorical           77.1%     147                  G6      2.0%
  Embarked     categorical            0.2%       3                   S     72.4%
...

COLUMN DETAILS
──────────────
...
▸ Age  ·  numeric_continuous  ·  float64  ·  float64 dtype: 88 distinct real values
    missing 177 (19.9%) · unique 88 (12.3%) · memory 7.0 KB · mode 24.0 ×30
    mean 29.7 ± 14.53 · 95% CI [28.63, 30.77] · median 28 [27, 29] · trimmed 29.27 · IQR 17.88 · MAD 9 · CV 0.4891
    geo mean 24.43
    min 0.42 · p1 1 · p5 4 · p25 20.12 · p75 38 · p95 56 · p99 65.87 · max 80 · sum 21,205.2
    skew 0.389 · kurtosis 0.178 · outliers IQR 1.5% · |z|>3 0.3% · mod-z 0.1% · zeros 0 · negatives 0 · decimals 1
    heaping ×1.0 · KDE modes 1 · bimodality coef 0.36 · lag-1 autocorr 0.05 (p 0.19)
    normality: not normal · Shapiro p 7e-08 · D'Agostino p 1e-04 · Jarque-Bera p 8e-05 · Anderson p 0.01
    shape: roughly symmetric, best fit normal(loc=29.7, scale=14.5)
    #1 norm(loc=29.7, scale=14.5) AIC 5,850.5 KS p 0.005
    #2 t(df=22.9, loc=29.5, scale=13.9) AIC 5,851.3 KS p 0.016
    #3 logistic(loc=29.1, scale=8.15) AIC 5,851.4 KS p 0.046
...

CORRELATIONS
────────────
  28 pairs tested, 23 significant after Benjamini-Hochberg correction (alpha 0.05); strongest shown below
  column A  column B  method      value  95% CI            other             n   p_adj  sig
  ────────  ────────  ─────────  ──────  ────────────────  ──────────────  ───  ──────  ───
  Pclass    Fare      spearman   -0.688  [-0.725, -0.648]  pearson -0.549  891  2e-124  *
  Survived  Sex       cramers_v  +0.543                                    891   5e-58  *

  multicollinearity (VIF): Pclass 1.7, Fare 1.6, Age 1.3, SibSp 1.3, Parch 1.2

  missingness that depends on other columns (evidence against MCAR):
    Cabin <- Pclass: median Pclass = 3 when missing vs 1 · p_adj 3e-91
    Age <- Embarked: 64% missing when Embarked=Q vs 14% when Embarked=S · p_adj 2e-23

TARGET ANALYSIS
───────────────
  target: Survived  (boolean)
  feature   type                method     association  test            n   p_adj  sig    AUC     MI  flag
  ────────  ──────────────────  ─────────  ───────────  ────────────  ───  ──────  ───  ─────  ─────  ────
  Sex       categorical         cramers_v       +0.543  chi-square    891   3e-58  *           0.218
  Pclass    ordinal-candidate   eta              0.338  mann-whitney  891   1e-23  *    0.319  0.084
  Fare      numeric_continuous  eta              0.257  mann-whitney  891   1e-21  *    0.692  0.101
  Embarked  categorical         cramers_v       +0.166  chi-square    889   3e-06  *           0.021
  Parch     numeric_discrete    eta              0.082  mann-whitney  891   5e-05  *    0.561  0.024
  Age       numeric_continuous  eta              0.077  mann-whitney  714    0.16       0.469  0.019
  SibSp     numeric_discrete    eta              0.035  mann-whitney  891  0.0094  *    0.543  0.033
  6/7 features significantly associated (BH-adjusted p < 0.05)

  is the target different when a feature is missing?
    yes  Cabin (77.1% missing): Survived=1: 30% when missing vs 67% when present · p_adj 6e-21
    yes  Age (19.9% missing): Survived=1: 29% when missing vs 41% when present · p_adj 0.0059

FEATURE SELECTION
─────────────────
  task            binary classification, 891 rows
  scores          held-out roc_auc: decision tree 0.818, random forest 0.831, lasso 0.843, ridge 0.847 (tuned by
                  5-fold CV on 712 rows, scored on 179 held-out rows)
  keep            Sex, Pclass, Embarked, Fare, Age
  borderline      -
  drop            SibSp, Parch

MULTIVARIATE
────────────
  PCA             recommended: 2 components (parallel analysis) keep 66% of the variance of 5 columns (KMO 0.54
                  miserable, Bartlett p 2e-153)
  factors         not useful: KMO 0.49 and Bartlett p 2e-122: the variables do not share enough variance for factors
  outlier rows    104 flagged by robust Mahalanobis and isolation forest
  outlier points  Age 2 (reassign), SibSp 30 (reassign), Fare 116 (reassign)
...

Plots

Report-quality PNGs (144 dpi), in the same dark design as the report page: a night surface, cyan for the data, neon pink for fitted curves and highlights, mint for what is kept, a cyan ↔ pink diverging scale for signed values. Each chart has a header with the column or question as title and the key numbers as subtitle, the chart, and a footer note explaining the method. Compact axis labels (12.5k, 3.4M), direct value labels instead of redundant axes, a categorical palette checked for colour-vision deficiencies on the dark surface, one axis per chart, and log axes chosen automatically for heavily skewed data. Only plots that bring information are drawn: nothing for empty, constant or id columns, no missing-values matrix when nothing is missing, no heatmap with fewer than 3 columns. Files are named <NN>_<column>_<kind>.png in the report folder.

• dataset overview: one row per column with its type, missing share, distinct count, a summary and a sparkline
• numeric: histogram in rows with the best fit drawn as expected counts, boxplot with mean and outliers, Q-Q plot against the best fit with a 95 % envelope (points outside it highlighted)
• categorical / boolean: top-N bars with shares and counts, sorted by frequency (or in scale order for ordinals), "other" and "missing" shown apart
• datetime: rows per period at an automatic resolution, rolling mean, peak annotated
• text: lengths in characters and words
• dataset: missing-values matrix, missingness correlation, Pearson / Spearman / Cramér's V / all-type association heatmaps (non-significant cells faded)
• pairs: joint plots with marginal distributions, hexbin density for large data, least-squares line and a median trend (honest for non-linear relations), levels jittered for discrete variables
• target: "what drives the target" ranking (non-significant greyed out), and per-feature plots: joint plot, boxplots sorted by median with individual points, or stacked shares with % labels
• feature selection: importance of every feature by every method with the selections boxed and the votes; the lasso path
• PCA: scree plot (variance per component, cumulative, parallel-analysis threshold, components kept) beside the loadings
• factor analysis: scree plot with the parallel-analysis threshold that sets the number of factors, the Kaiser (eigenvalue 1) and 60 % lines for reference and the number of factors kept, beside the rotated component matrix, items grouped by factor; alphas in the subtitle, removed items in the note
• outliers: every numeric column on one robust scale with its boxplot fences, the points flagged by 1 or 2-3 methods, the most extreme labelled by row; multivariate outliers: robust distance of every row against its cutoff
• time series: each series with its rolling mean and its trend / season / stationarity verdict, the autocorrelation with Bartlett's significance band and the seasonal lag, and the seasonal decomposition (observed, trend, seasonal, residual)

Development

git clone https://github.com/thibautdqb/elessar && cd elessar
pip install -e ".[dev]"
pytest                                        # unit tests
pytest -m validation tests/validation         # statistical validation by simulation (potentially tens of minutes)
ruff check src tests docs examples
python examples/demo.py

The animation at the top is generated from a real run of the command line and a screenshot of the report it wrote (needs Chrome or Chromium): python docs/make_terminal_svg.py train.csv Survived docs/terminal.svg. The decision-tree images are drawn from docs/diagrams/*.mmd (mermaid): python docs/make_diagrams.py.

License

MIT