stats

Column statistics for numeric data, from the command line.

stats reads numbers from standard input or files and prints the statistics you name, one output row per input column. It is a Python/numpy port of an older C++ tool, designed to sit in Unix pipelines.

$ seq 1 100 | stats n mean stdev q10 q90
         n       mean      stdev        q10        q90
       100       50.5      29.01       10.9       90.1

$ paste <(seq 1 10) <(seq 10 10 100) | stats -nh mean median
       5.5        5.5
        55         55

$ cat people.csv
height,weight,age
170,65,30
182,80,
165,,41
176,71,52
$ stats --csv -f people.csv n mean stdev
column          n       mean      stdev
height          4      173.2      7.365
weight          3         72       7.55
age             3         41         11

$ stats --csv -f people.csv -c age -T n mean median
stat          age
n               3
mean           41
median         41

The package also installs findgen, a small index generator (findgen 5 prints 0 through 4).

Install

Requires Python 3.10+ and numpy. It's best installed as an isolated tool:

$ pipx install git+https://github.com/donalgrant/pstats.git
# or
$ uv tool install git+https://github.com/donalgrant/pstats.git

To install from a local checkout for development, use python -m venv .venv && .venv/bin/pip install -e '.[dev]'.

Usage

stats [options] [STAT ...]

Statistic names and options can be given in any order. If you name no statistics, the default set is n mean stdev min median max. --describe gives a broader summary: n nnan mean stdev stderr min plq median puq max skew kurt, followed by any statistics you name.

General options

Input options

Weights and relations between columns (see below)

Histograms and confidence intervals (see Histograms and Bootstrap)

-x, -w and -e columns are chosen the same way as with -c, and are never treated as data columns themselves.

Output options

Input.

• Blank lines and anything after # are ignored.
• A first line with no numeric fields is taken as the column names. Use --header or --no-header-row to override this.
• Tokens like nan or NA count as missing values. So do empty delimited fields and fields missing from short rows.
• Missing values are left out of every statistic, and n counts only the values present. Use --strict to make missing values an error instead.
• Any other non-numeric token is an error.

Output.

• A header line comes first, unless -nh is given. Then there is one line per input column.
• By default each value is printed with format %10.4g, and statistics appear in the order you requested them.
• Rows get a leading label (the column name or number) when the input had a header row or -c was used. Otherwise the output is unlabeled, as in the original tool.
• csv and tsv output uses full precision.
• json output is a list of objects such as {"column": "height", "mean": 173.25}. Undefined values become null.

Exit status.

• 0 on success.
• 1 for input data problems, such as an unreadable file or a non-numeric value.
• 2 for a usage error, such as an unknown statistic.

Errors and warnings go to stderr.

Statistics

For the parameterized statistics (qN, momN, devN, ndevN, absmN), replace N with a number. For example, q25 is the 25th percentile, q2.5 the 2.5th percentile, and mom3 the third raw moment. N may be fractional.

Counts and location

Spread and shape

Pictures

Percentiles interpolate linearly between order statistics, which is the numpy and pandas default. Statistics that are undefined for the data print nan, for example stdev of a single value or kurt with fewer than 4 values.

Relations between columns

Use -x COL to name a reference column. Each other column y is then compared with it: correlated, and fitted by least squares with a straight line y = a + b·x. The output keeps one row per column, and these statistics can be mixed with ordinary ones.

$ cat obs.txt
t flux err temp
1 10.2 0.5 20.1
2 12.1 0.4 20.9
3 13.8 0.6 22.2
4 16.3 0.5 22.8
5 17.9 0.7 24.1
$ stats -f obs.txt -x t -c flux,temp corr slope slope_err r2
column       corr      slope  slope_err         r2
flux        0.998       1.96    0.07211      0.996
temp       0.9946       0.99    0.05972     0.9892

$ stats -f obs.txt -x t -e err -c flux slope slope_err intercept rchi2fit
column      slope  slope_err  intercept   rchi2fit
flux        1.974     0.1756      8.166     0.1701

$ stats -f obs.txt --matrix corr -c flux,temp,err
column       flux       temp        err
flux            1     0.9869     0.6623
temp       0.9869          1     0.7607
err        0.6623     0.7607          1

Rows where either the column or the reference is missing are skipped for that pair only.

Weights

• -w COL (frequency weights). Every statistic is computed as if each row appeared w times. For integer weights the results equal those for the expanded data, and n is Σw. Rows with zero weight are ignored, and negative weights are an error. trimmeanN and rcorr don't accept weights.
• -e COL (measurement errors σ). Rows are weighted by 1/σ².
  • mean is the inverse-variance weighted mean and stderr is 1/√Σ(1/σ²).
  • chi2 and rchi2 measure scatter about that mean.
  • The fit statistics become proper weighted least squares.
  • Statistics with no standard inverse-variance definition, such as median and stdev, are refused rather than silently computed unweighted.
  • Weight-independent statistics (n, min, max, range and so on) still work.

stats --list-stats marks each statistic with the weight modes it accepts.

Grouping

-g COL splits the rows by the value in COL, then computes every statistic for each group and column. The key column can hold text, such as filter names or sample IDs. Groups are listed in sorted order, numerically when every key is a number. Rows with no key are skipped, with a warning.

$ cat bands.txt
band flux err
V 15.2 0.4
B 13.1 0.5
R 17.0 0.6
V 14.8 0.3
B 12.7 0.4
R 16.4 0.5
V 15.5 0.5
B 13.4 0.6
R 16.9 0.4
$ stats -f bands.txt -g band -c flux n mean stderr
band column          n       mean     stderr
B    flux            3      13.07     0.2028
R    flux            3      16.77     0.1856
V    flux            3      15.17     0.2028

$ stats -f bands.txt -g band -c flux -e err mean stderr rchi2
band column       mean     stderr      rchi2
B    flux        12.97     0.2771     0.5184
R    flux        16.77     0.2771     0.4001
V    flux        15.05     0.2164     0.8225

Grouping combines with -x, -w, -e, --hist, --ci and every output format. In csv and json output the group appears as its own field.

Histograms

--hist prints a histogram of each column. With -F csv, tsv or json you get the plain lo, hi, count rows, without bars. With -g, each column uses the same bin edges for every group, so the histograms can be compared. With -w, the counts are sums of weights.

$ seq 1 50 | awk '{print $1*$1 % 37}' | stats --hist --bins 6
        lo         hi      count
         0          6          9  ████████████████████████████████▊
         6         12         11  ████████████████████████████████████████
        12         18          6  █████████████████████▉
        18         24          3  ██████████▉
        24         30         11  ████████████████████████████████████████
        30         36         10  ████████████████████████████████████▍

$ seq 1 50 | awk '{print $1*$1 % 37}' | stats n mean spark
         n       mean      spark
        50      17.86 ▆▆▆▅▃▃▃█▂█

Bootstrap confidence intervals

--ci LEVEL adds a STAT_lo and STAT_hi column after each statistic. These give a percentile-bootstrap interval, computed as follows:

1. The rows that have a value are resampled with replacement --bootstrap times (default 1000). Each row's weight and reference value travel with it. Rows with missing values are left out first, so every resample is the same size as the data behind the estimate.
2. The statistic is recomputed on each resample.
3. The interval is the central LEVEL% of those values.

This works for any numeric statistic, including the fit statistics.

$ seq 1 50 | awk '{print $1*$1 % 37}' | stats --ci 95 --seed 1 mean median
      mean    mean_lo    mean_hi     median  median_lo  median_hi
     17.86      14.62      20.96         16         11       25.5

Use --seed to get the same intervals on every run. The cost grows with the number of rows times the number of resamples: about 1.6 s for 100k rows and 1000 resamples, and there's a warning when it's likely to be slow.

findgen

findgen N [-o OFFSET] [-s STEP] [-r]

Prints OFFSET, OFFSET+STEP, … (N values), one per line. -r reverses the order. The name comes from IDL's FINDGEN.

Development

$ .venv/bin/pytest          # tests, including golden-output checks against the original pstats
$ .venv/bin/ruff check . && .venv/bin/ruff format --check .

To add a statistic, register a Stat in src/stats_cli/stats.py. Plain stat names must not end in a digit. A parameterized family is registered by its base name (q) with param= set. Sample caches the mean, deviations and standard deviation, so they're computed once per column.

License

MIT