cosmic-metrics

Per-observation quality metrics for NenuFAR Cosmic Dawn data. Read QUALITY statistics, calibration solutions, power spectra, horizon-filtered variance and optional GNSS ionosphere maps; inspect the resulting table, reject poor blocks and compute pspipe weights.

Documentation: Read the Docs

How the objects work together explains the diagram.

Install

Requires Python 3.10+ and the NenuFAR processing stack (aoquality, ps_eor, pspipe), installed with it from PyPI:

python -m pip install cosmic-metrics
python -m pip install 'cosmic-metrics[report]'  # Optional interactive HTML report
python -m pip install 'cosmic-metrics[iono]'  # Optional spinifex map reader

From a checkout, use python -m pip install -e . with the same extras.

Python workflow

A Dataset contains observation blocks, named source readers and a managed raw cache. Metric objects define calculations. compute_metrics() returns an independent MetricTable containing scalars and observation metadata.

import cosmic_metrics as cm
from cosmic_metrics import metrics as m

dataset = cm.Dataset.from_toml('metrics.toml')
noise = m.RegionPower('dt_V', **m.REG_B)
phase = m.PhaseChange(120, source='solutions_l2a')
table = dataset.compute_metrics([noise, phase], n_workers=1)
print(table.df)
print(table.missing_reasons)

# Example threshold only; choose science cuts for your dataset.
selection = table.select_blocks([
    cm.Rule(phase.name, '>', 3, scale='mad'),
], missing='reject')
weights = table.compute_weights(noise.name, selection=selection)
table.save('metrics.csv')
selection.save('selection.json', weights=weights)

print(dataset.cache.inspect())
preview = dataset.cache.clear(stale_only=True, dry_run=True)

Adapt examples/nt04_l3.toml to your input paths and save it as metrics.toml. QUALITY and calibration templates are resolved per block; source names choose which reader a metric uses. All block times are UTC MJD seconds. Cache validity uses exact time bounds and dependency signatures. For scripts with n_workers > 1, use a main guard.

Prepare the expensive raw arrays using only the dataset TOML, without defining metrics:

cosmic-metrics cache populate metrics.toml --dry-run
cosmic-metrics cache populate metrics.toml
cosmic-metrics cache status metrics.toml

The example uses Slurm/libpipe; set [worker].nodes="localhost" for local population, or use SSH host names/ranges. The CLI accepts the same host and worker overrides as nenucal, for example --hosts localhost --max-concurrent 4. It runs in the foreground; use a persistent terminal session for unattended work. Use --slurm-nodes 4 to override the Slurm allocation size for one run while retaining the account, partition and other Slurm options from the TOML. Valid cache entries are skipped; use --force to rebuild them explicitly. [cache].require=true makes later notebook computations raise on missing/stale PS or variance entries. QUALITY and calibration remain uncached. Run population before the Python example when using that configuration. See cache preparation for worker settings, progress logs and resuming interrupted jobs.

Command-line workflow

The dataset TOML can also define the metrics ([[metrics]]), selection stages ([[selection]]) and an output directory, so that no Python is needed:

cosmic-metrics compute metrics.toml --dry-run   # list blocks and metric columns
cosmic-metrics compute metrics.toml             # metrics.csv
cosmic-metrics select metrics.toml              # selection.json
cosmic-metrics report metrics.toml              # interactive report.html

The report shows every metric against LST or date, with linked selection across metric panels, the profile of a selected block in all metrics and a night × LST grid. See the command-line guide.

For RFI checks in a specific frequency band, configure a variance source and compute m.BandVariance((67, 68)) or m.BandVarianceRatio((67, 68)). These reduce the horizon-filtered channel variance around 67–68 MHz; the ratio uses dt_V as its noise reference. See the band-variance guide for units, filtering and interpretation.

Documentation

Start with the complete synthetic example, which needs no input data. The guide and generated API reference cover the current implementation:

• Dataset configuration, blocks and paths
• Metric definitions, array shapes and physical units
• Selection, persistence and pspipe weights
• Cache inspection, invalidation and cleanup
• Cache preparation locally and on Slurm
• Command-line analysis and report
• Custom readers and metrics
• Python API reference
• Development and documentation builds

The earlier metrics prototype is retained on the eric-prototype branch.

Validate and build docs

python -m pip install -e '.[test,docs,report]'
python -m pytest
python -m ruff check src tests docs/conf.py
python -m sphinx -W --keep-going -b html docs docs/_build/html