nenuprepipe 0.2.0

Pre-processing pipeline runner for NenuFAR Cosmic Dawn data

nenuprepipe — NenuFAR Cosmic Dawn pre-processing pipeline

prepipe converts raw NenuFAR L1 data to pre-calibrated L2 data ready for the nenuflow calibration pipeline.

It wraps nenudata, calpipe, and aostats from the nenucal-cd package and runs one spectral window at a time.

Prerequisites

• nenucal-cd >= 0.15
• aoquality >= 0.3
• DP3
• AOFlagger

Note: losoto installs the legacy progressbar package while nenucal requires progressbar2. They share the same import name, so depending on the installation order you may need to force progressbar2 back:

pip uninstall progressbar && pip install --force-reinstall progressbar2

Setup

pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe.git

mkdir -p ~/nenufar-ops/my-field && cd ~/nenufar-ops/my-field
prepipe init --project-code RP3A

See prepipe init below for what --project-code fills in. Edit project.toml/data_handler.toml/cal/* to adapt paths, node names, and obs_id patterns to your site — every field ships with an inline comment.

Then verify with a dry run before actually running anything (drop --dry-run once the printed plan looks right):

prepipe update project.toml --dry-run
prepipe run project.toml "202*EOR*" SW03 --dry-run

Logs are written to logs/prepipe_<command>_<SW>_<timestamp>.log and printed to the terminal.

Commands

prepipe init DIR

Like git init — sets up the current directory (create it and cd there first, it must be empty). Creates project.toml, data_handler.toml, and a cal/ directory with default parsets.

--project-code fills in first-guess L1/L2 data paths in data_handler.toml. L1 (raw archive) and L2 (your own processed output) are separate roots, since they're typically on different filesystems:

• L1 defaults to /databf/nenufar-nri/{code} (override with --l1-root-template)
• L2 defaults to /data/{user}/{code} (override with --l2-root-template)

bp_cal_path (bandpass solutions) is a plain relative path, not under either root -- like dppp_config in [l1_to_l2_config.*], it's resolved against data_handler.toml's own directory, not wherever you happen to run prepipe from.

prepipe update PROJECT.toml

Run this periodically (not tied to a particular SW or target) to keep the data handler in sync with the archive:

• update_data_handler registers newly-recorded raw observations (N1 obs_ids) between [update_data_handler].start/end (YYYYMMDD) in project.toml.
• update_n2_obs_ids groups N1 obs_ids -- named YYYYMMDD_HHMMSS_YYYYMMDD_HHMMSS_FIELD -- by UT12-to-UT12 night and FIELD into one N2 obs_id per group, named YYYYMMDD_FIELD (YYYYMMDD-2_FIELD, -3, ... if more than one N1 obs_id falls on the same night for that field). l1_to_l2 later splits each N2 obs_id into fixed-length time chunks, round-robin distributed across the node pool assigned here. prepipe run and most nenudata commands process N2 obs_ids, not N1.

prepipe run PROJECT.toml TARGET_OBS_IDS SW

Resolves each obs_id matching TARGET_OBS_IDS (for SW) to its calibrator via data_handler.toml's calibration map, calibrates it, then processes the target.

l1_to_l2, calpipe, quality_collect, and flag_bad_stations skip individual MSs/tasks already marked done by a previous run -- this is tracked by nenudata/calpipe themselves (crash-safe, per-MS), not by prepipe. Use --force-calibrator, --force-target, or --force-all on prepipe run to reprocess them anyway.

Calibrator steps (run once per calibrator, source auto-detected from the obs_id, e.g. ..._CASA_... → cal/casa_cal_bp.toml):

Step	Command	Description
l1_to_l2	nenudata l1_to_l2	Convert L1 → L2 via DP3 (flag + average)
calpipe	calpipe cal/{source}_cal_bp.toml	Bandpass calibration with DDECal
make_bp_sol	nenudata make_bp_sol	Average h5parms across SWs → combined bandpass solution
plot_bp_sol	soltool plot-spectra / plot-delay	Spectra + delay PDFs of the bandpass solution per obs → bp_sol_plots/
quality_collect	nenudata quality-collect	Run aoquality collect on every MS in parallel
quality_combine	nenudata quality-combine	Merge per-MS stats → one quality_l2_cal/<SW>/<obs_id>.qs per obs
plot_quality	aostats plot	Quality-stat PDF per obs from each combined .qs → quality_l2_cal/<SW>/plots/
find_bad_stations	aostats find-bad-stations	Flag outlier stations → bad_stations_casa.json
import_bad_stations	nenudata bad-stations import	Merge bad_stations_casa.json into the data handler registry

Target steps (always processed for TARGET_OBS_IDS/SW; only_n2 in [target] restricts resolution to the N2 obs_ids created by prepipe update):

Step	Command	Description
l1_to_l2	nenudata l1_to_l2	Convert L1 → L2_BP (apply bandpass, flag, average)
quality_collect	nenudata quality-collect	Run aoquality collect on every MS in parallel
quality_combine	nenudata quality-combine	Merge per-MS stats → one quality_l2_bp/<SW>/<obs_id>.qs per obs
plot_quality	aostats plot	Quality-stat PDF per obs from each combined .qs → quality_l2_bp/<SW>/plots/
plot_grid	aostats plot-grid	Diagnostic heatmaps (freq × obs, antenna × obs, LST × obs)
flag_bad_stations	nenudata bad-stations flag	Apply the data handler's bad-stations registry to the target MSs via DP3
push_l2	nenudata push_l2	Rsync L2 data to the nenuflow server

After push_l2, run import-n2 on the nenuflow server (the exact command is printed by push_l2):

ssh dawn
nenudata import-n2 /path/to/n2_obs_ids_push.json -c data_handler.toml

Skipping steps

Set any step to false in the relevant [calibrator.steps]/[target.steps] table of project.toml:

[target.steps]
quality_collect = false
quality_combine = false

Troubleshooting

On nancep (or similar shared machines)

Load DP3 and AOFlagger with use DP3 and use AOFlagger.

They can hit default per-process limits when processing many MSs concurrently. Raise these in your .bashrc:

ulimit -s 40960   # stack size
ulimit -u 40960   # max user processes
ulimit -n 40960   # max open files

Directory layout

nenuprepipe/                          # this repo -- the tool, pip-installed
├── pyproject.toml
└── nenuprepipe/
    ├── cli.py                        # init / update / run entry point
    ├── steps.py                      # pipeline step implementations
    ├── tests/                        # pytest suite for steps.py
    └── templates/                    # copied by `prepipe init` into a fresh deployment
        ├── project.toml
        ├── data_handler.toml
        └── cal/
            ├── casa_cal_bp.toml      # calpipe config — CASA / CasA bandpass
            ├── cyga_cal_bp.toml      # calpipe config — CygA bandpass
            ├── casa_cal_rll.toml     # calpipe config — CasA with smoothed solutions
            ├── Ateam_LBA.skymodel    # A-team intrinsic sky model
            ├── dppp_l1_to_l2_bp.parset      # DP3 parset — L1 → L2_BP (apply + flag + average)
            ├── dppp_l1_to_l2_bp_interpolate.parset  # same with frequency interpolation
            ├── dppp_l1_to_l2_cyga.parset    # DP3 parset — L1 → L2_12C40S (calibrator, 40s)
            ├── dppp_l1_to_l2_12c20s.parset  # DP3 parset — L1 → L2_12C20S (alternative averaging, 20s)
            ├── nenufar_largebw_sens1.5_I.lua  # AOFlagger strategy — target data
            └── nenufar_largebw_sens2.5.lua    # AOFlagger strategy — calibrator data

~/nenufar-ops/my-field/                # a deployment, created by `prepipe init` -- separate
├── project.toml                       # your live config (gitignored if you version this dir)
├── data_handler.toml                  # your live data handler config (gitignored)
└── cal/                               # your own copy of the default parsets, freely editable