NenuPrePipe

NenuPrePipe prepares raw NenuFAR Cosmic Dawn observations for calibration with NenuFlow. Starting from L1 data, it registers observations, associates targets with calibrators, creates quality-checked L2 MeasurementSets, identifies bad stations, applies the resulting flags, and optionally transfers the data to the NenuFlow server.

The intended flow is:

raw L1 archive
    → register observations and nights
    → calibrate the bandpass calibrator and find its bad stations
    → create bandpass-corrected target L2 data
    → apply station flags
    → inspect quality statistics
    → transfer to the NenuFlow server

NenuPrePipe orchestrates tools supplied by NenuCal and AOQuality. Use their documentation when you need lower-level control over data registration, calibration, transfers, quality statistics, or manual antenna decisions.

Installation

NenuPrePipe requires Python 3, nenucal >= 0.29, aoquality >= 0.3, DP3, and AOFlagger.

pip install nenuprepipe
# or the development version
pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe.git

On shared machines, make sure DP3 and AOFlagger are available in the shell in which prepipe runs.

losoto installs the legacy progressbar package, while NenuCal requires progressbar2. If the two conflict, restore the latter with:

pip uninstall progressbar
pip install --force-reinstall progressbar2

Quick start

1. Create a project directory

Keep the pre-processing and NenuFlow configuration for a field together:

mkdir -p ~/nenufar-ops/my-field
cd ~/nenufar-ops/my-field
prepipe init --project-code RP3A

This creates:

project.toml       prepipe workflow and step selection
data_handler.toml  observation locations, levels, nodes, and calibration map
cal/               calibrator configuration files (calpipe) and sky model

--project-code fills in the project code in the L1 archive path (/databf/nenufar-nri/CODE/...) and an L2 root under /data/$USER/CODE. Then review the generated files:

data_handler.toml

• [data_levels]: the path of L1 must match the archive layout of your observations, those of L2_BP and L2_12C40S point to where you want your processed data. Their other keys say how they are made from L1: bandpass, flagging, averaging, and the time chunks of the N2 nights (split, 180 time slots). See the NenuCal [data_levels] reference.
• [spectral_windows]: subband ranges of your observation setup (default SW01–SW04, subbands 192–433).
• cal_obs_id_patterns: calibrator obs_id patterns, if not only CasA and CygA.
• beam_corr = true in [data_levels.L2_BP] applies the beam correction (see Target).
• [remote_hosts.dawn]: only needed with push_l2.

project.toml

• [update_data_handler]: node assigned to the newly registered observations (where their DP3 jobs run, and %NODE% in their paths), and the start/end dates (YYYYMMDD) of the observations to register.
• [update_n2_obs_ids]: node(s) storing the L2 data of the N2 nights.
• [target.steps]: make_beam_corr (off by default, together with beam_corr above) and push_l2 (off by default, see step 6).
• [target.make_beam_corr]: beamsquint_freq, the optFrq of your observation parsets.
• [target.push_l2]: remote, matching a [remote_hosts.*] entry.

Relative calibration paths are resolved from the directory containing data_handler.toml.

Projects created before nenuprepipe 0.4 keep working: their [data_level_path] and [l1_to_l2_config] are still read by NenuCal, and a n_slots_per_chunk in [target.l1_to_l2] is still used, with a warning to move it into split.

For a full description of the data-handler fields, see the NenuCal data-management documentation.

2. Register new observations

prepipe update project.toml --dry-run
prepipe update project.toml

update scans the configured L1 archive over the date range in project.toml, registers new raw observations, and groups them into N2 night identifiers such as 20231208_NT04. Most later commands operate on these N2 identifiers rather than the original short observations.

Run update whenever new data arrive or the configured date range changes.

3. Preview the run

prepipe run project.toml "20231208_NT04" SW03 --dry-run

Check the selected targets, calibrators, paths, and commands before starting the real run.

Several targets can be processed in one run, with a glob pattern or a comma-separated list. Each target brings its own calibrator, so the run then processes all their calibrators too: every calibrator step runs once over all of them (calpipe once per calibrator source), then the target steps run over all targets. SW is a single spectral window name, such as SW03; run each spectral window separately.

prepipe run project.toml "202312*_NT04" SW03 --dry-run
prepipe run project.toml "20231208_NT04,20231210_NT04" SW03 --dry-run

4. Run the pre-processing

prepipe run project.toml "20231208_NT04" SW03

The run first prepares the associated calibrators, derives their bandpass solutions, and identifies their bad stations. It then creates the target L2 data, applies those station flags, collects and combines quality statistics, produces diagnostic plots, and pushes the selected L2 data if transfer is enabled.

Completed per-MS tasks are recorded by NenuCal and skipped safely on a later run. Use --force-calibrator, --force-target, or --force-all only when the corresponding products genuinely need to be rebuilt.

Each run ends with a summary of every step (done, skipped, failed, or not run) and the status table of its targets and calibrators. If a task fails, for example a DP3 conversion killed for lack of memory, its step lists the failed tasks with their log files and the run stops there with a non-zero exit status. Once the cause is fixed, rerun the same command: only the failed and remaining work is done.

To see how far each observation has got, without running anything:

prepipe status project.toml "20231208_*"            # summed over SWs
prepipe status project.toml "20231208_*" --per-sw   # one row per SW

5. Review quality and antenna flags

Inspect the calibrator and target plots written below quality_l2_cal/ and quality_l2_bp/, including the multi-observation grids. Confirm that the automatically selected stations are plausible:

nenudata bad-stations show "20231208_NT04" -c data_handler.toml
nenudata bad-stations show-history -c data_handler.toml

Use nenudata bad-stations add, remove, or clear for reviewed manual changes. NenuCal keeps an audit history. If the registry changes, the next prepipe run reflags the affected targets, but keeps their quality statistics; rerun with --force-target to recompute them on the new flags.

For interpreting plots or running additional diagnostics, see the AOQuality guide. For registry behavior and propagation from calibrator to target, see NenuCal bad stations.

6. Continue with NenuFlow

NenuFlow can run on the same server and use the prepared L2 data directly. In that case, keep push_l2 disabled (the default), create the MS list from the current data_handler.toml, and start NenuFlow locally.

If NenuFlow runs on another server, enable push_l2 in project.toml and configure the matching [remote_hosts.*] entry in data_handler.toml. It transfers the data and prints the exact import command to run on the destination server:

nenudata import-n2 /path/to/n2_obs_ids_push.json -c data_handler.toml

On either host, the registered L2 observations can then be turned into an MS list and processed by NenuFlow. The NenuFlow README contains the complete one-night example for both arrangements.

Once a night is calibrated and checked, nenudata make L3 writes its residual into a compressed L3 level, and nenudata remove L2_BP frees the L2 while keeping the NenuFlow solutions and sky models next to it (see NenuCal pipeline).

What prepipe run does

The workflow is configured through the [calibrator.steps] and [target.steps] tables in project.toml.

Calibrator

1. Convert raw L1 data to an averaged calibrator L2 level (nenudata make, as defined by the level in data_handler.toml).
2. Run calpipe and combine the bandpass solutions.
3. Plot solution spectra and delays.
4. Collect and combine AOQuality statistics.
5. Plot those statistics and identify bad stations.
6. Import the reviewed station list into the data-handler registry.

Target

1. Convert L1 to bandpass-corrected L2 data (nenudata make, as defined by the level in data_handler.toml).
2. Apply the registered bad-station flags.
3. Collect and combine AOQuality statistics on the flagged data.
4. Produce per-observation and cross-observation diagnostic plots.
5. Transfer the prepared data to the NenuFlow server when requested.

The bandpass carries the mini-array gain towards the calibrator, so the target keeps its own gain relative to it, which varies over the night with elevation. To correct it during the L1 to L2 conversion, enable make_beam_corr in [target.steps] and set beam_corr = true in [data_levels.L2_BP] of data_handler.toml; prepipe run warns when only one of the two is set. The correction assumes a perfect analog pointing and leaves errors of a few percent for the target calibration to absorb. Each correction is plotted to beam_corr_plots/<obs_id>_<sw>_beam_corr.pdf.

Set an individual step to false when it is intentionally handled elsewhere:

[target.steps]
quality_collect = false
quality_combine = false

Command summary

prepipe init [--project-code CODE]         create a deployment in the current
                                           (empty) directory
prepipe update PROJECT.toml                register observations and N2 nights
prepipe run PROJECT.toml OBS_IDS SW        prepare calibrator and target data
prepipe status PROJECT.toml [OBS_IDS]      show per-obs_id processing progress

Use --dry-run with update and run to inspect the planned actions. Logs are written under logs/ and also printed to the terminal. Run prepipe COMMAND --help for all options.

Troubleshooting shared machines

On nancep or similar hosts, load DP3 and AOFlagger using the local environment commands (for example, use DP3 and use AOFlagger). Large parallel runs may also require higher per-process limits:

ulimit -s 40960
ulimit -u 40960
ulimit -n 40960