NenuFlow

NenuFlow calibrates NenuFAR Cosmic Dawn L2 MeasurementSets. It takes the MSs of a data level from the NenuCal data handler (or MS lists), performs DI and optional DD calibration and subtraction, and writes calibrated data, solutions, sky models, and quality-control products.

For raw L1 observations, begin with NenuPrePipe. It creates the bandpass-corrected, quality-checked, station-flagged L2 data expected here.

NenuPrePipe L2 data
    → choose fast-track or self-calibration
    → nenuflow run L2_BP "<nights>:<SW>"   (checks the MSs and their station flags first)
    → inspect calibration and AOQuality products, and the summary of each run
    → import any newly detected bad stations, flag, calibrate again
    → nenudata make L3

Requirements

• Nextflow 23 or newer and Java 11 or newer (Java 17+ recommended)
• Singularity and a dawntools.sif image
• nenucal >= 0.30 on the host (installed with nenuflow), providing nenudata, modeltool, soltool and aostats

DP3, WSClean, AOFlagger and the AOQuality binary run inside the container. The NenuCal commands run on the host.

Check the host tools before setup:

java -version
nextflow -version
nenudata --help

If Nextflow is not installed, follow the official installation guide.

Quick start

1. Install and initialize

pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuflow.git

cd ~/nenufar-ops/my-field
nenuflow init

nenuflow init writes the nextflow.config of the site in the current directory, next to the data_handler.toml of the field (e.g. the directory created by prepipe init). It holds only what the site sets: the container image (CHANGEME), the bindings of the container, and the REVIEW settings (mode, resources, sky models, Slurm options). The defaults of every parameter, the resources of the processes and the profiles are in the nextflow.config of the package, which Nextflow reads first:

nenuflow config diff               # what ./nextflow.config changes, and what can be removed
nenuflow config diff --profile hq  # with the settings of a profile

2. Choose a calibration mode

mode has no default: set params.mode in nextflow.config, or give --mode to every run.

fast_track

Uses a catalog-based intrinsic sky model and does not run WSClean imaging:

build channels → DI calibration and A-team subtraction
               → DD calibration and subtraction (optional)
               → post-processing and diagnostics

Use this for the standard fast calibration path. NenuFlow builds the catalog and clustered models automatically unless intrinsic_catalog_model or intrinsic_catalog_model_cluster points to a fixed model.

selfcal

Builds the sky model iteratively with WSClean:

build channels → initial DI calibration
               → WSClean/self-calibration cycles
               → DD calibration and subtraction (optional)
               → post-processing and diagnostics

The final WSClean model is clustered for DD calibration.

In both modes, the subtractions and the DI apply interpolate the solutions linearly in time and frequency (soltool unwrap first unwraps their phases, as DP3 interpolates the stored values): nearest interpolation would leave steps between frequency solution intervals in the data, and power at the matching k_par in the power spectrum. The published solutions are unchanged.

--do_subtract false runs the DI calibration and A-team subtraction only (CORRECTED_DATA_L2_A, the L2a), without the DD round.

3. Run the pipeline

Select the MSs in the data handler, as nenudata does: one run per obs_id and spectral window, named <obs_id>_<level>_<sw>:

nenuflow run L2_BP "20231208_NT04:SW03" -profile local --mode fast_track
nenuflow run L2_BP "2025011*_NT04:SW34" -profile hq --mode fast_track

The MSs below the min_alt of the level (mean altitude, in degrees) are left out, e.g. the first and last time chunks of a night, as by nenudata make and make_ms_list:

[data_levels.L2_BP]
path = "/travail/fmertens/data/%YEAR%/%MONTH%/%OBS_ID%/L2_BP"
min_alt = 50

The altitude is read once from each MS, and kept in <MS>/ms_info.json.

An MS list (MS paths separated by blanks or new lines, as written by nenudata make_ms_list) gives one run, named after the file, e.g. ms_lists/20231208_NT04_L2_BP_SW03. The name gives the obs_id, level and spectral window: {obs_id}_{level}_{sw}.

The arguments from the first one starting with - go to nextflow run: the profile and the parameters, e.g. --mode fast_track --do_subtract false. Run nenuflow run from the directory of nextflow.config and data_handler.toml (--data-handler for another one).

Before anything starts, every run is checked, and any problem stops them all, listed with what to do:

• the MSs exist: one removed by nenudata remove is reported with the commands bringing it back (--allow-missing-ms runs without the missing ones); a run without MS never starts;
• they are flagged with the current bad stations of the registry (nenudata bad-stations flag; --allow-outdated-flags to run anyway);
• an MS list holds the MSs of the data handler, above the min_alt of the level (a warning otherwise);
• the parameters are nenuflow parameters (a misspelt one, on the command line or in nextflow.config, is an error), and the container image exists;
• with --from, the products of the stages before it (see Restarting from a stage).

The MSs are then locked for the time of the runs: nenudata make, bad-stations flag, quality-collect, remove, or another nenuflow run stop rather than use them.

Several runs, e.g. one per night, run four at a time (--jobs), or one after another with -profile local, as the runs would share the machine without knowing it. The output of each run goes to work/<name>/console.log and work/<name>/nextflow.log; a failed run does not stop the others, and the command lists them at the end. nenuflow run does not add -resume (see Restarting from a stage).

Each run ends with a summary, also in results/<name>/runs/<run_id>.log: the tasks failed, retried or ignored, with where they ran, their exit status and the end of their log; the columns written; the bad stations found and not in the registry; and the commands to run next:

=== 20250222_NT04_L2_BP_SW34: done in 3h49m (run 20261030-095915-a1b2) ===
  Tasks: 490 done, 2 retried, 1 ignored
  Retried: CalibrateDIcat:GetSolsPerDir on node0188, exit - (no exit status: killed or lost)
  ...
  Columns written: CORRECTED_DATA_L2_A (34 MS(s)), SUBTRACTED_DATA_L3 (34 MS(s))
  Next:
    nenudata make L3 "20250222_NT04:SW34"
    nenudata make L3_LST "20250222_NT04:SW34"
  Results: results/20250222_NT04_L2_BP_SW34, settings of the run: runs/20261030-095915-a1b2.json

nenuflow status shows the progress of the runs of the current directory: for each process, the tasks done, retried (failed attempts resubmitted by Nextflow), failed (not retried: they stop the run), running and queued (pending in Slurm), with the average and maximum run time of the tasks done:

nenuflow status                 # one table per running run
nenuflow status --combined      # all the runs together, per process
nenuflow status -a '2025*'      # also the runs that have ended, by name

nenudata status shows the state of the data of each night: MSs of each level, station flags, columns written by nenuflow, bad stations found and not imported, levels to remake, each problem with its command (see the NenuCal documentation).

nenuflow run also prints a progress report every --report-interval minutes (10 by default, 0 for none): the tasks of each run, and with -profile hq, the worker nodes with the tasks running on each and the time left in their allocation, the Slurm allocations waiting in the queue with the reason Slurm gives, and the tasks waiting for a free worker:

=== Progress at 10:30 (1h12m since the start) ===
Runs: 1 running, 1 waiting to start
  Run                       MSs  State    Done  Running  Waiting  Failed  Now working on
  20250112_NT04_L2_BP_SW34   34  running    51       10       16       0  DP3Calibrate (8 running, 16 waiting), SubtractSources (2 running)
  20250115_NT04_L2_BP_SW34    -  waiting     -        -        -       -

HyperQueue
  Worker nodes connected: 2
  Slurm allocations: 2 running, 2 waiting in the Slurm queue (reason: Priority)
  Tasks waiting for a free worker: 16 (16 DP3Calibrate)

  Node      Slurm job  Time left  Tasks running, with their runs
  node0156  310242     5h00m      2 DP3Calibrate (20250112, 20250115), 1 SubtractSources (20250112)
  node0160  310245     5h00m      idle

Compute profiles

Every process declares its CPUs and memory in the nextflow.config of the package, sized on a 33-MS NT04 run. DP3 calibration and WSClean imaging request cpus and mem_per_job (WSClean -j and DP3 numthreads follow); the other processes request 1 to 8 CPUs and 2 to 16 GB, capped at cpus and mem_per_job, so that they fit on a node next to the heavy ones. Choose cpus and mem_per_job so that a whole number of heavy jobs fills a node, e.g. cpus = 12 and mem_per_job = 43.GB for two jobs on a 24-core, 90 GB node. A DP3 calibration peaks at about 23 GB on 33 NT04 MSs.

Local

Use -profile local. Nextflow limits concurrency according to the CPUs and memory available to the local executor.

HyperQueue

With -profile hq and no HQ server running, nenuflow run starts one for the runs, with Slurm allocations of one worker node each, up to --hq-nodes: HQ submits them while tasks wait, packs the tasks on the workers from their CPU and memory requests, and releases a node after 5 min idle. The server and its workers are stopped at the end of the runs, and on Ctrl-C, kill or a closed terminal (SIGHUP is left ignored under nohup). A server left running by a run that could not stop it (kill -9) is stopped by the next nenuflow run of the directory, and its allocations still in the Slurm queue are cancelled.

export NENUFLOW_HQ_SLURM_OPTIONS="--clusters=astro_thin -p def --account=me --qos=myqos"
export NENUFLOW_HQ_WORKER_MEM=85GB
nenuflow run ms_lists/20250109_NT04_L2_BP_SW34 -profile hq --hq-nodes 5 --cpus 12 --mem_per_job 32.GB

NENUFLOW_HQ_WORKER_MEM (--hq-worker-mem) is the memory a worker offers to the tasks: keep it below the memory of a node. HQ does not limit the memory of a task, so the requests in nextflow.config must stay above the peak memory of the processes (peak_rss in the trace), and mem_per_job above that of DP3 calibration. The server log is in work/hq_<pid>/. The allocations are submitted with --no-requeue: an allocation whose launch fails ends, and HQ submits another, where Slurm would hold it in the queue (--requeue in --hq-slurm-options keeps Slurm's behaviour). The server checks its allocations every 2 min (HQ_AUTOALLOC_REFRESH_INTERVAL_MS), within the 5 min Slurm keeps an ended job: checking less often, it would count a cancelled or failed allocation as queued for hours, and submit no other.

The subtractions (SubtractSources) and AOqualityCollect mostly read and write the MSs: many at once saturate the shared file system, and behind the queued calibrations they start only once all of them have. With --hq-io-nodes M, they run on nodes of their own, up to M at once with --hq-io-tasks tasks each (12 by default: a DP3 subtraction uses about 1 core and 3.5 GB at most), which HQ starts as soon as they wait, next to the --hq-nodes nodes of the other tasks. M × --hq-io-tasks sets how many run at once: beyond what the file system serves, they only take longer. The small tasks run there too, and on the other nodes, wherever there is room. The hq profile of the package asks for the resource compute for the heavy tasks (DP3 calibration, WSClean), and for io for the I/O-bound ones, with their real use (1 CPU, 4 GB); ModelToolXmatch runs on the submission host, as it queries VizieR.

-profile hq needs hq 0.27 or newer: the scheduler of 0.26 can run for hours on the tasks of a few nights, and the server answers neither the clients nor the workers meanwhile. The release binaries need glibc 2.28; for an older system (glibc 2.17), build one in a manylinux2014 container, from the source of the release in src/:

docker run --rm -v $PWD/src:/src -w /src quay.io/pypa/manylinux2014_x86_64 bash -c '
  curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal && . ~/.cargo/env
  /opt/python/cp312-cp312/bin/pip install libclang
  export LIBCLANG_PATH=$(/opt/python/cp312-cp312/bin/python -c "import clang, os; print(os.path.dirname(clang.__file__) + \"/native\")")
  cargo build --release --locked -p hyperqueue --bin hq'   # binary in src/target/release/hq

With a server already running (hq server start and hq worker start --resource "mem=sum(<MiB>)" --resource "compute=sum(1000)" --resource "io=sum(6)" on each node), nenuflow run -profile hq uses it as is. The repository also contains scripts/hq_cluster.py and an example YAML file for starting and stopping a declared worker pool.

SLURM

Use -profile slurm and set the queue and cluster options of the site in nextflow.config (REVIEW in the file written by nenuflow init). A memory, cpus or time set for all processes in the profile is ignored, as the resources per process take precedence. Each process has a time limit of about five times its longest run, multiplied by the attempt number, so that a job stuck on a faulty node is killed and run elsewhere; jobs cancelled or lost by the scheduler are retried up to three times:

nenuflow run ms_lists/20231208_NT04_L2_BP_SW03 -profile slurm

Results and quality control

Each run writes to results/<name>/:

mslist.txt    the MSs of the run
runs/         per run: <run_id>.json, its settings (command, parameters, versions,
              sky models and parsets with their sha1, station flags), and
              <run_id>.log, its summary
models/       catalog, clustered, intrinsic, and apparent sky models; the
              clustered-model plots show the beam half-power and first-null
              contours at the band edges, at mid-observation; the spectra
              (*_spectra.png) and VLSSr cross-match (*_xmatch.png, .ecsv, .log)
              of the intrinsic catalogue and self-calibration models
solutions/    combined calibration solutions and plots
aoquality/    combined statistics, plots, and bad-station JSON files
wsclean/      self-calibration images and logs
report.html   Nextflow execution report
versions.txt  versions of nenuflow, Nextflow, nenucal, DP3, WSClean... and the
              container image
timeline.html execution timeline
trace.txt     process trace

Next to the MSs, each MS keeps its solutions (l2a_solutions.h5, l3_solutions.h5...) and apparent sky models, and the provenance record of each column written (provenance_CORRECTED_DATA_L2_A.json...: the run, the column it was made from, the station flags of the MS), read by nenudata make and nenudata status. The logs of the tasks of each MS go to the logs/ directory of the level, named after the run (SW34_T000_l2a_dp3_cal_<run_id>.log).

The VLSSr cross-match queries VizieR once per field: modeltool xmatch caches the query. On a machine without access to VizieR, the task fails (the summary shows why) with the modeltool cache-survey command to run on a machine with access.

Review both the solution plots and the AOQuality plots before accepting the run. The AOQuality documentation explains the plotted statistics and bad-station detection.

The summary of the run lists the bad stations found and not in the registry, with the commands importing and flagging them:

nenudata bad-stations import \
    "results/*/aoquality/bad_stations_l2a.json" \
    --note "reviewed NenuFlow detections"
nenudata bad-stations flag L2_BP "20231208_NT04:SW03"
nenudata bad-stations show-history

The columns calibrated before the new flags no longer match the MSs: calibrate again (nenudata status lists them, and nenudata make refuses them). See NenuCal bad stations for reviewing, adding, removing, and propagating station decisions.

Sky model from several nights

nenuflow skymodel builds a sky model from one WSClean image of several nights, calibrated beforehand (CORRECTED_DATA_L2_A, e.g. after a selfcal run of each night; skymodel_data_column otherwise), each night an obs_id of the data handler, or an MS list. It runs the skymodel mode with the checks and profiles of nenuflow run, and locks the MSs for reading only:

nenuflow skymodel L2_BP "20250110_NT04,20250112_NT04,20250115_NT04:SW03" -profile slurm

As in the selfcal mode, it images the data with wsclean_selfcal_config, keeps the sources within selfcal_model_radius, deattenuates them with the beam averaged over the observed time of all the nights (7 times per night, deattenuate_n_times), and clusters the result. The merged MS list and the products go to results/<level>_<sw>_<n>nights/ (--name otherwise):

mslist.txt  the MSs of all the nights
wsclean/    images and WSClean log
models/     skymodel_intrinsic.skymodel and skymodel_clustered.skymodel, the
            plot of the clusters, the spectra and VLSSr cross-match of the
            intrinsic model

To calibrate every night with this model, set intrinsic_catalog_model and intrinsic_catalog_model_cluster to the two models and run the fast_track mode on the same nights:

nenuflow run L2_BP "2025011*_NT04:SW03" -profile slurm --mode fast_track \
    --intrinsic_catalog_model results/L2_BP_SW03_3nights/models/skymodel_intrinsic.skymodel \
    --intrinsic_catalog_model_cluster results/L2_BP_SW03_3nights/models/skymodel_clustered.skymodel

The skymodel mode only reads the MSs, so -resume can rerun the steps after a failure without redoing the image, as long as the MSs were not calibrated again in between.

Restarting from a stage

Some calibration steps modify MeasurementSets in place, so Nextflow -resume cannot reliably identify their state: a cached task could be reused after its MS has changed. nenuflow run therefore does not add it; --from STAGE starts a run after the stages already done, once it has checked that their products exist in every MS, and that their columns still match the MSs (their provenance records: written with the current station flags, and not made from a column written again since):

Mode	--from	Starts with	Needs
both	di (default)	the first DI calibration
both	di_sub	the first DI subtraction, with its solutions in each MS	l2a_solutions.h5 (selfcal_init_solutions.h5 in selfcal mode)
fast_track	dd	the DD calibration	CORRECTED_DATA_L2_A
selfcal	image	the self-calibration image	CORRECTED_DATA_L2_A
selfcal	di_final	the second DI calibration, with the self-calibration model	CORRECTED_DATA_L2_A, results/<name>/models/selfcal_intrinsic.skymodel
selfcal	dd	the DD calibration	CORRECTED_DATA_L2_A, selfcal_final_solutions.h5, the self-calibration model
both	dd_sub	the DD subtraction, with the DD solutions in each MS	those of dd, l3_solutions.h5
both	dd_post	the plots and statistics of the DD round	SUBTRACTED_DATA_L3, l3_solutions.h5 (and those of dd in selfcal mode)

nenuflow run L2_BP "20231208_NT04:SW03" --mode fast_track --from dd_sub -profile local

No MS is read for these checks: a column without provenance record (not written, or before nenuflow 0.9) is not checked. The skip_* parameters of the pipeline are set by --from: neither on the command line nor in nextflow.config.

Three WSClean check images are optional diagnostics, published to wsclean/. Each images the full MS collection, which takes as long as a WSClean run:

Flag	Image
wsclean_check_before_di	<stage>_check_before: wide-field dirty image of the uncalibrated data, before the first DI round
wsclean_check_after_di	<stage>_check_after: the same after the final DI round, A-team subtracted (l2a in fast-track mode, selfcal_final in self-calibration mode)
wsclean_check_after_dd	l3_check_deep: deep clean after DD subtraction

For throughput-focused runs, disable them with e.g. --wsclean_check_before_di false. These switches do not skip the WSClean image that builds the self-calibration sky model.

Main configuration parameters

The defaults are in the nextflow.config of the package (nenuflow config diff shows its path); nextflow.config of the site, then the command line, override them.

Parameter	Default	Purpose
mode	none	fast_track or selfcal (skymodel is set by nenuflow skymodel)
do_subtract	true	Run DD calibration and source subtraction
wsclean_check_before_di	true	Check image of the uncalibrated data
wsclean_check_after_di	true	Check image after the final DI round
wsclean_check_after_dd	true	Deep check image after DD subtraction
sky_model_diagnostics	true	Plot the spectra of the intrinsic sky models and cross-match them with VLSSr
xmatch_resolution	12	Resolution of the sky models in arcmin, for the VLSSr cross-match
wsclean_temp_dir	null	Directory of WSClean's temporary files, e.g. a node-local disk (bound in the container); null: the work directory
cpus	10	CPUs of a DP3 calibration or WSClean job, and cap of the other processes
mem_per_job	40.GB	Memory of a DP3 calibration or WSClean job, and cap of the other processes
number_of_clusters	10	Number of DD calibration directions
solint	30	DI solution interval in time steps
solint_target	30	DD solution interval in averaged time steps (6 min at 3 × 4 s)
ntimesteps_to_average	3	Time averaging before DD calibration, in the calibration job (not kept on disk)
sols_per_dir_min_flux, sols_per_dir_max_flux	30, 50	Flux range in Jy of the faintest direction, which sets the solutions per direction
sols_per_dir_freq_mhz	66.5	Frequency at which the direction fluxes are compared
di_smoothness_main, di_smoothness_other	4e6, 1e6	DI frequency smoothing kernel in Hz of Main and of the other directions (null or 0: the parset value for all)
dd_solutions_per_direction	false	Also give the DD directions several solutions per interval, from their flux
dd_min_beam_cut	0.1	Remove from the apparent DD model of each MS the components whose beam is below this value at any frequency (0: keep all)
bad_stations_stat	DStd	Statistic used to find bad stations
modeltool_build_catalog	lcs165	Catalog used for automatic sky-model construction
modeltool_build_radius	12	Catalog-model radius in degrees
selfcal_model_radius	12	Radius in degrees of the sources kept from the WSClean self-calibration model (selfcal mode)
selfcal_min_beam_cut	0.1	Remove from the self-calibration model the components whose beam is below this value at any frequency (0: keep all)

DP3 thread counts are also set in the bundled parsets and should remain consistent with the requested resources.

Testing pipeline changes

For development, lint the pipeline and run the stub suite. Both also run in CI.

nextflow lint follows the include statements from the entrypoint, so it covers every module. It needs Nextflow 25.02 or newer:

nextflow lint nenuflow/main.nf

Stubs test pipeline wiring without a container or real data:

cd test/stub
./setup.sh
./run.sh

The suite covers fast_track with and without subtraction and with the skip flags, selfcal with subtraction, with skip_di, and without the optional check images, skymodel on two nights whose MSs share their names, and the runs that must fail (no mode, a missing MS). Besides the exit status, it checks that the DD solutions, the check images, the sky model products and the logs next to the MSs are published, or not, as each scenario requests. When nenuflow is installed, it also runs scenarios through nenuflow run: its checks, --from, and the files and summary of a run.

Give every new parameter its default in nenuflow/nextflow.config, and read every boolean one through flag('<name>') (from nenuflow/functions.nf): on the command line, --<name> false gives the string 'false', which Groovy treats as true.

Every new process must provide a matching stub: block with the outputs declared by that process.

End-to-end: process one night of data

This example follows one target night, 20231208_NT04, and one spectral window, SW03, from the raw archive through NenuFlow. Replace the project code, observation ID, level, paths, and execution profile with the values for your deployment.

1. Create one shared deployment

Install both pipelines, create the field directory, and initialize the NenuPrePipe data handler:

pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe.git
pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuflow.git

mkdir -p ~/nenufar-ops/my-field
cd ~/nenufar-ops/my-field
prepipe init --project-code RP3A

Edit project.toml and data_handler.toml. At minimum, verify the L1 archive, L2 output paths, processing nodes, date range, target/calibrator patterns, bandpass-solution path, and DP3 parsets.

2. Populate the data handler

Preview and then register observations and N2 night identifiers:

prepipe update project.toml --dry-run
prepipe update project.toml

Confirm that the requested night and its calibrator resolve correctly:

prepipe run project.toml "20231208_NT04" SW03 --dry-run

3. Run NenuPrePipe calibration and quality collection

prepipe run project.toml "20231208_NT04" SW03

This prepares the calibrator, derives the bandpass, creates the target L2 MeasurementSets, collects AOQuality statistics, proposes bad stations, applies the registered station flags, and transfers the L2 data when push_l2 is enabled.

If processing moves to another server, run the nenudata import-n2 command printed by push_l2 there before continuing.

4. Check and finalize antenna flags

Inspect the plots under quality_l2_cal/ and quality_l2_bp/. Check the registered stations and its history:

nenudata bad-stations show "20231208_NT04" -c data_handler.toml
nenudata bad-stations show-history -c data_handler.toml

Record any reviewed corrections with nenudata bad-stations add, remove, or clear. Then apply the current registry to the prepared target data:

nenudata bad-stations flag L2_BP "20231208_NT04:SW03" \
    -c data_handler.toml --only-n2

The command restores the pre-station-flag state before applying a changed registry, so removed stations do not remain flagged from an earlier decision.

5. Set up NenuFlow

In the same directory on the NenuFlow host:

nenuflow init

Edit nextflow.config: the absolute container path, and the REVIEW settings (mode, resources, sky models, Slurm options). nenuflow config diff shows what it changes in the defaults.

6. Check the night

nenudata status "20231208_NT04:SW03"

The MSs of each level, whether they are flagged with the current bad stations, and what was already calibrated.

7. Run NenuFlow

nenuflow run L2_BP "20231208_NT04:SW03" -profile local --mode fast_track

For a cluster, replace local with the configured hq or slurm profile. nenuflow run checks the MSs and their station flags first, and ends with a summary of the run and the commands to run next. Use --from to start after the stages already done (see Restarting from a stage).

8. Validate and record the result

Review:

1. results/20231208_NT04_L2_BP_SW03/solutions/ for calibration behavior;
2. models/ and, for self-calibration, wsclean/ for sky-model quality. In *_xmatch.png, the model / VLSSr flux ratio should stay near 1 with the distance from the centre: a drop away from it points to the beam correction. In *_spectra.png, spectral indices far flatter than the usual -0.7 point to a spectral tilt of the calibrated data;
3. aoquality/ for time, frequency, baseline, and antenna diagnostics;
4. the summary of the run (runs/<run_id>.log), report.html and timeline.html for execution issues.

If the summary lists bad stations not in the registry, review them and import them into the data handler used by the run, which keeps the audit trail with the night's data, then flag them:

nenudata bad-stations import \
    "results/20231208_NT04_L2_BP_SW03/aoquality/bad_stations_l2a.json" \
    --note "reviewed after NenuFlow 20231208_NT04 SW03"
nenudata bad-stations flag L2_BP "20231208_NT04:SW03"

If NenuFlow runs on another server, also copy the reviewed JSON back and import it into the original NenuPrePipe data handler, so the two station registries stay consistent.

flag restores the flags saved before the first bad-station flagging, then flags the current list. Then run NenuFlow again from the first calibration stage, so the new flags are included throughout the result, and review the new products before accepting the night:

nenuflow run L2_BP "20231208_NT04:SW03" -profile local --mode fast_track

9. Make the L3

nenudata make L3 "20231208_NT04:SW03"

make refuses a SUBTRACTED_DATA_L3 written before the current station flags, and remakes an L3 whose input was calibrated again only with --force.