pytesprocess

pytesprocess is the detector-processing layer used with raw acquisitions written by pytesdaqx. It provides random-event selection, software triggering, feature extraction, salting, IV/dIdV processing, noise/filter generation, and persistence of processed event products as Vaex HDF5 dataframes.

The current code targets Python 3.11+ and the modern pytesdaqx acquisition model (acquisition + stream).

Installation

For a development checkout:

pip install -e .

Runtime dependencies and the installed CLI entry point are defined entirely in pyproject.toml.

Command-line interface

Installing the package provides:

pytesprocess --help

The normal event-processing workflow is selected directly from the top-level command:

pytesprocess ACQUISITION \
    --config processing.yaml \
    --steps randoms trigger feature

--steps is mandatory. It accepts either whitespace-separated or comma-separated values, including mixed input:

--steps randoms trigger feature
--steps randoms,trigger,feature
--steps randoms,trigger feature

All are equivalent. The execution order is normalized internally to:

randoms -> salting -> trigger -> feature

The separate detector-characterization/filter workflows remain explicit:

pytesprocess filter ACQUISITION --config processing.yaml
pytesprocess ivsweep ACQUISITION

Example:

pytesprocess acquisition_I2_D20260807_T170432.zarr \
    --config continuous_data_processing_v2.yaml \
    --steps randoms,trigger,feature \
    --nrandoms 300 \
    --nevents 300 \
    --ncores 4

See docs/user/cli.md for the current CLI behavior.

Configuration

New processing configuration uses config_version: 2 and separates workflow sections explicitly:

config_version: 2

resources:
  filter_file: /path/to/filterdata.hdf5

trigger:
  global: {}
  channels: {}

feature:
  global:
    trace_length_msec: 20
    pretrigger_length_msec: 10
  presets: {}
  channels: {}

salting:
  global: {}
  channels: {}

filter: {}

Feature trace lengths are specified in milliseconds in v2 configuration. The resolver converts them to samples using the selected acquisition sample rate. Channel selectors include all, shell-style globs such as Z1P*, comma groups for applying one block to several independent channels, and explicit multi-channel expressions such as A|B.

See docs/user/configuration.md.

Processed dataframe identity

Each Vaex processing product has one dataframe-group identity:

dataframe_group_name
dataframe_group_id
dataframe_group_number
dataframe_file_index
processing_label

For example:

trigger_I2_D20260908_T123456/
  trigger_I2_D20260908_T123456_F0001.hdf5
  trigger_I2_D20260908_T123456_F0002.hdf5

F#### is a processing task/output shard index, not a second stream or series identifier. Raw provenance uses canonical stream fields such as stream_id, stream_number, and stream_trigger_index.

See docs/user/outputs.md.

Package structure

The current package layout intentionally separates reusable analysis objects from acquisition-processing workflows:

pytesprocess/
  core/       reusable analysis/data objects and algorithms
  process/    raw/dataframe processing executors
  config/     YAML loading, selection, resolution, validation
  salting/    salt metadata generation and waveform injection
  workflows/  multi-step orchestration used by the CLI
  cli/        command-line parsing and dispatch
  utils/      shared utilities and HDF5/dataframe helpers

See docs/developer/architecture.md for the current developer-oriented architecture.

Notes on multiprocessing

The existing Vaex/PyArrow and numerical-library thread limits are intentional. They were introduced to avoid thread oversubscription and unstable/slower multicore processing. They should not be removed or relocated without dedicated multicore benchmarking.