romanimpreprocess 1.0.0

Dependencies

You will need to install:

• galsim: This is a general-purpose package used for image simulations (romanisim is built on top of it).

• romanisim: This is a Roman-specific image simulator. romanimpreprocess.from_sim uses the romanisim functions rather than calling the whole script, since we are generating ramps from a 2D image rather than a catalog.

• romancal: The Roman pipeline to go from Level 1 (uncalibrated, 3D) to Level 2 (calibrated, 2D) data. Note: this is not used for scripts in romanimpreprocess.from_sim. Note that the romancal installation also installs some other modules that romanimpreprocess calls directly (specifically: stcal and roman_datamodels). Testing is being done with the romancal main branch, which is still evolving; in the event of issues with this, you could try:

  pip install git+https://github.com/spacetelescope/romancal@main

  (Once Roman is in routine science operations, we will replace this with a version number requirement.)

The romanimpreprocess workflows do not explicitly call the Calibration Reference Data System (CRDS), although both romanisim and romancal have the ability to do so. Rather, we are specifying calibration reference files in the YAML configurations. (Direct calls from romanimpreprocess to CRDS will likely be added as an option in the future.)

If you want to generate your own calibration files from flats and darks (as opposed to using externally provided ones) then you will want to download solid-waffle. This is being done for the summer 2025 run so that we test out the workflow and have some distribution of hot/low-response/etc. pixels, but of course the files are not “complete” as they have dummies for some of the calibrations that can only be done in flight.

Conventions

In a “glue” script such as romanimpreprocess, the linking of the conventions used by different tools can be a bit overwhelming. The current summary of the conventions is here.

Converting an OpenUniverse simulation to L1 format

You can convert an OpenUniverse truth image file by running the sim_to_isim script:

python3 -m romanimpreprocess.from_sim.sim_to_isim config.yaml

The simplest configuration config.yaml that you can run is as follows:

Here:

• IN is the input file.

• OUT is the output file.

• READS is the Multi-Accum table (here the group averages are range(0,1), range(1,2), etc.)

• SEED is the random number generator seed (if this is missing then a default is used, but this is definitely not recommended!).

This will generate both the output (simulated L1) file, and the truth WCS file, in this case sim1_asdf_wcshead.txt (which exists in the simulation but wouldn’t be part of the real data).

It is also possible to include a dictionary of calibration reference files:

(Note that some of these have additional data: see the from_sim Readme for a detailed list of requirements.)

Converting an L1 to L2 image

You can (partially) convert an L1 image (unprocessed data) to an L2 image (2D with instrument artifacts cleaned or flagged) by running the gen_cal_image script. Some “big picture” known issues at this point are:

• Only the internal steps (propagating bad pixel flags, saturation and cosmic ray flagging, linearity/IPC, dark, flat, bias) work right now. The WCS has to be externally provided, and outputs, while flattened, are still in instrumental units (DN/s).

• Not all of the metadata and error arrays populate correctly in this version. We’re working on it!

The script can be run via:

python3 -m romanimpreprocess.L1_to_L2.gen_cal_image config_L1_to_L2.yaml

The simplest configuration config_L1_to_L2.yaml that you can run is as follows:

---
# Input file
IN: 'sim1.asdf'
OUT: 'sim2.asdf'
FITSWCS: 'sim1_asdf_wcshead.txt'
CALDIR:
  linearitylegendre: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_linearitylegendre_DUMMY20250521_SCA10.asdf'
  gain: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_gain_DUMMY20250521_SCA10.asdf'
  dark: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_dark_DUMMY20250521_SCA10.asdf'
  read: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_read_DUMMY20250521_SCA10.asdf'
  ipc4d: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_ipc4d_DUMMY20250521_SCA10.asdf'
  flat: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_pflat_DUMMY20250521_SCA10.asdf'
  biascorr: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_biascorr_DUMMY20250521_SCA10.asdf'
  mask: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_mask_DUMMY20250521_SCA10.asdf'
  saturation: '/fs/scratch/PCON0003/cond0007/cal/roman_wfi_saturation_DUMMY20250521_SCA10.asdf'
...

Here:

• IN is the input (L1) file.

• OUT is the output (L2) file.

• The WCS and format is externally provided by one of the *WCS keywords (in this case: FITSWCS).

• CALDIR is a directory of calibration files to use (romanimpreprocess uses this in place of the *.imap files used in the SOC tools, but it would be straightforward for the calling script to write the imap files into a configuration YAML).

This will generate the output (simulated L2) file, with the provided WCS (in this case sim1_asdf_wcshead.txt) included.

See the L1_to_L2 Readme for detailed instructions and all the options.

Important note: Sometimes the SOC L2 data model is updated with new schema entries that aren’t in the PIT L2 files. This means legacy pipelines that don’t use the new schema entries will raise an exception if a dependency is updated with a newer version of the schema. We have marked these with the ["roman"]["meta"]["dummyfields"] entry in the ASDF tree and inserted a dummy (generally array of zeros) so that the pipeline will run. Please check for that field if you run code that expects a SOC L2 file.

Utilities

The utils/ folder includes some utilities that are intended to be called by the pipelines, but also that users might find useful for postprocessing, visualization, or other applications. See the utilities page for more details.

A few useful test scripts are in validation_tests.

Information for specific runs

You can find information on specific runs we have done (or are doing) as follows. The code in these directories will be updated in the future when we build new calibration files; but it deals with specific input formats that may change (as opposed to the rest of the repository that is intended to be fully general):

• Summer 2025 run (with FPT tests + OpenUniverse): here.

• July 2026 run: here.