Roman WFI Reference File Pipeline

Installation

It is recommended to begin with a clean environment, such as:

conda create -n wfirefpipe python=3.12

To install the package for general usage:

pip install .

OR To install the package for development and testing:

pip install -e .[docs,test]

OR To install the package utilizing any database functionality (needed for automation or QC checks) please follow the Additional instructions below

This will also install all of the dependencies.

NOTE: Installing this way uses the dependencies outlined in pyproject.toml. These dependencies mirror requirements.txt. If you would like to install with custom dependencies, just update them in requirements.txt and pip install -r requrements.txt

Users must create a config.yml living at: /wfi_reference_pipeline/src/wfi_reference_pipeline/config/config.yml Use example_config.yml in the same directory as a template.

Installation with RTB database access

RTB Database is utilized for automated processing of the reference file pipeline at STScI. If you are not setting up automated instances at STScI then you will not need access to the RTB Database and should skip this section.

To install with the RTB database on most platforms, the main additional step is to install the SQL driver packages with anaconda. This is the easiest way to ensure the correct SQL drivers are installed regardless of operating system. The one exception are Macs with M1 processors. For these installations, please see macOS M1 installation.

Notes: 1) You will need access to the Roman Grit group and might need to configure git on your machine to use a username and password that are registered with the Roman Grit group. 2) If you have 2FA enabled for your Grit account, you will need to use an access token to install rtbdb. Follow the instructions to create an access token. This access token can now be used as your user password when prompted during the install.

Using conda, please run the following in the specified order:

conda activate wfirefpipe
"pip install git+https://grit.stsci.edu/roman/rtb-database"
pip install .
conda install freetds

The last step is creating some SQL configuration files. You will need to set some environmental variables and create some driver configuration files. Please follow the instructions in the "Configuring SQL drivers" section of the README.md file from the RTB Database repository.

If experiencing any issues connecting to the server, please check out the rtbdb troubleshooting.

Contributing

To contribute to this project, please become familiar with our Contributing Guide

Documentation

To get the full documentation, install the package as described above, install sphinx, then go to the /src/docs/ directory and run:

make html

The documentation can then be found in doc/build/html/index.html.

Slack Integration

To use the Slack notifications, you must set up a Slack token. When you have a token, you can point to it with the environment variable WFI_SLACK_TOKEN.

Updating reference files on CRDS

Go to your .bash_profile and set environment variables to determine the crds state.

This is for the Roman test instance of CRDS.

export CRDS_SERVER_URL="https://roman-crds.stsci.edu"
export CRDS_PATH="/PATH/TO/ops_crds/"

Start with a clean environment where you will get the latest versions of romancal, roman attribute dictionary, and roman data models.

conda create -yn VMdevRFP_update_refs ipython
conda activate VMdevRFP_update_refs
pip install romancal crds

By doing this pip install you will get the latest released versions since this workflow is setup to replace old reference files with the most current.

Now sync crds to get all of the mappings updated.

crds sync --all

See the update_reference_files.py script in examples that was done for Build 17 in April 2025. Also see

navigate to central store's roman directory, then:
./RFP/DEV/py_scripts_notebooks/build_pyscripts/update_all_TVAC_CRDS_ref_files.py

Creating a New (SOC) Calibration Reference File Type in this Repository

These instructions are currently for adding new calibration reference file types are that Science Operations Center (SOC) will utilize in development with the Roman Calibration Pipeline (romancal). Instructions for external community derived products will be provided at a later date.

To create a new reference file type, please see New Reference Type Meta Data Workflow Example This allows a new effect or calibration reference file type to be defined within the structure of the repository.

For a detailed walkthrough on how to create a new reference type class, see the New Reference Type Class Example.

Once your module is implemented, you can follow the testing guidelines and examples here: Basic Testing for New Reference Types.

These resources provide step-by-step examples to help ensure both your implementation and test coverage follow the expected structure and conventions used throughout the pipeline.