V2DL3 - VERITAS (VEGAS and Eventdisplay) to DL3 Converter

V2DL3 is a tool to convert VERITAS data products to DL3 FITS format, allowing to use e.g. the gammapy science tools for the high-level analysis.

DL3 files include event lists, instrument response functions (IRFs) and observation index tables. The V2DL3 converter can be used to convert point-like and full-enclosure IRFs. The FITS output follows the data formats for gamma-ray astronomy as defined in open gamma-astro-data-formats (GADF) repository.

The V2DL3 project tries to share as many tools as possible between VEGAS and Eventdisplay, especially those used for writing the FITS files.

Two steps are required to convert VERITAS data products to DL3 FITS format and use them with gammapy. Each of these steps are covered by one of the following tools:

• converter of event lists and instrument response functions to DL3 (v2dl3-vegas for VEGAS, v2dl3-eventdisplay for Eventdisplay)
• v2dl3-generate-index-file tool to generate observation index tables

User installation

Users of V2DL3 can install the package with the following steps:

python3 -m venv v2dl3
source v2dl3/bin/activate
pip install v2dl3

All scripts (v2dl3-vegas, v2dl3-eventdisplay, v2dl3-generate-index-file, v2dl3-eventdisplay-query-runparameters, v2dl3-compareFitsFiles) are now available as callable command line tools.

V2DL3 for VEGAS

• VEGAS version >= 2.5.7
• Requirements are listed in the environment-vegas.yml file.
• Alternatively, a script which builds a Docker image with the latest V2DL3 and the prerequisite software for v2dl3-vegas is available. See utils/v2dl3-vegas-docker/README.md

Developer Installation (VEGAS)

Use the conda package manager to install the dependencies:

conda env create -f environment-vegas.yml

The environment v2dl3-vegas will be created and can be activated with:

conda activate v2dl3-vegas

Install now pyV2DL3:

pip install .

Docker recipe

To use a Docker image with v2dl3-vegas pre-installed, see utils/v2dl3-vegas-docker/README.md

The command line tool v2dl3 with VEGAS

Run v2dl3-vegas --help to see all options.

See README_vegas.md for more information on newer v2dl3-vegas features such as ITM reconstruction, full-enclosure, and event classes.

Make sure you have ROOT with pyROOT enabled and VEGAS(>=v2.5.7) installed to proceed. Now, lets create the DL3 fits files from the stage 5 files in the ./VEGAS/ folder.

One file at a time

To convert a single stage 5 file to DL3 fits you need to provide the path to the stage 5 file as well as the corresponding effective area file using the flag -f. The last argument is the name of the output directory .Beware that the file names for the outputs are inferred from the root file name (xxx.root -> xxx.fits)

v2dl3-vegas -f ./VEGAS/54809.med.ED.050.St5_Stereo.root ./VEGAS/EA_na21stan_medPoint_050_ED_GRISU.root ./test

Generate from a VEGAS stage6 run list

You can also provide a stage6 run list to the command line tool. In this case the last argument is the folder where all the output DL3 files will be saved. Beware that the file names for the outputs are inferred from the root file name (xxx.root -> xxx.fits)

v2dl3-vegas -l ./runlist.txt  ./test

Run lists may be generated via a utility script.

python utils/vegas_runlister.py --help

V2DL3 for EventDisplay

The pip installation as discussed above is recommended for all users.

Developer Installation (Eventdisplay)

Install dependencies and activate the environment using the conda package manager:

conda env create -f environment-eventdisplay.yml
conda activate v2dl3Eventdisplay
pip install -e .

Converting Eventdisplay data products to DL3

Run v2dl3-eventdisplay --help to see all options.

Convert an anasum output file to DL3. The following input is required:

• anasum file for a given run
• effective area file for the corresponding cut applied during the preparation of the anasum file (DL3 version)

Example for point-like analysis:

v2dl3-eventdisplay \
     -f ./64080.anasum.root \
        ./effectiveArea.root \
        --logfile test-pointlike.fits.log \
        test-pointlike-CI.fits.gz

Example for full-enclosure analysis:

v2dl3-eventdisplay --full-enclosure \
     -f ./64080.anasum.root \
        ./effectiveArea.root \
        --logfile test-full-enclosure.fits.log \
        test-full-enclosure-CI.fits.gz

Runs with observational parameters (i.e., zenith, night sky background) outside but close to corresponding IRF axes range can be converted with the one of the following two command line parameters:

• --fuzzy_boundary axis tolerance: This option interpolates the IRF at the boundary value if the run parameter value is within the given tolerance. The tolerance is defined as the ratio of the absolute difference between the boundary and run parameter value to the boundary. Repeat the option for each axis that needs a tolerance. This option is preferable over --force_extrapolation.
• --force_extrapolation: This option linearly extrapolates the IRF at the run parameter value when used with --interpolator_name RegularGridInterpolator. It is rejected for the default KNeighborsRegressor, which does not provide linear extrapolation. Use this option with caution since extrapolation is applied even for run parameter values very far from the corresponding IRF axes range.

Recommended options are: --fuzzy_boundary zenith 0.05 --fuzzy_boundary pedvar 0.5. For zenith values below 30 degrees and below the lowest available value on the IRF zenith axis, the lowest IRF boundary is used. The fuzzy boundary tolerance remains enforced for large zenith angles, where shower properties change significantly with small changes in zenith angle.

Further options are:

• --interpolator_name [KNeighborsRegressor|RegularGridInterpolator]: select the interpolator used to interpolate the IRFs values. The default is KNeighborsRegressor, which allows to robustly interpolate from a irregular grid of IRFS.
• --save_multiplicity: write telescope multiplicity as the EVENT_TYPE column in the EVENTS table.
• --instrument_epoch [epoch]: write instrument epoch as INSTRUME keyword.
• --db_fits_file [db_fits_file]: copy run information (e.g., weather, L3 rates) from DB fits file into header information. Requires access to DB fits files.
• --evt_filter [filter_file]: apply an event filter defined in a yaml or json file (examples below).

Eventdisplay CI

Test files for the Eventdisplay CI can be generated using the manual workflow of the v2dl3-eventdisplay.yml GitHub action.

Event filter examples

Selects all events with IsGamma == 1 (relevant only for anasum files produced with the WRITE_ALL_EVENTS set to 1).

IsGamma: 1

Select all events with energies between 1 and 10 TeV:

Energy: [1, 10]

Data storage and generating index files

Generate observation index and HDU tables for DL3 data storage are required to use with gammapy in for reading and analysis of the generated DL3 data. This steps is independent of VEGAS or Eventdisplay. The two index files are generated with the tool v2dl3-generate-index-file (check all options with v2dl3-generate-index-file --help).

The tables are described on the GADF website:

• Observation index table
• HDU index table

Contributing to V2DL3 and Developing Code

Your contribution is welcome!

A few remarks when contributing code:

• goal is to keep as much common code for converting from VEGAS or Eventdisplay data products
• put package specific code into the pyV2DL3/vegas and pyV2DL3/eventdisplay directories. As different environments are used for both packages, do not put any imports to vegas/eventdisplay in modules in pyV2DL3

To ensure readability, we try follow the Python PEP8 style guide. Best to install the precommit tool to check the code style before committing.

Functions and classes should contain a docstring with a short description.

Unit tests are encouraged and are available for few cases at this point. Unit tests are in the tests directory and can be called using pytest.

Use the python logging system instead of the ‘print()’ function to output text. This allows to pipe all output into a log file and for different logging levels (INFO, DEBUG, …).