THILOS

THILOS is a reduction software for the Broad Band Imaging mode of HIPERCAM at GTC.

Developed by Fabricio M. Pérez-Toledo

General Description

The HIpercam Long-exposure Observations Solution (THILOS) is a software for the automatic reduction of deep-imaging observations with long exposure times simultaneously across all five Sloan broad-band filters. The software applies observation-specific reduction steps, ensuring optimized treatment for different data types. Developed with a focus on simplicity and efficiency, THILOS streamlines the reduction pipeline, enabling researchers to obtain calibrated data ready for photometric studies.

Key Reduction Steps:

1. Applying the BPMs.
2. Creation of the Master Bias.
3. Creation of the Master Flat.
4. Application of master calibration frames to both standard star and science frames.
5. Removal of cosmic rays.
6. Illumination correction.
7. Sky subtraction.
8. Alignment of science frames.

Input Requirements:

The software requires the following frames as input:

• Bias frames
• Sky flat frames
• Photometric standard star frames
• Science frames

Outputs

The generated results consist of one image per observed band. For each image, the following corrections and calibrations will have been applied:

• Bias subtraction
• Flat-field correction (including fringing correction for the Sloan z band, if applicable)
• Image alignment and stacking

To address cosmetic defects, the LACosmic algorithm is used to handle cosmic ray removal. The BPMs are applied to each CCD at the beginning of the reduction process.

Requirements

Operative System

• Any: The software is designed to run within a Conda environment, ensuring compatibility across platforms.

Dependencies

The following Python packages are required (minimum versions specified), however, they will be installed automatically together the :

astroalign>=2.6.2
astrometry_net_client>=0.6.0
astropy>=7.2.0
astroquery>=0.4.11
ccdproc>=2.5.1
lacosmic>=1.4.0
loguru>=0.7.3
matplotlib>=3.10.8
numpy>=2.4.4
opencv-python>=4.13.0.92
pandas>=3.0.2
PyYAML>=6.0.3
sep>=1.4.1

Hardware Requirements

• RAM: Minimum 4GB (higher is recommended for large datasets).

Installation

Installing THILOS is straightforward. Follow these steps:

1. Activate your Conda environment (or create a new one if needed (see below)):

   conda activate <your_env>
   
   

2. Install THILOS using pip:

   pip install thilos
   
   

That's it! THILOS is now almost ready to use ;)

Optional: Creating a New Conda Environment

If you don’t have an existing Conda environment, you can create one specifically for THILOS with the following commands:

conda create -n thilos_env python=3.11 -y
conda activate thilos_env
pip install thilos

BEFORE YOU USE THILOS (FOR LINUX USERS)

HIPERCAM frames are data cubes, which means that each frame contains full temporal series of images for each of the five Sloan bands. Therefore, before running THILOS, you must split the frames into individual images for each band. This can be done using the joinup script provided by the HIPERCAM software package.

You MUST install HIPERCAM's image for docker. You should follow the HIPERCAM website instructions. https://github.com/HiPERCAM/hipercam

$ docker build -t hipercam:latest -f Software/hipercam.dockerfile .

The correct way to avoid to log in with hiperuser by default generating a permission incompatibility is using the following command:

$ docker run -it --rm --user root -v <PathToYourObservationData>:/home/hiperuser/data hipercam:latest

HOW TO GET INDIVIDUAL FRAMES?

First, you need to extract the frames from compressed file that you have downloaded from the GTC archive, FTP, etc. Then, you place the frames in the original/ folder of your observation directory. After that, you can run the following command to split the frames into individual images for each band and full temporal series::

$ joinup data/original/<some.fits>

Now, you need to setup the script (it's IRAF-like command line interface) to split the frames correctly. The next example shows how to setup the script for a specific observation. You can adjust the parameters according to your needs.

$ first - first frame to process [1]: 
$ last - last frame to grab [0]: 
$ trim - do you want to trim edges of windows? [False]: 
$ ccd - CCD(s) to join up [0 for all] [0]: 
$ aper - aperture file to generate regions for ds9 ['none' to ignore] [none]: 
$ bias - bias frame ['none' to ignore] [none]: 
$ dark - dark frame ['none' to ignore] [none]: 
$ flat - flat frame ['none' is normal choice with no bias] [none]: 
$ fmap - fringe map ['none' to ignore] [<PathToFringeMap>/fmap.hcm]: (only if you want to use one) 
$ msub - subtract median from each window? [False]: 
$ ndigit - number of digits to use for frame numbers in output names [4]: (Customizable)
$ dtype - output data type [float32]: 
$ dmax - maximum allowable amount of data to write out [GB] [100.0]: (Customizable)
$ nmax - maximum allowable number of frames to write out [10000]: (Customizable)
$ compress - internal HDU compression to apply [none]: 
$ odir - directory for the output files [<OutputFolderPath>]: (Customizable)

Rename the joinup output file

THILOS requires a specific naming convention for the output files generated by the joinup script. To facilitate this, a script has been added to rename the joinup output files accordingly. This ensures that THILOS can read the files and continue with the reduction process seamlessly. The usage of the script is as follows:

$ cd path/to/frames
$ python -m thilos.script.rename_joinup_fits

Additional notes:

The BPMs cannot be distributed due to a decompression issue in PyPI. They must be downloaded from the GTC website.

First-Time Setup

Once Conda is set up, you should run THILOS for the first time to create the file configuration.json that has to be configured after.

$ thilos -c

or

$ thilos --create_config

You must edit the configuration file, which is located in your frame directory.

NOTE: You must download the BPM files from the GTC website and include the path to the BPMs in the configuration file.

You need to set the following parameters in the configuration file:

Optional Setup: You need to adjust the setup according to your observation. For example, if you are working with Sloan z and need to remove the fringing, you must enable the option in the Reduction section and set the save_fringing parameter to true. Regarding the alignment can be modified, but they generally work well as they are.

The directory structure must follow the format <Your_Program>_<Your_OB>/. Inside this directory, you should have a raw/ folder where the original frames are stored, and a reduced/ folder where the reduced frames will be saved.

Your_program/
    configuration.json
    original/
    raw/
    reduced/

Running THILOS

After saving and updating the configuration file, you can run the command using the following argument. The software will execute successfully.

$ thilos -e

or

$ thilos --execute

Outputs and Results

Once the process is complete, you will find a collection of reduced frames in the reduced/ folder inside your frame directory. The output includes:

A. Reduced science frames:

• One version with the sky included.
• One version with the sky subtracted.
• Applied BPM to each frame and filled the values using a directional polynomial inpaint algorithm.

B. Aligned frames:

• Both sky-included and/or sky-subtracted versions.

C. Final reduced science frames:

• Both sky-included and/or sky-subtracted versions.

Project Structure

thilos/
    script/
        directional_polynomial_inpaint.py -> This is a script to fill the empty values generated by the BPM. Each pixel is assigned a new value determined from adjacent pixels.
        fringe_mask.py -> Using imaging frames, this script generates a fringe map that is applied to each frame for Sloan z-band frame.
        rename_joinup_fits.py -> The joinup always saves the FITS files with a new structure name that is incompatible with THILOS. This script solves this by changing them to a new structure name that varies depending on the frame type.
    
    config/
        configuration.json   -> Configuration file.
    
    aligning_hcam.py   -> Aligns the science frames.
    check_files.py     -> It determines which steps can be performed by the pipeline based on the available FITS files.
    Color_Codes.py     -> Gives color to the comments
    DRP.py             -> Handles all the sofware and manages the frames.
    rebinning.py       -> The skyflats are always in binning 1x1. This script allows you to modify the binning.
    reduction_hcam.py  -> Carries out the clean process.

Note about the frames

The code is designed to work with HIPERCAM frames. They must be in FITS format.

LICENSE

This software is under GPL v3.0 license. More information is available in the repository.

CONTACT

• Email: fabricio.perez@gtc.iac.es

• Repository: https://github.com/Kennicutt/THILOS