ariel-data-preprocessing 1.2a4

Signal correction module for Ariel Data Challenge 2025.

Ariel Data Preprocessing

This module contains the complete FGS1 and AIRS-CH0 signal data preprocessing pipeline for the Ariel Data Challenge.

Submodules

1. Signal correction - Complete 6-step calibration pipeline for raw telescope data
2. Signal extraction - Intelligent data reduction and spectral signal extraction

1. Signal correction

Implements the complete six-step signal correction pipeline outlined in the Calibrating and Binning Ariel Data notebook shared by the contest organizers. This module transforms raw Ariel telescope data into science-ready corrected signals.

Processing Pipeline:

1. ADC Conversion - Convert raw counts to physical units
2. Hot/Dead Pixel Masking - Remove problematic detector pixels
3. Linearity Correction - Account for non-linear detector response
4. Dark Current Subtraction - Remove thermal background noise
5. Correlated Double Sampling (CDS) - Reduce read noise via paired exposures
6. Flat Field Correction - Normalize pixel-to-pixel sensitivity variations

Key Features:

• Multiprocessing support for parallel planet processing
• Optional FGS1 downsampling to match AIRS-CH0 cadence (83% data reduction)
• Configurable processing steps (enable/disable individual corrections)
• HDF5 output for efficient large dataset storage

See the following notebooks for implementation details and performance analysis:

1. Signal correction
2. Signal correction optimization

Example use:

from ariel_data_preprocessing.signal_correction import SignalCorrection

signal_correction = SignalCorrection(
    input_data_path='data/raw',
    output_data_path='data/corrected',
    n_cpus=4,
    downsample_fgs=True,
    n_planets=100
)

signal_correction.run()

The signal correction pipeline will write the corrected frames and hot/dead pixel masks as an HDF5 archive called train.h5 by default with the following structure:

    train.h5:
    │
    ├── planet_id_1/
    │   ├── AIRS-CH0_signal   # Corrected spectrometer data
    │   ├── AIRS-CH0_mask    # Mask for spectrometer data
    │   ├── FGS1_signal      # Corrected guidance camera data
    │   └── FGS1_mask        # Mask for guidance camera data
    |
    ├── planet_id_2/
    │   ├── AIRS-CH0_signal  # Corrected spectrometer data
    │   ├── AIRS-CH0_mask    # Mask for spectrometer data
    │   ├── FGS1_signal      # Corrected guidance camera data
    │   └── FGS1_mask        # Mask for guidance camera data
    |
    └── ...

2. Signal extraction

Complete extraction pipeline for both AIRS-CH0 and FGS1 data

Takes signal corrected data from SignalCorrection() and extracts clean spectral signals through intelligent pixel selection and data reduction. This module transforms 3D detector arrays into focused time series suitable for exoplanet atmospheric analysis.

Processing Features:

• AIRS-CH0 Extraction: Selects brightest detector rows containing spectral traces, sums to create 1D spectra per frame
• FGS1 Extraction: Uses 2D block extraction to identify signal regions, collapses to single brightness value per frame
• Combined Output: Merges FGS1 and AIRS-CH0 signals (FGS1 as first column for transit detection)
• Adaptive Thresholding: Automatically selects signal-bearing pixels based on configurable intensity thresholds
• Optional Smoothing: Applies moving average filtering across wavelengths to reduce noise
• Massive Data Reduction: Achieves ~97-98% volume reduction while preserving transit signals

Key Benefits:

• Dramatically faster downstream processing due to reduced data volume
• Improved signal-to-noise ratio by focusing on high-signal detector regions
• Preserved exoplanet transit signatures with cleaner temporal structure
• Unified processing for both instrument types

See the following notebooks for implementation details and analysis:

1. AIRS-CH0 signal extraction
2. FGS1 signal extraction
3. Wavelength smoothing

Example usage:

from ariel_data_preprocessing.signal_extraction import SignalExtraction

signal_extraction = SignalExtraction(
    input_data_path='data/corrected',
    output_data_path='data/extracted',
    input_filename='train.h5',
    inclusion_threshold=0.75,
    smooth=True,
    smoothing_window=200
)

output_file = signal_extraction.run()

Output data will be written to train.h5 by default in the directory passed to output_data_path. The structure of the HDF5 archive combines both AIRS-CH0 and FGS1 signals:

    train.h5
    |
    ├── planet_1/
    │   ├── signal  # Shape: (n_frames, n_wavelengths + 1) - FGS1 + AIRS-CH0
    │   └── mask    # Shape: (n_wavelengths + 1,) - combined mask
    │
    ├── planet_2/
    │   ├── signal  # Shape: (n_frames, n_wavelengths + 1) - FGS1 + AIRS-CH0  
    │   └── mask    # Shape: (n_wavelengths + 1,) - combined mask
    │
    └── ...

Note: The first column of the signal dataset contains the extracted FGS1 brightness time series, followed by the AIRS-CH0 spectral channels. This structure facilitates easy transit detection using FGS1 data while providing wavelength-dependent atmospheric information from AIRS-CH0.