profe 1.1.0

Reduction Pipeline for OPTICAM Photometry of Exoplanets (PROFE in spanish)

PROFE: Pipeline de Reducción de OPTICAM para Fotometría de Exoplanetas

Reduction pipeline for OPTICAM photometry of exoplanets.

A Python-based pipeline to automate preprocessing and postprocessing of data acquired with the OPTICAM instrument on the 2.1 m telescope at OAN‑SPM, aimed for implementing the data reduction method proposed by Paez et. al (2026).

---

Features

• Preprocessing (profe -p):

  • Organize and standardize FITS files by target and date.
  • Update headers: compute and insert JD and UTMIDDLE keywords.
  • Apply a 3×3 median filter for hot-pixel correction.
  • Generate an image-count summary (images_summary.dat).

• Postprocessing (profe -o):

  • Multi-band binned light curves with RMS measurements (PDF & CSV).
  • Single-band ExoFOP-standardized light curve plots (PNG).
  • Time-averaging noise plots (Red vs White noise characterization via MC3).
  • Altitude–Azimuth trajectory plots.
  • Centroid movement plots.
  • Aperture visualization (Field of View) plots.
  • Radial (seeing) profile plots.
  • Comparison star light curves (up to 6 stars per band).
  • Automated WCS solving via Astrometry.net (source list method).
  • Transit timing data retrieval from TESS Transit Finder (TTF).
  • Consolidated ExoFOP notes report generation.
  • Archival copy of measurement tables per band.

---

Requirements

Dependencies defined in pyproject.toml:

• Python (≥3.11, <3.12)
• astropy (5.3.2)
• scipy (≥1.10, <2.0)
• matplotlib (≥3.10.3, <4.0.0)
• tqdm (≥4.67.1, <5.0.0)
• pandas (≥2.3.1, <3.0.0)
• mc3 (≥3.2.1, <4.0.0)
• photutils (≥2.2.0, <3.0.0)
• numpy (≥1.24, <2.0)
• astroquery (≥0.4.7)
• beautifulsoup4 (≥4.12.0)
• requests (≥2.31.0)

External Setup

WCS Solving — Provide an Astrometry.net API key:

1. Create a file named .astrometry_key at the root of your working directory.
2. Paste your 16-character API key inside the file.

Transit Timing (TTF) — Provide TESS Transit Finder credentials:

1. Create a file named .ttf_credentials at the root of your working directory.
2. Write your credentials in the format username:password.

ExoFOP Uploads — Provide ExoFOP credentials for individual file uploads:

1. Create a file named .exofop_credentials at the root of your working directory.
2. Write your credentials in the format username:password.

---

Installation

Install from PyPI:

pip install profe

For a development environment (includes testing, linting, and profiling tools):

git clone https://github.com/s-paez/profe.git
cd profe
pip install -e ".[dev]"

---

Usage

CLI Reference

PROFE provides a single entry point with mutually exclusive commands:

Command	Description
profe -p	Run the full preprocessing pipeline (organize + median filter).
profe --organice	Run only the file reorganization and header update stage.
profe --filter	Run only the median filter stage. Skips if corrected_3x3/ already exists.
profe -o [TARGET]	Run the full postprocessing and output generation pipeline. Optionally specify a target name to process only that target.
profe -pu [TARGET]	Prepare Upload: Pack local output products into an intermediate JSON manifest and prompt for an ExoFOP Data Tag.
profe -u [TARGET]	Upload: Iteratively post the prepared files to ExoFOP using individual file endpoints to preserve exact scientific file names.
profe man	Display the detailed built-in manual.
profe -h	Show the quick-reference help message.

Optional flags:

Flag	Description
-c CORES / --cores CORES	Number of CPU cores for preprocessing (default: all available). Only valid with -p, --organice, or --filter.
-o TARGET	When a target name is provided (e.g., profe -o "TOI-1234"), only that target is processed. Without a target, all targets are processed.

Preprocessing

PROFE recursively searches for original FITS files inside a data/ directory relative to where the command is executed. The pipeline organizes them, updates FITS headers (Julian Date and UTMIDDLE), and applies median filter corrections.

# Full preprocessing with all CPU cores
profe -p

# Full preprocessing with limited cores
profe -p -c 4

# Only reorganize files and update headers (no filter)
profe --organice

# Only apply the median filter (skips if corrected_3x3/ already exists)
profe --filter

Preprocessing Steps

1. Header Update — Computes JD (Julian Date at start) and UTMIDDLE (ISO mid-exposure) from existing UT and EXPOSURE keywords. Skips files already tagged with a PROFE HISTORY entry.
2. File Organization — Sorts FITS files into organized_data/{TARGET}/raw/{DATE}/ subdirectories based on OBJECT and DATE-OBS header keywords. Creates measurements/, lcs/, and exofop/ directories for science targets.
3. Summary Generation — Creates logs/images_summary.dat with image counts per target and date.
4. Median Filter — Applies a 3×3-pixel median filter (hot-pixel correction as proposed by Paez et al. 2026) and saves corrected images under corrected_3x3/. Adds HISTORY entries to both raw and filtered FITS headers to prevent reprocessing.

Output Directory Structure

After a successful preprocessing run, the data/ contents are translated into a structured organized_data/ hierarchy:

organized_data/
└── {TARGET}/
    ├── raw/
    │   └── {DATE}/              ← Original FITS (updated headers)
    ├── corrected_3x3/
    │   └── {DATE}/              ← 3×3 median filter products
    ├── measurements/
    │   └── {DATE}/              ← AIJ .tbl/.csv files (INPUT for -o)
    │       └── times/
    │           └── times.csv    ← Optional time intervals
    ├── lcs/
    │   └── {DATE}/              ← Normalized multi-band CSVs
    ├── plots/
    │   └── {DATE}/              ← Diagnostic PDFs and PNGs
    └── exofop/
        └── {DATE}/
            ├── {band}/          ← Standardized ExoFOP products
            └── AIJ/
                └── {band}/      ← AIJ fitpanel files for report generation

AstroImageJ Integration

After preprocessing, perform photometry in AstroImageJ using the files in corrected_3x3/. Save the measurement tables in the corresponding measurements/{DATE}/ directory.

Requirements for Measurement Files:

• Format: .tbl (tab-separated) or .csv (comma-separated).
• Naming: The filename must end with the band suffix (e.g., myfile_gp.tbl, obs1_rp.csv, or data_ip.tbl). Supported bands: g, gp, r, rp, i, ip.

Optional: Time Averaging & RMS Intervals

To define specific non-transit intervals for RMS calculation and noise analysis, create a times/times.csv file inside the date folder in measurements/:

• Columns: init_time, final_time (in JD or BJD units).
• Usage: If present, PROFE will use these intervals for legend RMS calculations and the time-averaging diagnostic plots. If absent, PROFE creates an empty template automatically.

Postprocessing

Run the following to generate all scientific products:

# Generate outputs for all targets
profe -o

# Generate outputs for a specific target only
profe -o "TOI-1234"

The postprocessing pipeline runs the following modules sequentially:

#	Module	Description
1	AltAzGuidingPlotter	Altitude–Azimuth polar plot and centroid displacement vs. time.
2	LightCurvePlotter	Multi-band 6-panel diagnostic light curves (PDF) + per-band ExoFOP PNGs + normalized CSV.
3	TimeAveragingPlotter	Red vs. white noise characterization using the MC3 time-averaging method (Cubillos et al. 2017).
4	FieldViewPlotter	Aperture visualization: source and sky annuli overlaid on a calibrated FITS image.
5	SeeingProfilePlotter	Radial brightness profile of the target star using photutils.RadialProfile.
6	ComparisonStarsPlotter	6-panel light curves for up to 6 comparison stars with sigma-clipped outlier removal.
7	AstrometrySolver	WCS solving via Astrometry.net using local source detection (Background2D + segmentation).
8	TransitDataManager	Retrieves predicted transit times from the TESS Transit Finder (TTF).
9	ReportGenerator	Generates a consolidated ExoFOP notes text file with multi-band metrics.

Each module checks for existing outputs before running and skips already-processed (target, date, band) triples.

Standardized ExoFOP Products

All files in exofop/{DATE}/{BAND}/ follow the naming convention: {TICID}-01_{YYYYMMDD}_OAN-SPM-2m1-OPTICAM_{BAND}_{TYPE}.{EXT}

TYPE	Description
_lightcurve	High-resolution 6-panel PNG plot of the light curve.
_field	Science image showing source and sky apertures.
_seeing-profile	Radial profile of the target star.
_WCS	WCS-solved FITS with astrometric header.
_compstar-lightcurves	Diagnostic plot of up to 6 comparison star light curves.
_measurements	A copy of the original photometry table (.tbl).

Additional products at the date level (exofop/{DATE}/):

| *_transit_times.dat | Predicted transit ingress, mid, and egress times from TTF. | | *_notes.txt | Consolidated ExoFOP report with multi-band metrics and timing analysis. |

ExoFOP Uploading

PROFE can directly upload all standardized products to the ExoFOP single-file upload endpoint (preserving the exact file names) for targets that have already completed the postprocessing stage.

1. Prepare Upload (profe -pu [TARGET]): Scans the local exofop/ directories and prompts you for a Data Tag for each pending date. It collects the valid files and generates an intermediate JSON metadata manifest.

2. Upload (profe -u [TARGET]): Reads the prepared manifests and iteratively uploads each file individually to ExoFOP.

   • Authenticates using your local .exofop_credentials file.
   • Automatically assigns the correct ExoFOP target and planet parameters.
   • Derives the correct ExoFOP description from each file's name (e.g., Light Curve, Field of View, WCS FITS Image).
   • Sets the 12-month proprietary period by default.

Note: The upload system ensures files are only processed once. If a file already exists on ExoFOP or an upload fails, PROFE will log the error and continue with the remaining items.

Logging

All pipeline runs generate timestamped log files in the logs/ directory:

logs/
├── profe_preprocess_20260420_143022.log
├── profe_organize_20260420_150112.log
├── profe_filter_20260420_151530.log
└── profe_output_20260420_160045.log

Each execution creates a new log file to prevent overwriting previous records.

Manual

To see the complete detailed manual and arguments at any time, run:

profe man

---

Architecture

profe/
├── cli.py                  ← Central CLI entry point (argparse)
├── logger.py               ← Timestamped logging configuration
├── preprocess/
│   ├── cli.py              ← Preprocessing orchestrator
│   ├── fits_processor.py   ← Header update, file organization, summary
│   └── median_filter.py    ← 3×3 median filter (multiprocessing)
└── output/
    ├── cli.py              ← Postprocessing orchestrator
    ├── naming.py           ← ExoFOP naming conventions & TOI→TIC resolution
    ├── alt_az_centroid.py   ← Altitude–Azimuth & centroid plots
    ├── light_curves.py      ← Multi-band light curves (PDF/PNG/CSV)
    ├── correlated_noise.py  ← Time-averaging red noise analysis
    ├── field_view.py        ← Aperture visualization plots
    ├── seeing_profile.py    ← Radial profile plots
    ├── comparison_stars.py  ← Comparison star light curves
    ├── astrometry_out.py    ← WCS solving via Astrometry.net
    ├── transit_info.py      ← TTF transit data retrieval
    └── report_generator.py  ← Consolidated ExoFOP notes

---

Profiling

To analyze the performance of the pipeline, you must have the development dependencies installed. Additionally, if you are testing this over a specific data directory, it is recommended to activate the virtual environment so the commands work globally:

# 1. Activate the poetry environment shell
poetry shell

# 2. Navigate to your working directory (e.g. where your data/ is) and run the profiler
# We use the `--html` flag to successfully export the interactive report
pyinstrument --html -o profe_profile.html -m profe.cli -p -c 4

This will generate a detailed interactive profe_profile.html report showing execution time for each function call.

Development & Contribution

We welcome contributions to improve PROFE! Please follow these steps to ensure a smooth process:

1. Fork the repository on GitHub and clone your fork locally:

   git clone https://github.com/<username>/profe.git
   cd profe
   

2. Create a new branch for your feature of bugfix:

   git checkout -b feat/new-feature
   git checkout -b fix/issue-123
   

3. Install development dependencies:

   pip install -e ".[dev]"
   

4. Enable and run pre-commit hooks (for code style and quality checks):

   pre-commit install
   pre-commit run --all-files
   

5. Commit and push your changes to your fork
6. Open a Pull Request from your fork to the main repository. In your PR description:
   • Explain the what and why of the change
   • Reference related issues

---

License

This project is licensed under the MIT License. See the LICENSE file for details.