fitsm

fitsm is a command-line interface and Python package for indexing FITS files into an SQL database. It extracts metadata from FITS headers and organizes it for easy querying.

For example, the command

fitsm index fits_folder

makes a SQLite database with metadata from FITS files and their corresponding observations. A python package then provides convenience functions. For example

from fitsm import db

con = db.connect("db.sqlite")

db.observations_files(con, "dark", "2020-04-01", filter="a", exposure=20.0, tolerance=5, past=3)

returns a list of dark calibration files

• with exposure times of 20 seconds +/- 5 seconds.
• taken at most 3 days prior to the science frames on a specific date.

Installation

fitsm is available on PyPI. It is recommended to install it in a fresh Python virtual environment. You can use uv for fast and reproducible environment management, or use venv/pip as you prefer.

Using PyPI (recommended)

uv venv
source .venv/bin/activate
uv pip install fitsm

From source

git clone https://github.com/lgrcia/fitsm.git
cd fitsm
uv venv
source .venv/bin/activate
uv sync
uv pip install -e .

This will install the fitsm CLI and all dependencies.

Instrument YAML Configuration

The instrument configuration YAML file defines how FITS header keywords are mapped to database fields and how instrument names are recognized. This file is required for the index command and is specified using the -i or --instruments option in the CLI. The CLI uses this configuration to correctly interpret FITS headers for different instruments and to standardize the metadata stored in the database.

Example Structure

default:
    instrument_names:
        default: ["default",]
    definition:
        keyword_instrument: "TELESCOP"
        keyword_object: "OBJECT"
        keyword_image_type: "IMAGETYP"
        keyword_light_images: "light"
        keyword_dark_images: "dark"
        keyword_flat_images: "flat"
        keyword_bias_images: "bias"
        keyword_observation_date: "DATE-OBS"
        keyword_exposure_time: "EXPTIME"
        keyword_filter: "FILTER"
        keyword_ra: "RA"
        keyword_dec: "DEC"
        keyword_jd: "JD"
        unit_ra : "deg"
        unit_dec : "deg"
        scale_jd : "utc"

speculoos:
    instrument_names:
        # these are all the possible names under the 'keyword_image_type' that
        # correspond to the Callisto instrument
        Callisto: ["speculoos-Callisto", "callisto"]
        Europa: ["speculoos-Europa",]
        Io: ["speculoos-Io",]
        Ganymede: ["speculoos-Ganymede",]
        Artemis: ["speculoos-Artemis", "artemis", "sno"]
    definition:
        keyword_light_images: "Light Frame"

Other:
    instrument_names:
        Trius-SX694: ["Trius-SX694",]
    definition:
        keyword_instrument: "INSTRUME"
        keyword_light_images: "Light_Frame"

Sections

• instrument_names: Maps instrument aliases to canonical names.
• definition: Maps FITS header keywords to logical fields used by the parser.

You can add more sections for different instruments as needed. The default section is used as a fallback.

CLI Usage

Index FITS Files

To index FITS files into a database, use:

fitsm index <folder> -i instruments.yaml [-o output.sqlite]

Arguments:

• <folder>: Path to the folder containing FITS files.
• -i, --instruments: Path to the instruments.yaml file defining instrument configurations. If not provided, a built-in default is used.
• -o, --output: (Optional) Path to the output database file. Defaults to db.sqlite in the folder.
• -p, --processes: (Optional) Number of processes to use for indexing (default: number of CPU cores).

Show Observations

Show observations from the database (supports regex, case-insensitive):

fitsm observations <db.sqlite> [-i INSTRUMENT] [-d DATE] [-f FILTER] [-o OBJECT] [--group-exposures True|False]

Show Files

Show files from the database (supports regex, case-insensitive):

fitsm files <db.sqlite> [-i INSTRUMENT] [-d DATE] [-f FILTER] [-o OBJECT]

Filtering Options (for show commands):

• -i, --instrument: (Optional, regex) Filter by instrument name.
• -d, --date: (Optional, regex) Filter by date (format: YYYY-MM-DD).
• -f, --filter: (Optional, regex) Filter by filter name.
• -o, --object: (Optional, regex) Filter by object name.
• --group-exposures: (Optional, for observations) Group all exposures together in the output (default: True).

Custom SQL Query

Run a custom SQL query on the database:

fitsm query <db.sqlite> "SELECT * FROM observations WHERE instrument REGEXP 'europa|io'"

Example:

fitsm index /path/to/fits/files -i /path/to/instruments.yaml -o mydata.sqlite -p 4
fitsm observations mydata.sqlite -i Europa -d 2024-01-01
fitsm files mydata.sqlite -f R
fitsm observations mydata.sqlite -i 'europa|io'  # regex, case-insensitive
fitsm query mydata.sqlite "SELECT * FROM files WHERE object REGEXP 'mars|jupiter'"

All regex filters are case-insensitive.

Development

Requirements

• Python 3.11+
• Dependencies listed in pyproject.toml.

Testing

Run unit tests using:

pytest

License

MIT License