ema-data-access

Lightweight Python tools to query and access EMA data.

Setup

Python environment (Poetry)

1. Install Poetry if you don't have it:

   curl -sSL https://install.python-poetry.org | python3 -
   

2. Create a virtual environment in the project directory:

   python3 -m venv venv
   source venv/bin/activate
   

3. Install dependencies:

   poetry install --extras "dev test"
   

4. Install pre-commit hooks:

   poetry run pre-commit install
   

Python environment (pip)

1. Create a virtual environment in the project directory:

   python3 -m venv venv
   source venv/bin/activate
   

2. Install the package, with the dev/test extras, from this checkout:

   pip install -e ".[dev,test]"
   

3. Install pre-commit hooks:

   pre-commit install
   

File naming conventions

Every file in the EMA archive must match one of the naming conventions below.

<payload> is one of mst, emb, emc, rpt, ldr. <data_level> is one of l0, l1, l1a, l1b, l2, l2a, l2b, l3, ql.

Table	Convention
ancillary	ema_l1_anc_sc_<apid>_<YYYYMMDD>.csv
manifest	<payload>_manifest_<YYYYMMDDHHMM>.txt, or moc_manifest_<YYYYMMDDHHMM>.txt for a payload-less MOC manifest
housekeeping	ema_l0_hsk_<payload>_<YYYYMMDD>.pkts
science (L0)	ema_l0_sci_<payload>_<YYYYMMDD>.pkts
science (L1a+)	ema_<payload>_<data_level>_<YYYYMMDDtHHMMSS>_<descriptor>_<pred_rec>_v<version>[-<subversion>].fits, where <pred_rec> is p (predicted) or r (reconstructed)
mission_events	ema_mission_events_<start_date:YYYYMMDD>_<end_date:YYYYMMDD>.xml

SPICE kernel naming conventions

SPICE kernels follow NAIF conventions rather than the ema_l*_* pattern above. <start_date>/<end_date> are YYYYMMDD. <version> is free-form alphanumeric (e.g. v001) for the first three conventions below, and digits-only for the rest.

Convention	Pattern	Example
Spacecraft ephemeris	ema_<type>_<start_date>_<end_date>_<version>.bsp, <type> one of pred, recon, ref	ema_recon_20240101_20240201_v001.bsp
Attitude	ema_<type>_<start_date>_<end_date>_<version>.bc, <type> one of rck, pck	ema_rck_20240101_20240201_v001.bc
Body ephemeris	ema_<body>_<version>.bsp, <body> one of sun, venus, earth, mars, wes, chi, roc, va28, rc76, sg6, jus	ema_sun_v001.bsp
Planetary ephemeris	<type><version>.bsp, <type> one of de, mar	de440.bsp
Leapseconds	naif<version>.tls	naif0012.tls
Planetary constants	pck<version>.tpc or .bpc	pck00011.tpc
Spacecraft clock / frames	ema_<type>_<version>.tsc or .tf, <type> one of sclk, fk	ema_sclk_0012.tsc

Command Line Utility

Query the ancillary table

Query the ancillary table for files matching a set of filters. An API key is optional — without one, only released files are returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-ancillary --apid 1234 --file-extension csv

or with CLI flags

$ ema-data-access --url <url> --api-key <your-api-key> query-ancillary --apid 1234 --file-extension csv

Other available filters: --file-name, --timetag-start, --timetag-end, --version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_ancillary?apid=1234&file_extension=csv"

Query the housekeeping table

Query the housekeeping table for files matching a set of filters. An API key is optional. Without one, only released files are returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-housekeeping --payload mst

Other available filters: --file-name, --timetag-start, --timetag-end, --version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_housekeeping?payload=mst"

Query the science table

Query the science table for files matching a set of filters. An API key is optional. Without one, only released files are returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-science --payload emb --data-level l1a

Other available filters: --file-name, --timetag-start, --timetag-end, --descriptor, --pred-rec, --file-extension, --major-version, --minor-version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_science?payload=emb&data_level=l1a"

Query the mission events table

Query the mission_events table for event files matching a set of filters. An API key is optional. Without one, only released files are returned.

Events span a date range, so --start-date and --end-date define a query window and any event whose own range overlaps that window is returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-mission-events --start-date 20240101 --end-date 20240110

Other available filters: --file-name, --version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_mission_events?start_date=20240101&end_date=20240110"

Query the manifest table

Query the manifest table for files matching a set of filters. Manifest rows are public, so no API key is required.

$ ema-data-access --url <url> query-manifest --payload emb

Other available filters: --file-name, --timetag-start, --timetag-end. Results are returned as JSON.

--payload moc matches MOC manifests, which have no payload of their own (moc_manifest_<YYYYMMDDHHMM>.txt).

Under the hood, this is equivalent to:

$ curl "<url>/query_manifest?payload=emb"

Query the spice table

Query the spice table for kernel files matching a set of filters. An API key is optional, but not needed in practice — SPICE kernels are always released.

$ ema-data-access --url <url> query-spice --file-root naif

Other available filters: --file-name, --min-date-j2000, --max-date-j2000, --min-date-datetime, --max-date-datetime, --delivery-date-start, --delivery-date-end, --od-number, --version, --limit. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl "<url>/query_spice?file_root=naif"

Build a SPICE metakernel

Build a metakernel covering a time window, given in seconds past J2000.

$ ema-data-access --url <url> metakernel --start-time 0 --end-time 100000 > mission.tm

Other available options: --kernel-types (comma-separated kernel_type names, e.g. ephem_reconstructed,ephem_predicted) to restrict which kernels are included, --list-files to get the file names instead of metakernel text, and --require-coverage to error instead of returning a partial metakernel if the window isn't fully covered.

Under the hood, this is equivalent to:

$ curl "<url>/metakernel?start_time=0&end_time=100000" -o mission.tm

Download the kernels and furnish the metakernel

A metakernel only lists kernel file names — it doesn't bundle the kernels themselves. Download every file it references into the same directory as the metakernel, then furnish it with SpiceyPy. SPICE kernels are always released, so no API key is required.

import ema_data_access
import spiceypy

start_time, end_time = 0, 100000
kernel_dir = "kernels"

for file_name in ema_data_access.metakernel(
    start_time=start_time, end_time=end_time, list_files=True
):
    ema_data_access.download(file_name, destination=kernel_dir)

with open(f"{kernel_dir}/mission.tm", "w") as f:
    f.write(ema_data_access.metakernel(start_time=start_time, end_time=end_time))

spiceypy.furnsh(f"{kernel_dir}/mission.tm")

Upload a file

Upload a local file to the EMA data archive. The file name must match a known EMA naming convention, and requires an API key with developer-level access — request one from the EMA PDC team.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> upload path/to/ema_l1_anc_sc_1234_20240101.csv

or with CLI flags

$ ema-data-access --url <url> --api-key <your-api-key> upload path/to/ema_l1_anc_sc_1234_20240101.csv

Under the hood, this requests a presigned upload URL and then PUTs the file to it, equivalent to:

$ RESPONSE=$(curl -s -X POST -H "x-api-key: $EMA_API_KEY" <url>/upload/ema_l1_anc_sc_1234_20240101.csv)
$ UPLOAD_URL=$(echo "$RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin)['upload_url'])")
$ curl -X PUT -H "Content-Type:" -T path/to/ema_l1_anc_sc_1234_20240101.csv "$UPLOAD_URL"

The -H "Content-Type:" (empty) is required — the presigned URL isn't signed with a content-type, and curl's guessed one will cause a signature mismatch against S3.

Download a file

Download a file from the EMA data archive by name. Unreleased files require an API key with at least team-level access.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> download ema_l1_anc_sc_1234_20240101.csv

or with CLI flags

$ ema-data-access --url <url> --api-key <your-api-key> download ema_l1_anc_sc_1234_20240101.csv

By default, the file is saved in the current directory under its own name. Pass --destination to save it elsewhere, either as a directory or a full file path:

$ ema-data-access --url <url> download ema_l1_anc_sc_1234_20240101.csv --destination path/to/dir

If the destination file already exists, the download is skipped. Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" -o ema_l1_anc_sc_1234_20240101.csv "<url>/download/ema_l1_anc_sc_1234_20240101.csv"

Importing as a package

import ema_data_access

ema_data_access.config["DATA_ACCESS_URL"] = "<url>"
ema_data_access.config["API_KEY"] = "<your-api-key>"

results = ema_data_access.query_ancillary(apid=1234, file_extension="csv")

results = ema_data_access.query_housekeeping(payload="mst")

results = ema_data_access.query_science(payload="emb", data_level="l1a")

results = ema_data_access.query_mission_events(
    start_date="20240101", end_date="20240110"
)

results = ema_data_access.query_manifest(payload="emb")

results = ema_data_access.query_spice(file_root="naif")

ema_data_access.upload("path/to/ema_l1_anc_sc_1234_20240101.csv")

ema_data_access.download("ema_l1_anc_sc_1234_20240101.csv", destination="path/to/dir")

Running tests

pytest