ema-data-access

Lightweight Python tools to query and access EMA data.

Setup

Python environment (Poetry)

1. Install Poetry if you don't have it:

   curl -sSL https://install.python-poetry.org | python3 -
   

2. Create a virtual environment in the project directory:

   python3 -m venv venv
   source venv/bin/activate
   

3. Install dependencies:

   poetry install --extras "dev test"
   

4. Install pre-commit hooks:

   poetry run pre-commit install
   

Python environment (pip)

1. Create a virtual environment in the project directory:

   python3 -m venv venv
   source venv/bin/activate
   

2. Install the package, with the dev/test extras, from this checkout:

   pip install -e ".[dev,test]"
   

3. Install pre-commit hooks:

   pre-commit install
   

File naming conventions

Every file in the EMA archive must match one of the naming conventions below.

<payload> is one of rept, mista, embirs, emacs, prb, cnc, pnc, irc. <source> is one of flight, softsim, flatsat. <data_level> is one of l0, l1, l1a, l1b, l2, l2a, l2b, l3, ql.

Table	Convention
ancillary	ema_l1_anc_sc_<apid>_<YYYYMMDD>.csv
manifest	<payload>_manifest_<YYYYMMDDHHMM>.txt, or moc_manifest_<YYYYMMDDHHMM>.txt for a payload-less MOC manifest
housekeeping	ema_l0_hsk_<payload>_<source>_<YYYYMMDD>.pkts
science (L0)	ema_l0_sci_<payload>_<source>_<YYYYMMDD>.pkts
science (L1a+)	ema_<payload>_<data_level>_<YYYYMMDDtHHMMSS>_<descriptor>_<pred_rec>_v<version>[-<subversion>].fits, where <pred_rec> is p (predicted) or r (reconstructed)
mission_events	ema_mission_events_<start_date:YYYYMMDD>_<end_date:YYYYMMDD>.xml

SPICE kernel naming conventions

SPICE kernels follow NAIF conventions rather than the ema_l*_* pattern above. <start_date>/<end_date> are YYYYMMDD. <version> is free-form alphanumeric (e.g. v001) for the first three conventions below, and digits-only for the rest.

Convention	Pattern	Example
Spacecraft ephemeris	ema_<type>_<start_date>_<end_date>_<version>.bsp, <type> one of pred, recon, ref	ema_recon_20240101_20240201_v001.bsp
Attitude	ema_<type>_<start_date>_<end_date>_<version>.bc, <type> one of rck, pck	ema_rck_20240101_20240201_v001.bc
Body ephemeris	ema_<body>_<version>.bsp, <body> one of sun, venus, earth, mars, wes, chi, roc, va28, rc76, sg6, jus	ema_sun_v001.bsp
Planetary ephemeris	<type><version>.bsp, <type> one of de, mar	de440.bsp
Leapseconds	naif<version>.tls	naif0012.tls
Planetary constants	pck<version>.tpc or .bpc	pck00011.tpc
Spacecraft clock / frames	ema_<type>_<version>.tsc or .tf, <type> one of sclk, fk	ema_sclk_0012.tsc

Command Line Utility

Query the ancillary table

Query the ancillary table for files matching a set of filters. An API key is optional — without one, only released files are returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-ancillary --apid 1234 --file-extension csv

or with CLI flags

$ ema-data-access --url <url> --api-key <your-api-key> query-ancillary --apid 1234 --file-extension csv

Other available filters: --file-name, --timetag-start, --timetag-end, --version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_ancillary?apid=1234&file_extension=csv"

Query the housekeeping table

Query the housekeeping table for files matching a set of filters. An API key is optional. Without one, only released files are returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-housekeeping --payload mista

Other available filters: --file-name, --timetag-start, --timetag-end, --version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_housekeeping?payload=mista"

Query the science table

Query the science table for files matching a set of filters. An API key is optional. Without one, only released files are returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-science --payload embirs --data-level l1a

Other available filters: --file-name, --timetag-start, --timetag-end, --descriptor, --pred-rec, --file-extension, --major-version, --minor-version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_science?payload=embirs&data_level=l1a"

Query the mission events table

Query the mission_events table for event files matching a set of filters. An API key is optional. Without one, only released files are returned.

Events span a date range, so --start-date and --end-date define a query window and any event whose own range overlaps that window is returned.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-mission-events --start-date 20240101 --end-date 20240110

Other available filters: --file-name, --version, --md5checksum. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_mission_events?start_date=20240101&end_date=20240110"

Query the manifest table

Query the manifest table for files matching a set of filters. Manifest rows are public, so no API key is required.

$ ema-data-access --url <url> query-manifest --payload embirs

Other available filters: --file-name, --timetag-start, --timetag-end. Results are returned as JSON.

--payload moc matches MOC manifests, which have no payload of their own (moc_manifest_<YYYYMMDDHHMM>.txt).

Under the hood, this is equivalent to:

$ curl "<url>/query_manifest?payload=embirs"

Query the spice table

Query the spice table for kernel files matching a set of filters. An API key is optional, but not needed in practice — SPICE kernels are always released.

$ ema-data-access --url <url> query-spice --file-root naif

Other available filters: --file-name, --min-date-j2000, --max-date-j2000, --min-date-datetime, --max-date-datetime, --delivery-date-start, --delivery-date-end, --od-number, --version, --limit. Results are returned as JSON.

Under the hood, this is equivalent to:

$ curl "<url>/query_spice?file_root=naif"

Build a SPICE metakernel

Build a metakernel covering a time window, given in seconds past J2000.

$ ema-data-access --url <url> metakernel --start-time 0 --end-time 100000 > mission.tm

Other available options: --kernel-types (comma-separated kernel_type names, e.g. ephem_reconstructed,ephem_predicted) to restrict which kernels are included, --list-files to get the file names instead of metakernel text, and --require-coverage to error instead of returning a partial metakernel if the window isn't fully covered.

Under the hood, this is equivalent to:

$ curl "<url>/metakernel?start_time=0&end_time=100000" -o mission.tm

Download the kernels and furnish the metakernel

A metakernel only lists kernel file names — it doesn't bundle the kernels themselves. Download every file it references into the same directory as the metakernel, then furnish it with SpiceyPy. SPICE kernels are always released, so no API key is required.

import ema_data_access
import spiceypy

start_time, end_time = 0, 100000
kernel_dir = "kernels"

file_names = ema_data_access.metakernel(
    start_time=start_time, end_time=end_time, list_files=True
)
ema_data_access.download(file_names, destination=kernel_dir)

with open(f"{kernel_dir}/mission.tm", "w") as f:
    f.write(ema_data_access.metakernel(start_time=start_time, end_time=end_time))

spiceypy.furnsh(f"{kernel_dir}/mission.tm")

Upload a file

Upload a local file, or every file in a directory, to the EMA data archive. File names must match a known EMA naming convention, and uploading requires an API key with developer-level access — request one from the EMA PDC team. Files the API rejects are listed in an error once the rest are uploaded.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> upload path/to/ema_l1_anc_sc_1234_20240101.csv

or with CLI flags

$ ema-data-access --url <url> --api-key <your-api-key> upload path/to/dir

Under the hood, this requests presigned upload URLs and then PUTs each file, equivalent to:

$ RESPONSE=$(curl -s -X POST -H "x-api-key: $EMA_API_KEY" -H "Content-Type: application/json" \
    -d '{"filenames": ["ema_l1_anc_sc_1234_20240101.csv"]}' <url>/upload)
$ UPLOAD_URL=$(echo "$RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin)['results'][0]['upload_url'])")
$ curl -X PUT -H "Content-Type:" -T path/to/ema_l1_anc_sc_1234_20240101.csv "$UPLOAD_URL"

The -H "Content-Type:" (empty) is required — the presigned URL isn't signed with a content-type, and curl's guessed one will cause a signature mismatch against S3.

Download files

Download one or more files from the EMA data archive by name. Unreleased files require an API key with at least team-level access.

$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> download ema_l1_anc_sc_1234_20240101.csv

or with CLI flags

$ ema-data-access --url <url> --api-key <your-api-key> download ema_l1_anc_sc_1234_20240101.csv

By default, files are saved in the current directory. Pass --destination to save them in another directory:

$ ema-data-access --url <url> download naif0012.tls de440.bsp --destination path/to/dir

Under the hood, this requests presigned download URLs for all the files in one call, then downloads each one:

$ curl -X POST -H "x-api-key: $EMA_API_KEY" -H "Content-Type: application/json" \
    -d '{"filenames": ["naif0012.tls", "de440.bsp"]}' "<url>/download"

Each entry in the returned results has a download_url. Fetch each one without the API key; the presigned URL carries its own authorization:

$ curl -o naif0012.tls "<download_url>"

Importing as a package

import ema_data_access

ema_data_access.config["DATA_ACCESS_URL"] = "<url>"
ema_data_access.config["API_KEY"] = "<your-api-key>"

results = ema_data_access.query_ancillary(apid=1234, file_extension="csv")

results = ema_data_access.query_housekeeping(payload="mista")

results = ema_data_access.query_science(payload="embirs", data_level="l1a")

results = ema_data_access.query_mission_events(
    start_date="20240101", end_date="20240110"
)

results = ema_data_access.query_manifest(payload="embirs")

results = ema_data_access.query_spice(file_root="naif")

ema_data_access.upload("path/to/ema_l1_anc_sc_1234_20240101.csv")

ema_data_access.download(["naif0012.tls", "de440.bsp"], destination="path/to/dir")

Running tests

pytest