metadata-asf

Python library and CLI to collect SAR acquisition metadata from the ASF API (asf_search) and export it as daily Parquet catalogs.

metadata-asf is read-only by design: it never downloads products, stores no Earthdata credentials, and ships without any heavy I/O dependency. The daily Parquet files it produces are the interface contract with the companion repository [fetch-asf] (planned), which performs the actual product transfer.

Scope

• Provider: ASF, through asf_search.
• Multi-mission by construction; first target mission is NISAR.
• NISAR products collected: L1 — RSLC, L2 — GSLC.
• Geographic filter configurable via WKT (oceans by default).
• Output: one Parquet file per acquisition day, {mission}_ocean_YYYYMMDD.parquet.

Out of scope (see AGENTS.md): downloading products, Earthdata authentication, and any scientific SAR processing. These live in fetch-asf.

Installation (development)

pip install -e ".[dev]"
pre-commit install

Python ≥ 3.10 is required. The project uses a src/ layout with Hatchling as build backend, version derived from the Git history via hatch-vcs, and an entry point metadata-asf registering metadata_asf.cli:main.

Usage

CLI — harvest

Query the ASF API and write the daily Parquet catalogs:

metadata-asf harvest \
  --mission NISAR \
  --outputdir ./out \
  --log-verbosity INFO \
  --start 2025-01-01 \
  --stop 2025-01-31 \
  --conf config.yaml

The window is harvested sequentially, one day at a time: each day is searched, extracted and written to its own Parquet file. A day whose file already exists is skipped, so a partially finished span can simply be re-run to completion.

* --start can be omitted only if a window is provided by --conf (date_range).

Precedence is CLI > --conf file > mission profile defaults. See config.example.yaml for a complete working example.

CLI — report

Render a self-contained HTML report on a directory of daily Parquet catalogs (read-only, no API calls, no JavaScript in the output):

metadata-asf report \
  --catalogdir ./out \
  --outputfile ./out/report.html

The report covers volume & completeness (records, daily files, span, missing/empty days), the product/instrument mix (platform, product type, level, beam, polarization) and footprint geometry quality (invalid / antimeridian / near-polar). It is illustrated with matplotlib figures (daily volume + cumulative records, mix panels, a footprint world map and a geometry-issue chart) rendered server-side and inlined as base64 PNGs, so the single HTML file stands alone with no JavaScript and no external assets.

Python API

The public, versioned API consumed by fetch-asf:

from pathlib import Path
import datetime as dt

from metadata_asf import search, extract, export

products = search.search(
    mission="NISAR",
    start=dt.date(2025, 1, 1),
    end=dt.date(2025, 1, 31),
    intersects_with=None,          # falls back to the profile's ocean WKT
    max_results=10_000,
)

df = extract.to_dataframe(products, mission="NISAR")
written = export.write_daily_parquet(df, output_dir=Path("./out"))

Parquet schema (guaranteed)

Files are written with the PyArrow engine, Snappy compression and no index; every timestamp is UTC. Any schema change implies a major version bump plus migration notes (see AGENTS.md, section 7).

Adding a mission

A mission is fully described by a profile in src/metadata_asf/profiles/. Adding one amounts to: creating the module, registering it in the MISSIONS dict and adding tests. The core code (search, extract, export) never changes — see AGENTS.md, section 3.

Development

make install          # editable install + dev dependencies
make test             # pytest with branch coverage (>= 80 % required)
make check-codestyle  # ruff + black --check
make mypy             # mypy --strict on src/ and tests/

Integration tests performing real ASF API calls opt in explicitly: pytest -m integration.

License

MIT — see LICENSE.