geoprepare

A Python package to prepare (download, extract, process input data) for GEOCIF and related models

Free software: MIT license
Documentation: https://ritviksahajpal.github.io/geoprepare

Installation

Install from PyPI

pip install --upgrade geoprepare

Install from GitHub (development)

pip install --upgrade --no-deps --force-reinstall git+https://github.com/ritviksahajpal/geoprepare.git

Local editable install

pip install -e ".[dev]"

CDS API (for AgERA5)

If you intend to download AgERA5 data, install the CDS API by following the instructions here.

MODIS data (octvi)

Install the octvi package to download MODIS data:

pip install git+https://github.com/ritviksahajpal/octvi.git

Downloading from the NASA DAACs requires a personal app key. After installation, run octviconfig in your command prompt. Information on obtaining app keys can be found here.

Pipeline

geoprepare follows a three-stage pipeline:

Download (geodownload) - Download and preprocess global EO datasets to dir_download and dir_intermed
Extract (geoextract) - Extract EO variable statistics per admin region to dir_output
Merge (geomerge) - Merge extracted EO files into per-country/crop CSV files for ML models and AgMet graphics

All datasets store files in year-specific subfolders (e.g., dir_intermed/cpc_tmax/2024/, dir_download/nsidc/2025/).

Additional utilities:

Move (geomove) - One-time migration of existing flat directories to year-specific subfolders
Check (geocheck) - Validate that expected TIF files exist in dir_intermed after download
Diagnostics (diagnostics) - Count and summarize files in the data directories

Analysis tools (separate from the production pipeline):

Extract cells (extract_cells) - Per-cell EO timeseries dump for downstream cell-mask optimisation by geocif.cell_optimizer

Usage

config_dir = "/path/to/config"  # full path to your config directory

cfg_geoprepare = [f"{config_dir}/geobase.txt", f"{config_dir}/countries.txt", f"{config_dir}/crops.txt", f"{config_dir}/geoextract.txt"]

1. Download data (`geodownload`)

Downloads and preprocesses global EO datasets. Only requires geobase.txt. The [DATASETS] section controls which datasets are downloaded. Each dataset is processed to global 0.05° TIF files in dir_intermed.

from geoprepare import geodownload
geodownload.run([f"{config_dir}/geobase.txt"])

2. Migrate to year subfolders (`geomove`)

Moves existing files from flat directories into year-specific subfolders. Run this once after upgrading to a version with year-subfolder support. All datasets are handled: CPC, ESI, NDVI, NSIDC, CHIRPS-GEFS, LST, Soil Moisture, AgERA5, VHI, FPAR, and AEF.

from geoprepare import geomove

# Preview what would be moved (no files are changed)
geomove.run([f"{config_dir}/geobase.txt"], dry_run=True)

# Execute the migration
geomove.run([f"{config_dir}/geobase.txt"])

3. Validate downloads (`geocheck`)

Checks that all expected TIF files exist in dir_intermed and are non-empty. Writes a timestamped report to dir_logs/check/.

from geoprepare import geocheck
geocheck.run([f"{config_dir}/geobase.txt"])

4. Extract crop masks and EO data (`geoextract`)

Extracts EO variable statistics (mean, median, etc.) for each admin region, crop, and growing season.

from geoprepare import geoextract
geoextract.run(cfg_geoprepare)

5. Merge extracted data (`geomerge`)

Merges per-region/year EO CSV files into a single CSV per country-crop-season combination.

from geoprepare import geomerge
geomerge.run(cfg_geoprepare)

Optional: Per-cell EO timeseries dump (`extract_cells`)

Analysis tool, separate from the production pipeline. Produces a long-format parquet of per-cell EO values per region (one row per cell × year × doy), consumed by geocif.cell_optimizer for GA-based cell-mask optimisation. Uses the same config files as geoextract; reads its own [CELL_OPTIMIZER] section from geoextract.txt for the variable list.

from geoprepare import extract_cells
extract_cells.run(cfg_geoprepare)

Output: ${PATHS:dir_output}/cell_optimizer/{country}/{crop}/{country}_{crop}_s{season}_cells.parquet. Schema: country, region, region_id, cell_id, lat, lon, afi, year, doy, <one float32 column per configured variable>. cell_id is stable across runs as long as the AFI raster and its CRS don't change.

Config files

File	Purpose	Used by
`geobase.txt`	Paths, dataset settings, boundary file column mappings, logging	both
`countries.txt`	Per-country config (boundary files, admin levels, seasons, crops)	both
`crops.txt`	Crop masks, calendar category settings (EWCM, AMIS)	both
`geoextract.txt`	Extraction-only settings (method, threshold, parallelism)	geoprepare
`geocif.txt`	Indices/ML/agmet settings, country overrides, runtime selections	geocif

Order matters: Config files are loaded left-to-right. When the same key appears in multiple files, the last file wins. The tool-specific file (geoextract.txt or geocif.txt) must be last so its [DEFAULT] values (countries, method, etc.) override the shared defaults in countries.txt.

config_dir = "/path/to/config"

cfg_geoprepare = [f"{config_dir}/geobase.txt", f"{config_dir}/countries.txt", f"{config_dir}/crops.txt", f"{config_dir}/geoextract.txt"]
cfg_geocif = [f"{config_dir}/geobase.txt", f"{config_dir}/countries.txt", f"{config_dir}/crops.txt", f"{config_dir}/geocif.txt"]

geobase.txt

Shared paths, dataset settings, boundary file column mappings, and logging. Key sections:

[DATASETS] — Which datasets to download (e.g. ['CHIRPS', 'CPC', 'NDVI', 'ESI', 'NSIDC'])
[PATHS] — All directory paths, derived from dir_base
Per-dataset sections ([CHIRPS], [CPC], [FLDAS], etc.) — Dataset-specific settings like data URLs, variables, fill values
Boundary file sections ([adm_shapefile], [gaul1_asap_v04], etc.) — Column mappings from shapefile fields to standard names (ADM0_NAME, ADM1_NAME, ADM_ID)
[DEFAULT] — Shared defaults: start_year, end_year, parallel_process, fraction_cpus

countries.txt

Per-country configuration. Each country section specifies boundary file, admin level, seasons, crops, and EO variables. Countries are grouped by calendar category:

AMIS countries — Inherit defaults, override crops as needed
EWCM countries — Set category = EWCM, use_cropland_mask = True, custom calendar_file and boundary_file
[DEFAULT] — Shared defaults including eo_model (list of EO variables to extract)

crops.txt

Crop mask filenames (e.g. [maize] mask = Percent_Maize.tif) and calendar category settings ([EWCM], [AMIS]).

geoextract.txt

Extraction settings for geoprepare. [DEFAULT] section sets method, redo, threshold, floor/ceil, parallel_extract, countries, and forecast_seasons. Optional [CELL_OPTIMIZER] section (read by extract_cells) configures the per-cell EO timeseries dump — variables = ["ndvi", "tmax", "tmin", "precip"] for the built-in short-name mapping, or a {column: eo_var} dict for explicit control.

geocif.txt

ML and agmet settings for geocif. Contains [AGMET] plotting config, per-country crop overrides, ML model definitions, and [ML] hyperparameters.

Supported datasets

Dataset	Description	Source
AEF	AlphaEarth Foundations satellite embeddings (64-band, 10m)	source.coop
AGERA5	Agrometeorological indicators (precipitation, temperature)	CDS
AVHRR	Long-term NDVI	NOAA NCEI
CHIRPS	Rainfall estimates (v2 and v3)	CHC
CHIRPS-GEFS	15-day precipitation forecasts	CHC
CPC	Temperature (Tmax, Tmin) and precipitation	NOAA CPC
ESI	Evaporative Stress Index (4-week, 12-week)	SERVIR
FLDAS	Land surface model outputs (soil moisture, precip, temp)	NASA
FPAR	Fraction of Absorbed Photosynthetically Active Radiation	JRC
LST	Land Surface Temperature (MODIS MOD11C1)	NASA
NDVI	Vegetation index from MODIS (MOD09CMG)	NASA
NOAA S2S	Monthly-to-seasonal t2m/precip forecasts + hindcasts (4 models, leads 1-6); feeds pre-season yield forecasting incl. seasons that have not started yet	NOAA PSL
NSIDC	SMAP L4 soil moisture (surface, rootzone)	NASA NSIDC
SOIL-MOISTURE	NASA-USDA soil moisture (surface as1, subsurface as2)	NASA
VHI	Vegetation Health Index	NOAA STAR
VIIRS	Vegetation index from VIIRS (VNP09CMG)	NASA

Directory layout

All datasets organize files into year-specific subfolders. After running geomove (or on fresh downloads), the directory structure looks like:

dir_download/
  nsidc/2025/*.h5, nsidc/2026/*.h5
  chirps_gefs/2026/*.tif
  fpar/2024/*.tif, fpar/2025/*.tif
  modis_lst/*.hdf                     (flat - pymodis manages this)
  ...

dir_intermed/
  cpc_tmax/2024/*.tif, cpc_tmax/2025/*.tif
  cpc_tmin/2024/*.tif, ...
  cpc_precip/2024/*.tif, ...
  chirps/v3/global/2024/*.tif, ...    (CHIRPS already used year subfolders)
  chirps_gefs/2026/*.tif
  esi_4wk/2024/*.tif, ...
  esi_12wk/2024/*.tif, ...
  ndvi/2024/*.tif, ...
  lst/2024/*.tif, ...
  nsidc/subdaily/2025/*.tif
  nsidc/daily/surface/2025/*.tif
  nsidc/daily/rootzone/2025/*.tif
  soil_moisture_as1/2024/*.tif, ...
  soil_moisture_as2/2024/*.tif, ...
  agera5/tif/{variable}/2024/*.tif, ...
  vhi/global/2024/*.tif, ...
  aef/{country}/2018/*.tif, ..., aef/{country}/aef_avg_global.tif
  fldas/.../2024/*.tif, ...           (FLDAS already used year subfolders)

Behaviour changes in 0.6.328 (2026-09-30 audit)

The code audit in the geocif repo (tasks/audit_2026-09-30.md) closed these geoprepare bugs. Values in merged files change, so delete the per-year CID CSVs of affected projects before re-running indices_runner.

redo = True now recomputes the current year too (geoextract); with redo = False every year up to the current one keeps the days it has and fills the gaps. Revised prelim-to-final values therefore need a redo.
One crop-mask rule. Static and monthly layers (AEF, SoilGrids, DEM, PI, FLDAS, CHIRPS-MFC) use the same strict afi > floor cell selection as the daily extraction and an AFI-weighted mean instead of a plain mean. Files already on disk are not touched: re-extract them with redo = True or delete their CSVs first.
geomerge no longer interpolates forecast leads or static layers when it gap-fills daily variables; a missing forecast month stays missing. Daily variables join on country/region/year/doy only (float lat/lon noise no longer splits rows), only *_{var}_{crop}.csv files directly under a variable directory are merged, and a combination that fails deletes the previous run's merged CSV so downstream stages fail loudly.
Harvest-season labels end on the last harvest day (one-day off-by-one removed) and dekads are (doy-1)//10 + 1, capped at 36.
CHIRPS-GEFS extracts the full 16-day window across a year boundary (the next-year combination is added automatically from 17 December, end_year need not include it); pre-season rows join on integer region ids; countries.csv zone text joins on the slugged country name and never doubles rows.
Downloads are written to a .part file, checked against Content-Length and renamed atomically, with request timeouts (FLDAS, NSIDC, AEF, ESI, CPC).
read_config raises FileNotFoundError for a missing config file instead of silently continuing; check_or_clean reads the year token before the variable name (a region id that looks like a year no longer deletes complete files) and never deletes static layers.
CHIRPS prelim marker is removed only after the final-derived tif is in place.

Upload package to PyPI

# 1. Bump version
uvx bump2version patch --current-version X.X.X --new-version X.X.Y pyproject.toml geoprepare/__init__.py

# 2. Clean, build, upload
rm -rf dist/ build/ *.egg-info/
uv build
uvx twine upload dist/geoprepare-X.X.Y*

Credits

This project was supported by NASA Applied Sciences Grant No. 80NSSC17K0625 through the NASA Harvest Consortium, and the NASA Acres Consortium under NASA Grant #80NSSC23M0034.

Metadata

Release files for geoprepare 0.6.331

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for geoprepare 0.6.331
File	Size	Uploaded
geoprepare-0.6.331.tar.gz	341.8 kB	Oct 4, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for geoprepare 0.6.331
File	Interpreter	ABI	Platform	Reset
geoprepare-0.6.331-py3-none-any.whl 318.4 kB Oct 4, 2026	Python 3	none	any	Details

Total release size: 660.2 kB

Release files / geoprepare-0.6.331.tar.gz

Download URL	geoprepare-0.6.331.tar.gz
Size	341.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`e128f18b84c7f4725b8b0e120fe8c7bc9c9bf6f6498310ed3354550c91593135`
BLAKE2b-256 checksum How to use checksums	`caf97a3e086e6f3268aab9613cfffd986a4b222c4d9dd42d81733fa44b5235e4`
Upload date	Oct 4, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.11`

Release files / geoprepare-0.6.331-py3-none-any.whl

Download URL	geoprepare-0.6.331-py3-none-any.whl
Size	318.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`4f9015ab73c567167048f748c003582c51692f0428cd5882b1c425ad33732e06`
BLAKE2b-256 checksum How to use checksums	`ba6111b9ccd02a5142684ae77a8e2939b9f8fa08ba6f97ddd71ce512e6060dc4`
Upload date	Oct 4, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.11`