Hestia Distribution

Prior and posterior distributions of crop yield and input use (fertiliser, pesticide, irrigation), per country and per product, for the Hestia platform. Use them to get an expected range for a value, or to score how plausible a reported value is.

What it does

For a (country, product) pair the library produces a (mu, sigma) normal distribution for each quantity, built in two layers:

• Prior — the starting belief, derived from FAOSTAT. For yield it is the national yield time series; for inputs it is national input use. The mean is an area-weighted average over the years, and the spread is the year-to-year variation of that national figure, floored by a cross-country coefficient of variation so a country with a flat series is not given an implausibly tight interval. Priors are cheap and exist for every (country, product) FAO reports.

• Posterior — the prior updated with Hestia's own Cycle data for that pair, via Bayesian MCMC (a small Stan model sampled with cmdstanpy). Where enough real cycles exist the posterior is dominated by them and departs from the prior; where they don't, it stays close to the prior. A posterior is only published if the data can support it: at least MIN_CYCLES cycles (default 30) and a fit that converged (r̂ ≤ 1.01). Otherwise no posterior is written and the prior stands — so a consumer always has something to fall back to.

The read path (getting a distribution back out) is pure lookup: it reads pre-built files from a local folder and needs no sampler. Only generating posteriors needs cmdstanpy and CmdStan.

Install

pip install hestia_earth.distribution

Reading distributions needs only this. To generate posterior files, also install the sampler:

pip install hestia_earth.distribution[stats]
./install-cmdstan.sh

cmdstanpy drives CmdStan, which is compiled on the machine rather than installed from pip — hence the second step. It needs a C++ toolchain (build-essential on Debian/Ubuntu, the Xcode command line tools on macOS) and takes a few minutes the first time.

Getting the files

The library reads pre-built distribution files from a local folder, set by DISTRIBUTION_DATA_FOLDER (default ./data). The files are published at https://api.hestia.earth/distribution/files as a JSON list of download URLs — the four FAO prior CSVs and one posterior CSV per country.

Run this once to populate the folder (no credentials needed):

import json
import os
import urllib.request

DATA_FOLDER = os.getenv("DISTRIBUTION_DATA_FOLDER", "./data")

files = json.load(urllib.request.urlopen("https://api.hestia.earth/distribution/files"))
for group, subfolder in [("priorFiles", "prior_files"), ("posteriorFiles", "posterior_files")]:
    folder = os.path.join(DATA_FOLDER, subfolder)
    os.makedirs(folder, exist_ok=True)
    for f in files[group]:
        urllib.request.urlretrieve(f["url"], os.path.join(folder, f["name"]))
    print(f"downloaded {len(files[group])} {subfolder}")

That lays out <DATA_FOLDER>/prior_files/ and <DATA_FOLDER>/posterior_files/, which is exactly where the accessors below look. The download URLs are short-lived, so fetch the list fresh each time rather than saving a URL to reuse. Each file is then read once per process and cached, so checking many values costs one read per file, not one per lookup.

Getting distribution data

Each quantity has its own module, and every one exposes the same two accessors:

• get_prior(...) — the FAO-derived prior, always available where FAO has data.
• get_post(...) — the posterior, or (None, None) when none was published for that pair. When it is (None, None), fall back to get_prior(...).

Both return a (mu, sigma) tuple.

from hestia_earth.distribution.prior_yield import get_prior as get_prior_yield
from hestia_earth.distribution.posterior_yield import get_post as get_post_yield

get_prior_yield("GADM-GBR", "wheatGrain")   # -> (8061.2, 3736.7)   FAO prior
get_post_yield("GADM-GBR", "wheatGrain")    # -> (7900.4, 1970.5)   updated with Hestia cycles
get_post_yield("GADM-GBR", "oatGrain")      # -> (None, None)       no posterior; use the prior

The other quantities follow the same shape, differing only in what identifies the row:

# Fertiliser: keyed by product AND input. The prior is now crop-specific - derived as
# yield x crop nutrient removal / recovery efficiency, where the removal is the nutrient in the
# harvested product - so a product is required (it was per-input before).
from hestia_earth.distribution.prior_fert import get_prior as get_prior_fert
from hestia_earth.distribution.posterior_fert import get_post as get_post_fert
get_prior_fert("GADM-GBR", "inorganicNitrogenFertiliserUnspecifiedKgN", "wheatGrain")  # (mu, sigma)
get_post_fert("GADM-GBR", "wheatGrain", "inorganicNitrogenFertiliserUnspecifiedKgN")  # (mu, sigma)

# Pesticide and irrigation: the prior is per country only.
from hestia_earth.distribution.prior_pest import get_prior as get_prior_pest
from hestia_earth.distribution.posterior_pest import get_post as get_post_pest
get_prior_pest("GADM-GBR")                  # (mu, sigma)
get_post_pest("GADM-GBR", "wheatGrain")     # (mu, sigma) or (None, None)

from hestia_earth.distribution.prior_irrigation import get_prior as get_prior_irri
from hestia_earth.distribution.posterior_irrigation import get_post as get_post_irri
get_prior_irri("GADM-GBR")                  # (mu, sigma)
get_post_irri("GADM-GBR", "wheatGrain")     # (mu, sigma) or (None, None)

get_post returns the mean of the posterior ensemble. If you need the full ensemble of draws rather than its mean, call get_post_ensemble(country_id, product_id) from the same module, which returns (mu_ensemble, sd_ensemble) as lists.

Validating a value

A (mu, sigma) distribution gives a confidence interval directly: with the usual 95% interval, a value is an outlier when it falls outside mu ± 1.96 · sigma. Prefer the posterior and fall back to the prior:

from hestia_earth.distribution.posterior_yield import get_post as get_post_yield
from hestia_earth.distribution.prior_yield import get_prior as get_prior_yield

def yield_interval(country_id, product_id, z=1.96):
    mu, sigma = get_post_yield(country_id, product_id)
    if mu is None:                                   # no posterior -> use the prior
        mu, sigma = get_prior_yield(country_id, product_id)
    if mu is None:                                   # no distribution at all
        return None
    return (max(mu - z * sigma, 0), mu + z * sigma)  # yield cannot be negative

yield_interval("GADM-GBR", "wheatGrain")   # -> (4038.2, 11762.7)

Joint (multivariate) plausibility

The interval above checks each quantity on its own. To score a combination — e.g. "is 8500 kg/ha of wheat alongside 200 kg N/ha jointly plausible for the UK?" — use the multivariate fit, which accounts for the correlation between yield and input use. This path needs scipy (included in the [stats] extra) and reads the underlying cycle data.

from hestia_earth.distribution.utils.MCMC_mv import calculate_fit_2d

# candidate follows the column order [Nitrogen (kg N), Grain yield (kg/ha)]
likelihood, _ranges = calculate_fit_2d([200, 8500], "GADM-GBR", "wheatGrain")
# likelihood ~ how plausible the pair is (Monte Carlo integration over the joint density);
# roughly, the fraction of observed samples it stands above. Above ~5% is acceptable.

Generating distributions

Building prior and posterior files (rather than reading them) is done through each module's generate_* and update_all_post functions — for example generate_prior_yield_file(overwrite=True) and update_all_post(country_id, product_ids=..., overwrite=False) in posterior_yield. Generating posteriors samples the Stan model and therefore needs the [stats] extra and CmdStan installed (see Install). Passing product_ids restricts the run to the products that actually have cycles in a country, which is far cheaper than walking every product.

Configuration reference