ASTA

Config-driven analysis of agricultural evidence datasets. The first dataset is ERA (Evidence for Resilient Agriculture); the architecture assumes more will follow.

What it does

ERA pairs a management practice against a control and records an outcome — one row per observation, 112,859 of them drawn from 2,011 studies across 52 African countries. ASTA answers which country × crop × practice × outcome combinations the evidence base actually covers, and how large the measured effects are.

It produces nine charts, four maps, and a CSV behind each one:

Install

The map stack is binary — PyGMT needs the GMT C library plus its DCW country polygons and GSHHG shorelines, which pip cannot provide. Use pixi:

pixi install
pixi run python -c "import pygmt; pygmt.clib.Session()"   # proves GMT initialised

Charts alone work from a plain pip install (pip install -e .); only the maps need conda-forge.

Data

The ERA files are not in this repo — the compilation is 85 MB. Obtain them from CIAT/ERA_dev and place them here:

inputs/ERA/
├── 01a. ERA_Compiled_full.csv
├── Practice_Codes.csv
└── Outcome_Codes.csv

Then point dir_base in config/asta_base.txt at your checkout.

The source files are Windows-1252 encoded, not UTF-8. config/era.txt already declares this; a plain UTF-8 read fails on the accented characters in the practice definitions.

Run

pixi run analysis

or

from asta import analysis
analysis.run(["config/asta_base.txt", "config/era.txt"])

Output lands in a date-stamped tree:

outputs/era_combinations/<Month_DD_YYYY>/
├── plots/    F1–F9 as PNG + PDF
├── maps/     M1–M4 as PNG + PDF
└── tables/   the aggregation behind every figure

Downloading the papers

pixi run download-papers

Attempts to fetch the open-access PDF behind each of the 2,011 studies — Unpaywall, then OpenAlex, Semantic Scholar, CGSpace (CGIAR's repository; matches are verified against the item's DOI), CORE (with a key), and the bibliography's own URL last — every payload validated by PDF magic bytes. Nothing bypasses paywalls: publishers that refuse scripted access (MDPI, Wiley, Elsevier return 403 regardless of the request) are recorded as failures, not worked around.

One licensed extension: with an Elsevier API key in the gitignored config/papers_local.txt (see the comment in [PAPERS]), 10.1016 DOIs that have no OA copy are requested from Elsevier's Article Retrieval API — text and data mining permitted under an institutional ScienceDirect subscription, so full text is only served on an entitled network (campus IP or VPN). Without a key, behaviour is unchanged.

The bibliography itself (ERA_Bibliography.csv) is a cache, built on first run from the ERAg package's .rda — the Dataverse copy of the same table is behind a guestbook the API cannot pass.

PDFs land in inputs/ERA/papers/ under a standard name:

{DATASET}_{STUDYCODE}__{FirstAuthorSurname}_{Year}.pdf
e.g.  ERA_NJ0001__Abass_2018.pdf

Beside them, manifest.csv holds one row per study — downloaded or not — with a truthful status (downloaded, already_present, no_bibliography, no_doi, no_oa_location, download_failed, not_a_pdf, not_attempted) plus the DOI, resolver, URL and failure detail. missing.csv is the subset that did not download, with the reason. Reruns are idempotent: successes are skipped, failures retried (retry_failed), and a paper fetched manually and dropped in under its standard name is picked up as already_present.

Knobs live in [PAPERS] in config/era.txt; set max_papers = 15 for a smoke test.

Configuration

config/asta_base.txt holds paths, logging, and the list of active datasets. config/era.txt holds everything tunable about the ERA analysis. The knobs that change conclusions:

Adding a dataset

Three edits, none of which touch existing code:

1. Add asta/datasets/<NAME>.py exposing run(base, section="<NAME>").
2. Add a [<NAME>] section to a config file.
3. Add a branch to the dispatch loop in asta/analysis.py and list the dataset in [DATASETS] datasets.

BaseASTA (asta/base.py) already gives you config parsing, resolved paths, a date-stamped output tree, a logger, and save_table().

Design notes

Colour is not ad hoc. asta/style.py holds a palette validated for colour-vision deficiency all-pairs in both light and dark: three categorical slots (the three ERA outcome pillars), a single-hue sequential ramp for counts, and a blue–grey–red diverging ramp pinned at zero for signed effects. Counts span three orders of magnitude, so count heatmaps use a log scale and label it.

Effect sizes are aggregated on yi, the log response ratio, and back-transformed for display. The pc column is floored at −100 % with a 99th percentile of +1991 %, so its median would misstate the centre.

The UpSet plot (F11) is hand-rolled on matplotlib rather than using the upsetplot package, for two reasons: the package crashes under pandas 3 / matplotlib 3.11 (deprecated inplace chained assignment, then a hard failure in ax.scatter), and it builds and styles its own figure, which fights style.py.

F11 targets outcomes, not practices, and that choice is measured rather than assumed. UpSet exists to reveal higher-order intersections; practice bundles do not have them — 90.3 % of bundles are a single practice or a pair, and the top 25 bundles contain no three-practice bundle at all (the first ranks 33rd), so an UpSet there would duplicate F6 and F1. Outcome sets carry 39.5 % of studies at three or more members, four times the practice-bundle share, and no other figure crosses outcomes against each other.