Notanda — Open-Access API Harvester

Notanda is the public product name and PyPI distribution name. The repository, Python import package and CLI keep their established technical names Open-Access-API-Harvester, harvester and harvester.

This repository publishes the reviewed 1.3.2 product-core source. It is not a hosted service and the recommended accompanied beta path remains beta@notanda.io.

A public Installer Preview is available under GitHub Releases solely as a technical pre-release: https://github.com/RudiRegenwurm/Open-Access-API-Harvester/releases/tag/v1.3.0-installer-preview.1 Those installers are unsigned on Windows/Linux and ad-hoc signed on macOS, may trigger SmartScreen/Gatekeeper warnings, and are not the recommended installation path for non-technical users.

A headless, resumable, idempotent CLI pipeline that discovers Open-Access scholarly works through OpenAlex Topics, cross-checks them against Europe PMC, falls back to Unpaywall for OA locations, validates every downloaded artifact, and stores the result as a deterministic flat corpus ready for a downstream pre-ingestion pipeline. Schema 4 additionally retains provider observations, merge decisions and acquisitions in an append-only Evidence Ledger while the established read model remains compatible.

DISCOVERY → NORMALIZATION → DEDUPLICATION → ACQUISITION → VALIDATION → INGESTION → PROVENANCE/REPORTING

It is not a web application, document-management system, search engine, OCR system or LLM application. It is a document harvesting pipeline with persistent state.

Quick start

With Python 3.10 or newer, install the PyPI distribution in a fresh virtual environment:

python -m venv .venv
# Linux/macOS:
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1
python -m pip install notanda
harvester --version

For an isolated application install with pipx:

pipx install notanda

For development from a repository checkout:

python3 -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"

With the local control center (no command line needed after this)

harvester serve

Opens a local web UI at http://localhost:8765/ for harvesting, browsing the corpus, inspecting runs and provenance, verifying integrity, retrying failures and editing settings. Credentials can be entered in its Settings page on first run.

New Harvest offers two search modes. Conventional Search uses the query you type, exactly as typed. Assisted Search turns a plain-language research question into one compact retrieval query you can edit, previews up to ten real discovery results without downloading anything, and only then hands the approved query to the same harvest pipeline. Assisted Search needs a query-advisor credential; Conventional Search does not and is unaffected when the advisor is missing or unavailable.

Experimental local MCP server

The repository also contains an optional local MCP component under mcp/. It exposes two deliberately narrow tools — search_literature and get_evidence — and persists the request, provider observation, exact returned result and SHA-256 manifest under one stable evidence ID. It performs no full-text download or corpus write and opens no network server.

The separate notanda-mcp Beta package uses the reviewed notanda==1.3.1 PyPI release as its core. With Python 3.11 or 3.12, install it using python -m pip install "notanda-mcp==0.1.0b2". The MCP Registry entry describes the package, local stdio transport, required OpenAlex API key and writable evidence directory. The package is not included in the notanda 1.3.2 distribution. The technical live search and restart/receipt checks passed on 2026-10-02; Thomas's external unaccompanied test is still pending. Client configuration, evidence format and verification steps are in mcp/README.md.

From the command line

export HARVESTER_OPENALEX_API_KEY='…'          # free: openalex.org/settings/api
export HARVESTER_CONTACT_EMAIL='ops@example.org'  # required by Unpaywall
export HARVESTER_ADVISOR_API_KEY='…'           # optional: Assisted Search in the UI only

harvester discover --topic-id T10159 --limit 25    # dry run, downloads nothing
harvester harvest  --topic-id T10159 --limit 25    # real harvest
harvester verify --deep                            # check the corpus against state

Both drive the same engine and the same state database: a run started in the browser can be resumed from the terminal.

Output

The downstream contract is a flat directory of deterministic sibling files:

<storage_root>/
├── doi_10_1371_journal_pone_0123456_5d41402abc4b.pdf     primary full text
├── doi_10_1371_journal_pone_0123456_5d41402abc4b.xml     when legitimately available
└── doi_10_1371_journal_pone_0123456_5d41402abc4b.json    mandatory sidecar

The sidecar keeps bibliographic metadata, artifact metadata, provenance and harvest state in separate blocks. Missing information is null; nothing is ever invented.

{
  "document_id": "doi_10_1371_journal_pone_0123456_5d41402abc4b",
  "doi": "10.1371/journal.pone.0123456",
  "title": "…",
  "abstract": "…",
  "oa_status": "gold",
  "oa_status_source": "openalex",
  "domain_tags": ["Social Sciences", "Psychology", "…"],
  "artifacts": { "pdf": { "sha256": "…", "size_bytes": 123456, "…": "…" } },
  "provenance": {
    "discovered_via": ["openalex"],
    "cross_checked_via": ["europe_pmc"],
    "acquired_via": "unpaywall",
    "resolved_url": "…",
    "http_status": 200
  },
  "harvest": { "run_id": "…", "status": "COMPLETED", "attempts": 1 }
}

What it guarantees

Commands

The evidence bundle contains Ledger records and file references, not the PDF/XML artifacts or the complete corpus directory. A portable corpus export with files is still in development.

Exit codes: 0 success · 1 failure · 2 configuration error · 3 completed with document failures · 4 suspended (provider budget) · 130 interrupted.

Tests

pytest            # offline suite — no network, no credentials required
pytest -m live    # controlled real-provider smoke tests

The offline suite runs against deterministic in-process mock providers covering normal responses, pagination, throttling, budget exhaustion, timeouts, malformed payloads, missing full text, alternative locations and duplicates.

Documentation

Code signing policy

Windows installers are currently unsigned.

Dependencies

httpx (streaming HTTP with a pluggable transport), pypdf (PDF structural validation), defusedxml (safe XML parsing). Everything else is the standard library: configuration, SQLite persistence, CLI, logging, hashing, concurrency, retry — and the web UI, which uses http.server plus a front end with no build step, so harvester serve is the only command needed to run it.

License

Unless a file or notice says otherwise, the product source code and documentation in this repository are licensed under the MIT License, copyright 2026 Rudolf Kiechle.

The bundled IBM Plex and Source Serif 4 font files are not MIT-licensed; they remain under the SIL Open Font License 1.1 included beside the files. External Python dependencies are not vendored and retain their own licenses. The Notanda name, emblem, wordmark, lockup, favicons and avatar are excluded from the MIT grant. Details and exact file scopes are in THIRD_PARTY_NOTICES.md and TRADEMARKS.md.