Research Engine Academic Journal Plugin

marginalia-ai-plugin-academic-journal adds scholarly literature to a Research Engine corpus. It discovers papers, resolves legal open-access copies, downloads and ingests them, extracts their bibliographies, and records cites edges between ingested papers — snowballing through references to a bounded depth.

It contributes:

Providers

Metadata calls go only to the hosts in plugin.yaml's network_allowlist; the plugin refuses any other API host. Open-access PDFs are then downloaded from wherever the metadata points — publisher sites and institutional repositories — which is why the manifest declares network: full.

Every provider is rate-limited per process with a token bucket, and a circuit breaker stops calling a provider after five consecutive failures for 60 seconds:

HTTP 429 responses are retried with the server's Retry-After (capped at 60 seconds).

Install

The plugin depends only on marginalia-ai-sdk; install it into the same environment as marginalia-ai 0.6.2 or newer.

python -m pip install marginalia-ai marginalia-ai-plugin-academic-journal

With pipx:

pipx install marginalia-ai
pipx inject marginalia-ai marginalia-ai-plugin-academic-journal

Installation only makes the static manifest discoverable. Core imports nothing until you approve the exact distribution:

research-engine plugin list                          # academic-journal: available
research-engine plugin audit academic-journal        # contributions, permissions, hash, database
research-engine plugin enable academic-journal
research-engine plugin migrate academic-journal      # creates/upgrades the acad_* tables

Restart the MCP server afterwards. Core refuses to load the plugin until migrate has recorded revision 2. Upgrading the package changes its version and manifest hash, so it needs research-engine plugin approve-upgrade academic-journal (and migrate again when a release adds a revision).

Configuration

Core 0.6.2 supplies the configured corpus database URL through the plugin context; this includes values loaded from core's .env. The plugin never searches for .env files itself. Standalone scripts may still supply RE_DB_URL explicitly.

Provider keys and contact emails are listed under Providers. They are never written to the database: request parameters and error messages are redacted before the API call log and job errors store them.

Data and storage

Mutable state lives under the data directory core assigns, ~/.research-engine/plugin-data/academic-journal/ by default:

plugin-data/academic-journal/
  papers/<paper-id>.pdf     # acquired and manually imported PDFs

Nothing is written to the package directory or the working directory. Papers acquired by 0.1.x under ~/.research-engine/papers/ keep their recorded paths and are not moved.

Database

The plugin owns these tables in the corpus database, outside core's migration chain:

Only research-engine plugin migrate academic-journal changes the schema. It holds an advisory lock, applies each revision in its own transaction, and refuses — without touching anything — when an applied file's checksum has changed or the database was migrated by a newer release. There is no downgrade. A database created by 0.1.x migrates in place with every row kept.

Uninstalling the distribution or running plugin forget leaves these tables, the ingested documents, and their cites edges intact.

Pipeline and workers

discovered → resolved → acquired → ingested → citations_extracted → complete
                ↘ unresolvable

Discovery tools insert papers and queue a job; workers do the rest. Workers are asyncio tasks started by academic-journal.start_workers inside the MCP server process:

• Ownership: the server process. No subprocess or thread is created, so nothing outlives it.
• Shutdown: academic-journal.stop_workers, or the server exiting.
• Concurrency: one worker per stage per process. Several servers can share a database — claims use FOR UPDATE SKIP LOCKED and enqueueing is serialised per paper and stage.
• Restart: start_workers is idempotent and restarts only stopped stages. A job interrupted mid-run is reclaimed once its lock is older than ACAD_JOB_LEASE_SECONDS. A worker that loses the database backs off and keeps polling; pipeline_status shows the last error.

Failed jobs retry with exponential backoff up to five attempts; academic-journal.retry_failed re-queues them. Citation extraction uses core's extraction service (and therefore core's LLM configuration) and snowballs through cited DOIs to depth 2 within the snowball budget.

Manual PDFs and licensing

The plugin only fetches copies that providers report as open access. For anything else, download the PDF yourself through access you are entitled to and import it:

academic-journal.import_manual_pdf(file_path="/absolute/path/paper.pdf", doi="10.…")

The file is copied into the plugin data directory and queued for ingestion. You are responsible for having the right to store and process it; the plugin does not check licences or redistribute anything.

Compatibility

The public API is the SDK; the plugin imports nothing from research_engine.

Known limitations

• Core 0.6.0 registers post-ingestion hooks but does not call them. The ingestion stage links each document to its paper directly, so the pipeline does not depend on the hook.
• DOIs from different providers are matched exactly as stored, so a DOI that differs only in letter case can produce two paper rows.

Privacy and security

Enabled plugins run in the core process with its privileges; enable only distributions you trust, and review plugin audit before approving. Queries, DOIs, and your contact email (when configured) are sent to the providers above under their terms. Downloaded PDFs and paper metadata stay in your data directory and database.

Development

uv sync --extra dev
uv run ruff check acad tests
uv run pytest tests/unit tests/contract -q

Integration tests need the exact core artifact and a disposable PostgreSQL. With Docker available they start one; otherwise set ACAD_TEST_DATABASE_URL to a throwaway database whose name contains test. They never use RE_DB_URL, so they cannot reach your research corpus.

uv sync --extra dev --group integration
uv run pytest tests/integration -q

Provider tests that call the real APIs are opt-in: ACAD_LIVE_TESTS=1 uv run pytest tests/live.

scripts/release_smoke.py exercises a built wheel end to end — enable, migrate, load, fixture-backed discovery, acquisition, ingestion, and citation edges — against a disposable database, with the provider APIs and the embedding server stubbed locally:

RE_DB_URL=postgresql+asyncpg://user:pass@host:5432/acad_smoke_test \
    /path/to/clean-venv/bin/python scripts/release_smoke.py

Support

Report problems at issues. See the changelog and, for maintainers, RELEASING.md. Licensed under Apache-2.0.