invenio-pidbox

InvenioRDM integration that serves people, organizations and citations from a commonmeta SQLite store, read through commonmeta-py and commonmeta-rs.

The store is the one a commonmeta import already maintains — ORCID persons, ROR organizations, works and the relations between them. This package reads it, and adds no tables of its own: no application-specific people or organizations tables, no citation store, no schema. Everything it exposes is a view onto that one file.

Features

Profiles — people and organization profile pages for InvenioRDM, with list and search pages, template context injected on the profile routes, and a stylesheet and search-result components shipped as webpack bundles.

Deposit autocomplete — creator, affiliation and funder suggestions served from commonmeta-rs Tantivy search, with the deposit form's overridable components registered to route lookups to pidbox when configured.

Record relations — affiliations and funders resolved from the store as well as the vocabularies, so a record may name any organization the store knows. Opt-in, because it changes which records an instance accepts.

Facet labels — search facets fall back to the store for ids the vocabulary cannot label, so an organization accepted by the relations above does not appear in the sidebar as a bare ROR id.

Citations — citation counts and citing works, read from the store's pid_relations table and served as commonmeta works, plus a pidbox:citations custom field for the works a record itself cites.

Bounded store access — every read runs with a deadline behind a circuit breaker, so a store that has gone slow degrades the feature that reads it instead of holding request threads.

Requirements

• Python 3.14+
• A commonmeta SQLite database readable through commonmeta-rs

Internally, commonmeta-rs reads pid_annotations records with source_id 4 for people (ORCID) and source_id 3 for organizations (ROR), works from pid_records, and citations from pid_relations.

Installation

uv add invenio-pidbox

This package depends on commonmeta-py, commonmeta-rs, and invenio-app-rdm at runtime, so installing invenio-pidbox pulls in the profile backend automatically.

Configuration

Required

• PIDBOX_SQLITE_PATH — path to the commonmeta SQLite store, used by every feature below. Defaults to /var/lib/commonmeta/commonmeta.sqlite3.

Deposit autocomplete

• APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES
• APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES_PROVIDER
• PIDBOX_AUTOCOMPLETE_API

To enable pidbox autocomplete in the deposit form, set:

APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES = "search"
APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES_PROVIDER = "pidbox"

The endpoint URLs can be overridden with:

PIDBOX_AUTOCOMPLETE_API = {
    "names": "/api/pidbox/names",
    "affiliations": "/api/pidbox/affiliations",
    "funders": "/api/pidbox/funders",
}

The funders endpoint returns only organizations with "funder" in their types list, so the deposit form offers exactly the set a commit will accept.

Record relations

PIDBOX_RECORD_RELATIONS = True

Off by default. InvenioRDM validates every affiliation and funder id on each record commit against the affiliations and funders vocabularies, so a record naming an organization that was never imported into them is rejected with InvalidRelationValue. With this on, the store is consulted first and the vocabulary second, making the accepted set the union of the two rather than a swap — an unreadable store degrades to exactly the stock behaviour.

Turning this on changes which records an instance will accept, which is why installing the package does not.

Facet labels

Facets label their buckets from the vocabularies, so an organization accepted by the relations above but absent from the vocabulary shows as a raw ROR id. The store-backed labels fill only those gaps, asking the vocabulary first so a curated, localized title keeps winning:

from invenio_pidbox.facets import StoreAffiliationsLabels, StoreFundersLabels

Use them where the facet is declared, in place of the stock AffiliationsLabels / FundersLabels — for example value_labels=StoreFundersLabels("funders").

Store access

• PIDBOX_STORE_TIMEOUT — seconds a single store read may take before the caller gives up (default 2.0; 0 disables the deadline)
• PIDBOX_STORE_FAILURE_THRESHOLD — consecutive timeouts before the store is treated as unavailable (default 3)
• PIDBOX_STORE_RECOVERY_TIME — seconds to leave an unavailable store alone before trying again (default 60.0)

These exist because a store shared with a writer can get very slow: reads that normally take a millisecond have blocked for minutes while an import ran against the same file, and since a relation is resolved on every record commit, that was enough to exhaust the request threads and have gunicorn kill the worker. A slow store is therefore treated as an unavailable one.

Routes

Each API route is also registered without the /api prefix, because the same extension is registered on the UI app and the API app, and Invenio mounts the API app at /api and strips the prefix.

Autocomplete requests carrying ?suggest= are capped at 50 results; the profile search pages are fixed to 10 results per page and 100 in total.

Citations

GET /api/pidbox/citations/<doi>

<doi> is a bare DOI or a DOI URL. The response uses the usual hits envelope, and each hit is a commonmeta work, unchanged:

{"hits": {"hits": [{"id": "https://doi.org/10.1234/a", "title": "Work a"}], "total": 3}}

Which relations count as a citation (References, Cites, IsSupplementedBy — DataCite's definition) is decided by commonmeta-rs, not by this package.

total is the citation count and can exceed the number of hits: it is read from the relation index, while hits are hydrated works, so a citing DOI whose work has not been imported yet is counted but cannot be listed.

Concept DOIs are also left out of the hits. A versioned publisher — Rogue Scholar, Zenodo — registers a version DOI and a parent DOI standing over it, and both cite, so the same work would otherwise appear twice under two identifiers with one citation text between them. The parent is the one carrying a HasVersion relation, and it is dropped in favour of the specific version that did the citing. The count knows nothing about versions, so such a citation counts two and lists one.

Listing hydrates every citing work, so size=0 returns just the count, which is answered from an index and stays cheap on a heavily cited DOI. Otherwise size (default 10, max 100) and page paginate the response.

The pidbox:citations custom field

Records can also carry the works they cite, as a list of {identifier, scheme, reference} entries indexed with the record:

{
  "identifier": "10.59350/4q8j1-1ap35",
  "scheme": "doi",
  "reference": "Willighagen, E. (2007, May 25). Numbers are copyrighted?. <i>Chem-bla-ics</i>."
}

reference is the citing work rendered as a formatted citation, so a landing page can show a reference list rather than a column of DOIs. It uses the instance's own RDM_CITATION_STYLES_DEFAULT (apa unless changed), so a stored reference reads the way the citation box on the same page does. RDM has no config for the locale — it renders in the requesting user's language, which a background sweep does not have — so PIDBOX_CITATIONS_LOCALE supplies it, defaulting to the en-US RDM itself falls back to.

This is the record-side view of a citation, and it is independent of the store-side one above — the field says what a record cites, the store says what cites a DOI. They point in opposite directions, so their numbers are not expected to match.

The field is not registered automatically, because a custom field has to exist in the search mapping before a record can use it. To enable it:

from invenio_pidbox.custom_fields import (
    CITATIONS_FACET,
    CITATIONS_QUERY_FIELD,
    CitationsCF,
    PIDBOX_NAMESPACE,
)

RDM_NAMESPACES = {**RDM_NAMESPACES, **PIDBOX_NAMESPACE}
RDM_CUSTOM_FIELDS = [*RDM_CUSTOM_FIELDS, CitationsCF(name="pidbox:citations")]
RDM_FACETS = {**RDM_FACETS, "citations": CITATIONS_FACET}

Then create the mapping:

invenio rdm-records custom-fields init -f pidbox:citations

CITATIONS_QUERY_FIELD is a QueryParser mapping entry, so adding "citations": CITATIONS_QUERY_FIELD to RDM_SEARCH's parser mapping makes citations:10.5555/12345678 searchable. Declaring CITATIONS_FACET does not display it — the facet also has to be named in RDM_SEARCH's facets list.

commonmeta-rs fills the field when it pushes a record: citing works become IsReferencedBy relations, the InvenioRDM serializer maps those to custom_fields["pidbox:citations"], and commonmeta push/put --to inveniordm sends them to the REST API. Reading records back reverses it. So the store stays the source of truth and the field is a projection of it onto each record.

That name is hardcoded in commonmeta-rs, which is why this package requires commonmeta-rs >= 0.9.80. Earlier versions write rs:citations, and against them the field is defined, indexed, and never populated.

This field comes from Rogue Scholar, where it was rs:citations. It is unchanged apart from the namespace, and the two names are different fields: an instance holding records under the old one has to migrate and reindex them, and until it does, those records read as having no citations.

Keeping the field current

A push writes the citations a record had at that moment. What makes it stale afterwards is other records arriving, which is not an event on this record, so nothing in InvenioRDM would ever notice.

The package therefore ships a sweep, as a job in the Jobs administration dashboard:

Refresh citations   (pidbox_refresh_citations)

Installing the package makes the job available there; it does not schedule it. Whether it runs at all, how often, and with what arguments is set in the dashboard — this one edits records, which is not something installing a package should start doing on a timer. Running it needs the Celery worker a standard deployment already has.

For every record with a DOI the sweep compares count_sqlite_citations — an index lookup — against the number of citations the record already stores, and skips the record without hydrating anything when they agree. Only a mismatch pays for the full read, and only a genuine difference in identifiers is written back, as a metadata edit through the records service (same version, same DOI, new revision).

For a first run, set the dashboard's custom arguments to bound it and check the counts it reports:

{"max_records": 100}

It returns {"scanned", "hydrated", "updated", "failed"}. The same task can be called directly, for example from invenio shell:

from invenio_pidbox.tasks import refresh_citations

refresh_citations(max_records=100)

Rendering a citation costs about 25 ms, so a reference already stored for a DOI is kept rather than rendered again: a run only formats citations the record did not already have. Records written by commonmeta push carry identifiers with no reference text, and the sweep backfills those.

Four behaviours worth knowing:

• The dashboard's since is ignored. A record's citations change when other records are imported, which does not modify the record, so no timestamp on it reflects the change — filtering by one would skip exactly the records that need updating. Being cheap on unchanged records is what replaces it.
• A record cited by a work the store has not imported has a count higher than anything that can be listed, so it takes the slow path on every run and writes nothing. Correct, but not free. A citing work with a concept DOI (below) does the same.
• The sweep never clears a record's citations. Every store read degrades to "none" when the store is unavailable, which is indistinguishable from a DOI nothing cites, so a record that holds citations keeps them rather than risk emptying every record in the instance.

How It Works

When a request path matches /people/<orcid> or /organizations/<ror>, the extension loads the corresponding commonmeta record from SQLite and exposes template-friendly objects such as person, organization, employment, identifiers, relations, location, orcid, ror, and search_config. It also registers the profile routes and a country_name template filter, so it runs against a standard invenio-app-rdm installation.

For deposit autocomplete it patches invenio-app-rdm's form config at runtime to inject the endpoint URLs, and loads overridable components that switch the person, organization and funder remote selects to those endpoints. Awards lookup stays on the existing awards API.

Reads are optional throughout, and each has a defined fallback: a profile falls back to a record search, an autocomplete to no suggestions, a relation to the vocabulary, a facet label to the bare id, a citation listing to none. An absent, unreadable or slow store therefore costs a feature, not the page.

Development

uv sync --extra tests
uv run pytest

Entry points

License

MIT