invenio-archive-it

Links to archived versions of InvenioRDM records in Archive-It.

Archiving is set up per community. A community names the Archive-It collection that archives its website, for example collection 22103 for https://jabberwocky.weecology.org/. A job run from the admin UI looks up that website in the collection and stores on each of the community's records which of its URLs were captured, and when. The record's landing page sidebar lists them under External resources → Archived in, the same way Zenodo shows software archived in Software Heritage.

Requires InvenioRDM v14 or later.

How it works

1. A community admin sets the community's Archive-It collection (ia:collection). The website to look up is the community's own website (metadata.website). Example: collection ID 22103 for https://jabberwocky.weecology.org/.
2. An admin runs the Update Archive-It captures job for one community, or for every community with a collection. Each community gets one Celery task, rate-limited to 6 per minute (as Archive-It's Wayback service is shared across all users). A lookup Archive-It rate-limits is tried again later, up to three times, and the run waits for it.
3. The task sends one prefix query to https://wayback.archive-it.org/{collection}/timemap/cdx for the website, which returns every capture under it. It then matches the captures to the URLs of the records whose default community this is. Those are identifiers and related identifiers with scheme url; Archive-It playback URLs are first reduced to the page they show.
4. Each record's matched captures (first and last capture per URL) are stored in the record custom field ia:captures. Only records whose captures changed are written.
5. The sidebar shows one Archive-It entry per captured URL, with the dates it was archived beneath it, written for the reader's language (Archived September 13, 2026). The entry links to the Wayback calendar, e.g. https://wayback.archive-it.org/22103/*/https://jabberwocky.weecology.org/….

If the lookup fails or Archive-It refuses it, every record keeps its previous captures.

A job set to notify by mail says which community a run was for: a run that names one community shows it, and a run over every community lists those whose lookup did not finish. This overrides invenio-jobs' run_notification templates, because the mail is built from the job's title and the community cannot be passed in. Jobs whose runs carry no community are unaffected, and an instance's own templates still override these. The subject line stays invenio-jobs' own (Job Failed: Update Archive-It captures); it is built in Python, out of a template's reach.

Installation

uv add git+https://codeberg.org/front-matter/invenio-archive-it

Add the custom fields and the sidebar link to invenio.cfg. Merge them with any namespaces, custom fields and external links you already have:

from invenio_archive_it.custom_fields import (
    ARCHIVE_IT_COMMUNITY_CUSTOM_FIELDS,
    ARCHIVE_IT_COMMUNITY_CUSTOM_FIELDS_UI,
    ARCHIVE_IT_CUSTOM_FIELDS,
    ARCHIVE_IT_NAMESPACE,
)
from invenio_archive_it.links import archived_versions_render

RDM_NAMESPACES = {**ARCHIVE_IT_NAMESPACE}
RDM_CUSTOM_FIELDS = [*ARCHIVE_IT_CUSTOM_FIELDS]
COMMUNITIES_CUSTOM_FIELDS = [*ARCHIVE_IT_COMMUNITY_CUSTOM_FIELDS]
COMMUNITIES_CUSTOM_FIELDS_UI = [ARCHIVE_IT_COMMUNITY_CUSTOM_FIELDS_UI]
APP_RDM_RECORD_LANDING_PAGE_EXTERNAL_LINKS = [
    {"id": "archive_it", "render": archived_versions_render},
]

COMMUNITIES_NAMESPACES defaults to RDM_NAMESPACES, so ia covers both. To add the collection field to an existing community settings section instead of its own "Web archiving" section, add its entry from ARCHIVE_IT_COMMUNITY_CUSTOM_FIELDS_UI["fields"] to that section.

Create the search mappings:

invenio rdm-records custom-fields init -f ia:captures
invenio communities custom-fields init -f ia:collection

Copy the sidebar icon into the instance's static folder with invenio collect (invenio-cli assets build runs it too).

To run lookups, open Update Archive-It captures under Administration → Jobs and start a run. Pick a community, or leave the field empty for every community with a collection. The module adds no Celery beat schedule.

Configuration

Important: Misconfiguring ARCHIVE_IT_WAYBACK_URL will cause sidebar links to fail silently. The module validates the URL on startup; check the logs if configured endpoints are unreachable.

Troubleshooting

• Lookups fail silently: Archives at Archive-It's Wayback service are behind bot protection. Check logs for error=blocked; verify your server can reach the CDX API. Failed lookups preserve existing captures; no data is lost.
• Lookup errors in logs: Monitor warnings with error=http_error, error=network, or error=invalid_response to catch misconfigurations or API changes.
• Rate limited: Archive-It answers a caller it limits with a page titled "Rate limit reached" and a 200, logged as error=rate_limited, with authenticated= saying whether a key was sent. The lookup is tried again later (ARCHIVE_IT_RATE_LIMIT_*); the warning Archive-It still rate-limits the lookup; not trying again means the retries ran out. Ask Archive-It for a CDX key and set ARCHIVE_IT_CDX_AUTH_TOKEN. If authenticated=True and the limit remains, the key does not lift it.
• Answer cut short: a lookup asks only for pages and their revisit records, but a site can still have more than Archive-It's 50,000-capture maximum. The warning Archive-It answered only part of the collection then names the last URL reached; records past it keep the captures they had rather than losing them.
• Sidebar links appear broken: Ensure ARCHIVE_IT_WAYBACK_URL is correct. The module validates it on startup; misconfiguration will be logged at warning level.
• Specific community not found: Only records whose default community matches are updated. Verify the community is set correctly on records.
• Captures are written to the published record directly. This doesn't create a new version or a draft. An open draft of the record gets the same captures, so publishing it doesn't bring back old ones.
• Anything else that updates records must keep ia:captures. An update through the records service replaces all custom fields with what it sends.
• Anyone who can edit a community's settings can set its collection. To make it admin-only, restrict the field in the instance.

Development

uv sync --all-extras
uv run pytest
uv run ruff check .
uv run black --check .

Translations

The sidebar text and the custom field labels are translated into English, German, Spanish and French. Catalogs live in invenio_archive_it/translations/<language>/LC_MESSAGES/messages.po; English is the source language, so its messages stay untranslated. Dates are not in the catalogs — Babel formats them for the reader's locale (Archived September 13, 2026, Archiviert am 13. September 2026).

After changing or adding a translatable string:

uv run pybabel extract -F invenio_archive_it/translations/babel.ini \
  -k _ -k gettext -k lazy_gettext \
  -o invenio_archive_it/translations/messages.pot .
uv run pybabel update -i invenio_archive_it/translations/messages.pot \
  -d invenio_archive_it/translations

Then fill in the new messages and run uv run pytest tests/test_translations.py, which fails on a message left untranslated or marked fuzzy. To add a language, pybabel init -l <language> and add it to LOCALES in that test.

The .mo files gettext reads are committed next to the .po files, so an instance installed from a checkout has the translations without a build step. Recompile them after editing a catalog:

uv run pybabel compile -d invenio_archive_it/translations

Building the package recompiles them as well (hatch_build.py), so a released wheel cannot carry a catalog that is out of date with its .po file.

License

MIT