rer.linkchecker

A Plone addon that checks every internal and external link found in site contents (fields and Volto blocks) and generates a report of the broken ones.

Features

• Adds a portal_linkchecker tool that crawls the whole site, collecting every link found in content fields and in Volto blocks (resolveuid/... links included).
• Checks internal links by resolving them against the catalog, and external links concurrently over HTTP (configurable timeout, thread pool size, and per-host throttling).
• Caches external link statuses for a configurable TTL, so repeated runs only re-check links that are due.
• Distinguishes real broken links from bot-protection responses (403, 429, LinkedIn's 999) and from http:// links that only work over https:// (reported so they can be fixed in place, not counted as broken).
• Reports the conditions that are not a plain http status with their own negative status, so they can be told apart in the CSV: -1 timeout, -2 works only over https (update the link), -3 connection error.
• Does not verify TLS certificates: only reachability matters here, and many otherwise working servers omit their intermediate certificate (browsers fetch it themselves, requests does not), which would be reported as a broken link.
• Exposes the results as a CSV report (PAGE, LINK, TYPE, STATUS, DESCRIPTION) via tool.get_rows().
• Ships a check_broken_links console script to run a check from the command line or from cron, without going through the web.

Installation

Install rer.linkchecker with uv:

uv add rer.linkchecker

or add it to a zc.buildout-based project as a develop egg / source, alongside plone.volto (required: it provides the blocks/blocks_layout fields the linkchecker scans, and plone.distribution used by addPloneSite).

Then add rer.linkchecker to the eggs of your Plone instance and install the add-on from the Plone control panel (or via a GenericSetup profile) as usual.

Usage

The portal_linkchecker tool

from plone import api

tool = api.portal.get_tool("portal_linkchecker")
tool.check_site()  # crawl the whole site and check every link

for uid, broken_links in tool.get_page_with_broken_links():
    ...  # [(link, status), ...] per content UID

for item in tool.get_broken_links():
    ...  # flat view: one dict per broken link, with the page it sits on

import csv

with open("broken_links.csv", "w", newline="") as fh:
    writer = csv.writer(fh, quoting=csv.QUOTE_ALL)
    for row in tool.get_rows():
        writer.writerow(row)

get_broken_links() is the flat source every report is built from, and get_rows() is its csv rendering. Both include the bot-protected links (each with its own description), while get_page_with_broken_links() leaves them out: it answers "how many real problems are there". Filter either with tool.filter_links(items, status=[404], link_type="EXTERNAL").

check_site(ttl=3600 * 6, timeout=15, max_workers=10) accepts:

• ttl: seconds a cached external link status stays valid (0 forces a full recheck).
• timeout: per-request timeout, in seconds, for external links.
• max_workers: number of concurrent threads checking external links.

The check_broken_links console script

Installed as a standard console_scripts entry point ([project.scripts] in pyproject.toml), so it lands in bin/ both in a uv-managed virtualenv and in a zc.buildout instance. Since it runs Zope/ZODB code, it must be launched through zconsole/instance run, not called directly:

# uv-managed instance (Makefile: make check-broken-links)
./bin/zconsole run instance/etc/zope.conf ./.venv/bin/check_broken_links

# zc.buildout instance
./bin/instance run bin/check_broken_links

Options:

• --ttl, --workers, --timeout: same meaning as on check_site().
• --output-dir: also dump the run as a csv in this directory (default: no csv).
• --site-id: id of the Plone site to check (default: the PLONE_SITE_ID env var, or Plone).
• --url <url>: verify a single url and log its status, without touching the site.
• --content <path-or-UID>: verify a single content's links and log them, without touching the site.

The script stores the report in the site, which is where the REST API endpoints below read it from: that is the normal way to get at the result. With --output-dir it also writes <output-dir>/<siteid>_broken_links_<YYYYMMDD-HHMMSS>.csv, byte for byte the csv @linkchecker-csv serves — worth it only to keep an archive of past runs, since the site holds the last one alone.

The REST API endpoints

Two read-only endpoints on the site root expose the stored report, both guarded by the rer.linkchecker.ViewReport permission (granted to Manager, Site Administrator and Editor).

They never run a check: a full check takes minutes on a medium site and would time the request out, so they only serve what the last run stored. Refresh the data out of band, with the check_broken_links script from cron. Because the data is therefore asynchronous, both responses carry the timestamp it dates from.

# the report as json, batched (plone.restapi conventions: b_start, b_size)
curl -u user:pass "$SITE/++api++/@linkchecker" -H 'Accept: application/json'

# the same report as a csv download
curl -u user:pass -OJ "$SITE/++api++/@linkchecker-csv"

Both accept the same filters:

• status: repeatable, e.g. ?status=404&status=-2 (negative values are the STATUS_* constants: -1 timeout, -2 https-only, -3 connection error).
• type: INTERNAL or EXTERNAL.

An invalid filter value answers 400 rather than silently returning an empty report.

@linkchecker returns last_update and duration for the whole run, items_total, the batched items, and a summary listing {status, status_description, count} sorted by count. The summary is computed over the unfiltered report on purpose, so the counts a UI shows in its filter chips do not move as filters are applied. When no check has ever run, last_update is null and items/summary are empty, which a UI can tell apart from "nothing is broken".

Each item describes the page it sits on with the usual plone.restapi field names, so a client can treat it as any other content reference, and names the link's own fields apart from those:

{
  "@id": "http://site/bandi-e-avvisi",
  "@type": "Document",
  "title": "Bandi e avvisi",
  "UID": "1a568f09734340dfba2c1a53730b9cf6",
  "link": "https://unimc.it/careerday",
  "link_type": "EXTERNAL",
  "status": 404,
  "status_description": "Not Found",
  "last_update": "2026-07-30T03:00:12"
}

@linkchecker-csv returns the exact same csv the console script writes (same columns, same quoting), names the file after the date of the data rather than of the download, and repeats that date in the X-Linkchecker-Last-Update response header.

Development

Prerequisites

• An operating system that runs all the requirements mentioned.
• uv
• Make
• Git
• Docker (optional)

Setup

git clone git@github.com:RegioneER/rer-linkchecker.git
cd rer-linkchecker/backend
make install

Common tasks

make start                  # start a Plone instance on localhost:8080
make create-site            # create a new site from scratch
make check-broken-links     # run the linkchecker and write a csv report
make test                   # run the test suite

Contribute

• Issue tracker
• Source code

License

The project is licensed under GPLv2.

Credits

Developed with the support of Regione Emilia Romagna.

Regione Emilia Romagna supports the PloneGov initiative.

Authors

This product was developed by RedTurtle Technology team.