rer.linkchecker

A Plone addon that checks every internal and external link found in site contents (fields and Volto blocks) and generates a report of the broken ones.

Features

• Adds a portal_linkchecker tool that crawls the whole site, collecting every link found in content fields and in Volto blocks (resolveuid/... links included).
• Checks internal links by resolving them against the catalog, and external links concurrently over HTTP (configurable connect and read timeouts, thread pool size, and per-host throttling).
• Gives up on a host after a few consecutive connection errors: its remaining links get the same outcome without being fetched, so a dead host does not hold the run up for minutes.
• Reports the links to a blacklisted domain (one contents must never link to, e.g. the editorial backend of the sites) without fetching them: see Blacklisted domains.
• Caches external link statuses for a configurable TTL, so repeated runs only re-check links that are due.
• Distinguishes real broken links from bot-protection responses (403, 429, LinkedIn's 999: the link usually works for a human, so it is not counted as broken) and from http:// links that only work over https:// (reported apart, so the url can be updated in place).
• Reports the conditions that are not a plain http status with their own negative status, so they can be told apart in the CSV: -1 timeout, -2 works only over https (update the link), -3 connection error, -4 blacklisted domain.
• Does not verify TLS certificates: only reachability matters here, and many otherwise working servers omit their intermediate certificate (browsers fetch it themselves, requests does not), which would be reported as a broken link.
• Exposes the results as a CSV report via tool.get_rows(), with the same columns and labels as the report panel: site content, link to check, link type, outcome (the status code), outcome description and the action to take (to fix, to update, to check), translated in the language of the request.
• Ships a check_broken_links console script to run a check from the command line or from cron, without going through the web.

Installation

Install rer.linkchecker with uv:

uv add rer.linkchecker

or add it to a zc.buildout-based project as a develop egg / source, alongside plone.volto (required: it provides the blocks/blocks_layout fields the linkchecker scans, and plone.distribution used by addPloneSite).

Then add rer.linkchecker to the eggs of your Plone instance and install the add-on from the Plone control panel (or via a GenericSetup profile) as usual.

Usage

The portal_linkchecker tool

from plone import api

tool = api.portal.get_tool("portal_linkchecker")
tool.check_site()  # crawl the whole site and check every link

for uid, broken_links in tool.get_page_with_broken_links():
    ...  # [(link, status), ...] per content UID

for item in tool.get_broken_links():
    ...  # flat view: one dict per broken link, with the page it sits on

import csv

with open("broken_links.csv", "w", newline="") as fh:
    writer = csv.writer(fh, quoting=csv.QUOTE_ALL)
    for row in tool.get_rows():
        writer.writerow(row)

get_broken_links() is the flat source every report is built from, and get_rows() is its csv rendering. Both include the bot-protected links (each with its own description), while get_page_with_broken_links() leaves them out: it answers "how many real problems are there". Filter either with tool.filter_links(items, status=[404], link_type="EXTERNAL").

check_site(ttl=3600 * 6, timeout=15, max_workers=10, connect_timeout=5, host_max_failures=3) accepts:

• ttl: seconds a cached external link status stays valid (0 forces a full recheck).
• timeout: per-request timeout, in seconds, for an external link to answer once connected.
• max_workers: number of concurrent threads checking external links.
• connect_timeout: seconds allowed to open the connection, never more than timeout. A host that does not accept the connection in time is a connection error (-3), not a timeout: raising the timeout would not bring it back.
• host_max_failures: consecutive connection errors after which a host is given up on for the run (0 never gives up). Any other outcome, an http error or a timeout included, proves the host accepts connections and resets the count: a timeout is about a slow page, and the next page of the same host may well answer. The links left unchecked get the outcome of the failure that tripped it and are not cached, so the next run tries them again; the log lists each host given up on, with the number of links it left unchecked.

Blacklisted domains

Set RER_LINKCHECKER_BLACKLIST_DOMAINS in the environment of the instance (and of the cron running the script) to a comma separated list of domains contents must never link to:

RER_LINKCHECKER_BLACKLIST_DOMAINS="applicazioni.regione.emilia-romagna.it,*-iam.regione.emilia-romagna.it"

A domain covers its subdomains too (x.applicazioni.regione.emilia-romagna.it), but not a host that merely ends with the same letters (notapplicazioni.regione.emilia-romagna.it). A domain with a * is a pattern matched over the whole host, * standing for any characters, dots included: *-iam.regione.emilia-romagna.it covers the editorial domain of every site (ambiente-iam.…, salute-iam.…), but not iam.regione.emilia-romagna.it. Case and port do not matter.

Their links are reported as -4, to fix, without any request, and are not cached: the variable is read at every check, so changing it takes effect at the next run. The blacklist wins over the internal links too: when a check runs on a blacklisted domain (e.g. from the editorial domain of the site), the absolute links to it are reported all the same.

The check_broken_links console script

Installed as a standard console_scripts entry point ([project.scripts] in pyproject.toml), so it lands in bin/ both in a uv-managed virtualenv and in a zc.buildout instance. Since it runs Zope/ZODB code, it must be launched through zconsole/instance run, not called directly:

# uv-managed instance (Makefile: make check-broken-links)
./bin/zconsole run instance/etc/zope.conf ./.venv/bin/check_broken_links

# zc.buildout instance
./bin/instance run bin/check_broken_links

Options:

• --ttl, --workers, --timeout, --connect-timeout, --host-max-failures: same meaning as the check_site() arguments ttl, max_workers, timeout, connect_timeout, host_max_failures.
• --output-dir: also dump the run as a csv in this directory (default: no csv).
• --site-id: id of the Plone site to check (default: the PLONE_SITE_ID env var, or Plone).
• --url <url>: verify a single url and log its status, without touching the site.
• --content <path-or-UID>: verify a single content's links and log them, without touching the site.

The script stores the report in the site, which is where the REST API endpoints below read it from: that is the normal way to get at the result. With --output-dir it also writes <output-dir>/<siteid>_broken_links_<YYYYMMDD-HHMMSS>.csv, byte for byte the csv @linkchecker-csv serves — worth it only to keep an archive of past runs, since the site holds the last one alone.

The REST API endpoints

Two read-only endpoints on the site root expose the stored report, both guarded by the rer.linkchecker.ViewReport permission (granted to Manager, Site Administrator and Editor).

They never run a check: a full check takes minutes on a medium site and would time the request out, so they only serve what the last run stored. Refresh the data out of band, with the check_broken_links script from cron. Because the data is therefore asynchronous, both responses carry the timestamp it dates from.

# the report as json, batched (plone.restapi conventions: b_start, b_size)
curl -u user:pass "$SITE/++api++/@linkchecker" -H 'Accept: application/json'

# the same report as a csv download
curl -u user:pass -OJ "$SITE/++api++/@linkchecker-csv"

Both accept the same filters:

• status: repeatable, e.g. ?status=404&status=-2 (negative values are the STATUS_* constants: -1 timeout, -2 https-only, -3 connection error, -4 blacklisted domain).
• type: INTERNAL or EXTERNAL.

An invalid filter value answers 400 rather than silently returning an empty report.

@linkchecker returns last_update and duration for the whole run, items_total, the batched items, and a summary listing {status, status_description, count} sorted by count. The summary is computed over the unfiltered report on purpose, so the counts a UI shows in its filter chips do not move as filters are applied. When no check has ever run, last_update is null and items/summary are empty, which a UI can tell apart from "nothing is broken".

Each item describes the page it sits on with the usual plone.restapi field names, so a client can treat it as any other content reference, and names the link's own fields apart from those:

{
  "@id": "http://site/bandi-e-avvisi",
  "@type": "Document",
  "title": "Bandi e avvisi",
  "UID": "1a568f09734340dfba2c1a53730b9cf6",
  "link": "https://unimc.it/careerday",
  "link_type": "EXTERNAL",
  "status": 404,
  "status_description": "Not Found",
  "action": "fix",
  "last_update": "2026-07-30T03:00:12"
}

action is what an editor has to do about the link: fix for 404 and 410 (the resource is gone) and -4 (the domain must not be linked), update for -2 (the url only needs https), check for any other outcome.

@linkchecker-csv returns the exact same csv the console script writes (same columns, same quoting), names the file after the date of the data rather than of the download, and repeats that date in the X-Linkchecker-Last-Update response header.

Development

Prerequisites

• An operating system that runs all the requirements mentioned.
• uv
• Make
• Git
• Docker (optional)

Setup

git clone git@github.com:RegioneER/rer-linkchecker.git
cd rer-linkchecker/backend
make install

Common tasks

make start                  # start a Plone instance on localhost:8080
make create-site            # create a new site from scratch
make check-broken-links     # run the linkchecker and store the report in the site
                            # (extra flags: ARGS="--ttl 0 --output-dir /tmp")
make test                   # run the test suite

Contribute

• Issue tracker
• Source code

License

The project is licensed under GPLv2.

Credits

Developed with the support of Regione Emilia Romagna.

Regione Emilia Romagna supports the PloneGov initiative.

Authors

This product was developed by RedTurtle Technology team.