linkmend

linkmend finds external links that have rotted (gone dead) inside a static site's Markdown content, and mends them by swapping in the closest-in-time archive capture it can find — from the Wayback Machine, WebCite, or archive.today. It ships as both a linkmend CLI for walking a content tree and a Python library for callers who want to resolve individual URLs themselves.

Install

The core package has zero runtime dependencies:

pip install linkmend

Two optional extras unlock more:

# Needed by `linkmend check`: reads a post's frontmatter `date` (for
# picking a near-in-time capture) and walks comment sidecar files.
pip install linkmend[hugo]

# The bundled requests-backed HTTP client, used for live link probes
# and archive lookups. Skip this if you'll supply your own HTTPClient
# (see "How it picks a replacement" below) or only ever run --offline.
pip install linkmend[requests]

# Both, which is what `linkmend check` needs for a normal (online) run:
pip install "linkmend[hugo,requests]"

Quick start

CLI

Point check at a directory of Markdown content. By default it reports what it finds without touching any files:

linkmend check --content path/to/site/content

Add --rewrite to swap dead links for archive replacements in place, and --report to write a JSON summary you can keep or hand to import-report later:

linkmend check --content path/to/site/content \
    --rewrite \
    --report rotted.json

Sample output:

files with external links: 2
external links checked:    4
rotted (with replacement): 2
rotted (no replacement):   1

Run --offline to resolve only from what's already cached (no network calls at all — useful in CI, or to preview a rewrite before spending API calls):

linkmend check --content path/to/site/content --offline --rewrite

To proactively archive a URL before it goes dead:

linkmend save-url https://example.com/some/page

On success this prints the archived URL and exits 0; if the Wayback Machine declines or the request fails, it prints an error to stderr and exits 1.

Run linkmend <subcommand> --help for the full flag list (check, save-url, and import-report each have one).

Library

The same resolution check uses is available directly, for callers who want to check individual URLs rather than walk a directory:

from datetime import UTC, datetime
from linkmend import ArchiveCapture, LinkCheckDB, LinkChecker

db_path = "linkmend.db"

# In practice this cache is already populated by a prior `check` run;
# seeded here so the example is self-contained.
db = LinkCheckDB(db_path)
now = datetime.now(UTC)
db.record_seen("http://dead.example/some/page", context_date=now)
db.record_archive_captures(
    "http://dead.example/some/page",
    [
        ArchiveCapture(
            provider="wayback",
            archive_url="https://web.archive.org/web/20120602000000/http://dead.example/some/page",
            captured_at=datetime(2012, 6, 2, tzinfo=UTC),
            discovered_at=now,
        )
    ],
)
db.close()

checker = LinkChecker(db_path, offline=True)
result = checker.check(
    "http://dead.example/some/page",
    context_date=datetime(2012, 6, 1, tzinfo=UTC),
)
print(result.is_alive, result.replacement_url, result.replacement_source)
# False https://web.archive.org/web/20120602000000/http://dead.example/some/page wayback
checker.close()

Drop offline=True and pass an http client (see requests_http_client, or supply your own — anything matching the HTTPClient protocol works) to probe links live and query the archive providers.

How it picks a replacement

For each URL, check tries, in order:

1. A cached probe result that's still fresh (alive and within the TTL).
2. A live HTTP probe (unless --offline).
3. Caller-supplied fallback URLs, if any were given.
4. Archive providers, in provider_priority order — wayback, webcitation, then archive.is by default.

Once a provider returns any captures for a dead URL, linkmend picks the one closest in time to the post's publication date (its frontmatter date, or the sidecar's parent post's date for comment links) — not necessarily the newest or oldest capture. Ties (same distance) are broken by provider_priority, so an equally-close Wayback capture wins over an equally-close archive.today one. This logic lives in LinkCheckDB.best_capture (src/linkmend/db.py) and is shared by both the online and --offline resolution paths.

Cache location

linkmend keeps a SQLite cache of every URL it has seen, its latest probe result, and any archive captures it has discovered, so repeat runs don't re-probe or re-query archives needlessly. Resolved, in order:

1. $LINKMEND_DB, if set — an explicit path to the database file.
2. $XDG_DATA_HOME/linkmend/linkmend.db, if XDG_DATA_HOME is set.
3. ~/.local/share/linkmend/linkmend.db, otherwise.

Every subcommand also accepts --db PATH to override this directly for a single invocation.

Reusing a previous report

import-report reseeds the archive_captures cache table from a report check --report previously wrote:

linkmend import-report rotted.json

This exists as a recovery path: if the cache database is lost (a new machine, a cleared CI cache, a deleted linkmend.db) but a report from an earlier run survives, import-report restores the replacement URLs it already found without re-querying rate-limited archive APIs from scratch. It only imports entries that name an actual archive capture — entries with no replacement, or where the replacement came from a caller-supplied fallback rather than an archive, are skipped.

Limitations

• Inline Markdown links only. The extractor handles [text](url "title") and the image-wrapped form [![alt](thumb)](target). Reference-style links ([text][ref] with a separate [ref]: url definition) and bare images are not detected and will be left untouched.
• Comment sidecars are opt-in and schema-specific. The bundled HUGO_COMMENT_SIDECAR preset expects comments.json beside a post's index.md. Other layouts need a custom SidecarSpec.
• Archive availability is not guaranteed. A dead link with no capture in any provider is reported, not fixed. Do not put linkmend check in a deploy gate — a third-party archive outage would then break your deploy.
• Soft-404 detection is extension-driven, and only one-way. A URL ending in a non-HTML extension (pdf, docx, zip, png, and a handful more) that answers 200 text/html is judged dead: the host served its fallback shell instead of the file, which is what Cloudflare Pages, most SPA hosts, and many WordPress installs do in place of a 404. The converse is not detected — an extensionless URL answering text/html could legitimately be anything, so a soft 404 on a plain page still reads as alive. 401, 403, and 429 remain ambiguous and are left alone regardless of content type.
• A soft 404 on your own site gets archived, not fixed. Once a link is judged dead it falls through to the archive providers, which is right for a genuinely dead external link. If the soft 404 comes from your own site — a file you moved or dropped during a migration — the fix belongs upstream, in your content. linkmend cannot tell the two apart and will not try.

The CLI wires up only the HUGO_COMMENT_SIDECAR preset in this release (0.2.0); a different sidecar shape means using the library directly with your own SidecarSpec, not a CLI flag.

Project structure

src/linkmend/
    __init__.py    Public API surface (__all__)
    __main__.py    `python -m linkmend` entry point
    cli.py         argparse CLI: check / save-url / import-report
    checker.py     LinkChecker — resolution policy (cache → probe → fallback → archive)
    db.py          SQLite persistence (LinkCheckDB, best_capture)
    markdown.py    Inline-Markdown-link extraction and rewriting
    comments.py    JSON comment-sidecar walking (SidecarSpec, HUGO_COMMENT_SIDECAR)
    providers.py   Archive provider integrations + HTTPClient protocol
    report.py      Report writer and import-report reader
tests/
    test_*.py      One test module per area above

Licence

Apache License 2.0 — see LICENSE.