archive-md-urls 0.0.6

Turn URLs in Markdown files into archive.org snapshots

archive-md-urls: Turn URLs in Markdown files into archive.org snapshots

archive-md-urls scans Markdown files for URLs and if possible turns them into links to snapshots from archive.org. If a publication date can be extracted from the file (more info), the snapshots closest to this date will be used. If no date can be found, the latest available snapshots are used instead.

This is very useful when you use a static site generator for your personal homepage that supports Markdown for writing blogposts and pages, e.g. Pelican, Jekyll or Hugo. Older content published years ago is likely to contain link rot: links that are simply broken or now point to a different target compared to when you wrote the content. In an ideal scenario, archive-md-urls will not only fix these URLs, but also link to a snapshot that shows how a website or social media profile/post looked like when you wrote the content.

archive-md-urls tries to be smart and does not simply replace every URL it finds. Instead, it uses a list of URLs which are considered 'stable' and are therefore ignored: URLs that already point to archive.org snapshots, intra-site links (e.g. a link to another blogpost on the same homepage) and URLs that contain persistent identifiers.

Example showcase

Input file example_blogpost.md:

Tile: Example blog post
author: Stefan
date: 2013-11-06

This fake blog post from 2013 links to [example.com](http://www.example.com/), a homepage that has dramatically changed in the meantime.

But it also links to URLs which can be considered 'stable':

- [here](https://web.archive.org/web/20000622042643/http://www.google.com/) we already link to an archive.org snapshot
- [here](https://doi.org/10.1080/32498327493.2014.358732798) the link contains a persistent identifier
- and [here]({filename}/blog/2012/2012-02-05-an-even-older-blogpost.md) we link to a different post on our own homepage (Pelican format, Jekyll and Hugo intra-site links are supported too)

In addition, google.com is mentioned but not explicitly linked.

And finally, [here](www.some-madeup-link-that-hasnt-been-archived.com) we link to a homepage that doesn't have any corresponding archive.org snapshots.

Output from archive-md-urls example_blogpost.md:

Tile: Example blog post
author: Stefan
date: 2013-11-06

This fake blog post from 2013 links to [example.com](http://web.archive.org/web/20131106211912/http://www.example.com/), a homepage that has dramatically changed in the meantime.

But it also links to URLs which can be considered 'stable':

- [here](https://web.archive.org/web/20000622042643/http://www.google.com/) we already link to an archive.org snapshot
- [here](https://doi.org/10.1080/32498327493.2014.358732798) the link contains a persistent identifier
- and [here]({filename}/blog/2012/2012-02-05-an-even-older-blogpost.md) we link to a different post on our own homepage (Pelican format, Jekyll and Hugo intra-site links are supported too)

In addition, google.com is mentioned but not explicitly linked.

And finally, [here](www.some-madeup-link-that-hasnt-been-archived.com) we link to a homepage that doesn't have any corresponding archive.org snapshots.

Note how only the first link to example.com has been altered and points to a snapshot close in time to the publication date of this fake blog post (6th November 2013). Also note that URLs which are mentioned but not explicitly linked are ignored.

Install

You can install archive-md-urls via pip:

pip install archive-md-urls

However, using Pipx is recommended:

pipx install archive-md-urls

Usage

Important: archive-md-urls modifies your files directly in-place. It is recommended that the files you want to change are under version control so you can review the changes.

Once installed, you can pass any number of Markdown files or directories containing Markdown files to archive-md-urls:

# Update two files
archive-md-urls my-file.md another-file.md
Updated 13 links in 2 files.
# Update files in a directory
archive-md-urls myblog/content/blog/2014
Updated 97 links in 20 files.
# You can also combine files and directories
archive-md-urls myblog/content/blog/2014 my-file.md
Updated 103 links 21 files.

By default, directories are not searched recursively for Markdown files. For recursive search, use the -r flag (use this with caution!):

# Update URLs in all Markdown files of myblog
archive-md-urls -r myblog/content
Updated 160 links in 32 files.

Note that Markdown files are identified by the file ending .md, other file endings are ignored.

A note about speed

archive-md-urls uses asyncio with HTTPX to make asynchronous API calls. However, do not expect to get fast results, especially (but not only) when you try to change a larger amount of URLs. The Wayback Machine API can be slow or even unavailable. If archive-md-urls has to cancel the operation because of that, just re-run it on the same files again later. Links that have already been updated before will be skipped because archive.org links are considered stable.

Setup for local development

git clone https://github.com/sbaack/archive-md-urls.git
cd archive-md-urls
# Create and activate a virtualenv with your preferred tool. Example:
python -m venv project_venv
source project_venv/bin/activate
# Install dev-requirements:
make setup  # Executes python -m pip install -r dev-requirements.txt

Use pip-tools to update dependencies:

make update
# Or execute pip-tools yourself:
# pip-compile --upgrade -o dev-requirements.txt dev-requirements.in
# pip-sync dev-requirements.txt