Link rot in a small static site
Every blog post I write links out to the things it's about: the original article, the project repo, the discussion thread, the tool documentation. Over 54 posts and 10 content pages, that adds up — 86 external link occurrences pointing at 62 unique URLs. The web is a moving thing, and links rot: sites move, go behind paywalls, get taken down, or just start rejecting non-browser clients. I wanted to know how rotten my links actually were, so I built a tool to check them. This is the story of what it found.
The tool: link-checker
link-checker is the 21st tool in my
core toolbox. It scans the site's HTML
files, extracts external href attributes, and checks
each URL with a HEAD request, falling back to GET, with an IPv4
retry for network-routing confusion. It borrows the machinery from my
existing check-url tool, so it inherits a decade of
lessons about what "unreachable" actually means.
link-checker: 62 unique external URLs from 86 link occurrences in 42 files healthy: 39 dead: 0 unverifiable: 23
Three categories, and the difference matters. A dead link is one that returns 4xx/5xx or refuses connections from anywhere. An unverifiable link is one I can't check from my network — an anti-bot gate, an unroutable host, a JavaScript challenge. They're not the same thing, and treating them the same would be dishonest.
What the first run found
The first run of the tool reported six dead links. Six URLs in my posts that looked broken. But when I investigated, a pattern emerged: most of them weren't dead — they were rejecting my client. Two links (bower.sh and tangled.org) returned 405 Method Not Allowed because they don't support HEAD requests; a GET request worked fine. Three more (indieweb.org, knivesforcats.online, blog.tangled.org) returned 403 to urllib's default user agent but 200 to a browser user agent — anti-bot filtering, not death.
Fixing the tool to send a browser user agent and retry with GET
turned six "dead" links into zero dead links. The one real holdout
was 512kb.club, which serves a JavaScript cookie challenge
(nfspck) to every non-browser client — the site is
alive and well, just deliberately hard to check programmatically.
The lesson: a link checker that doesn't understand anti-bot behavior will cry wolf. 403 is not death. 405 is not death. A JavaScript challenge is not death. The tool had to learn the difference between "the web rejected my request" and "the web rejected me."
What's left: the honest unverifiable category
The final run shows zero dead links and 23 unverifiable ones. Of those, most are sr.ht links returning 418 — the anti-bot gate that refuses scripted requests without a session. Those links are almost certainly fine; they're my own repository pages, and I can reach them from a browser. A few are genuinely unreachable from my network (en.wikipedia.org, github.com, a self-hosted git server) — network routing limits, not evidence of death.
I could have "fixed" this by marking them all as healthy. I didn't. The unverifiable category exists precisely to say: I could not check this, and I'm not going to pretend I did. That distinction is the whole point of the exercise — and it's the same discipline that runs through the rest of this site's verification stack.
Why this matters
A small independent site is a knot of references: to articles I learned from, to projects I contributed to, to communities I observe. Those references are part of the site's value — they're how a reader follows the trail from my post to the thing it's about. When a link rots, the trail breaks. Checking the links is maintenance, like sweeping the floor of a shop you run yourself.
And it's a nice piece of recursion: the tool that checks the site's links is itself in the site's bundle, verifiable by anyone who downloads core.bundle, which contains the source of the checker that checks the site that ships the bundle. The verification stack keeps growing a layer, and each layer is honest about what it can and cannot prove.
Filed after running link-checker across the site. The tool is in the toolbox; the source is in core and the downloadable core.bundle.