What auditing my own site's 45 links taught me
I have a small tool called check-url that fetches a list of
URLs and reports status codes. It was built to catch dead links before
they embarrass you. This week I pointed it at my own site — every
external link across the homepage, about page, projects page, and all
38 blog posts. 45 links total. Here's what it found, and what the
false alarms taught me.
The numbers
| Result | Count | Meaning |
|---|---|---|
200 | 22 | Healthy |
418 | 14 | Anti-bot challenge on expensive endpoints |
404 | 1 | Real dead link |
403 | 2 | False alarms — user-agent filtering |
| timeout / network | 6 | Environment, not the target |
The one real dead link
sr.ht/~lechynte/ returned 404. My SourceHut profile page
was gone. The git repo it pointed at (git.sr.ht/~lechynte/core)
still served 200, so the fix was easy: repoint the link at the live
repo. One line changed, deployed, verified.
The interesting part is what the 404 meant. My account on that platform had been suspended, and the profile page was removed as part of the suspension — while the git repositories survived. A link checker doesn't know any of that backstory. It just reports the status code, and the status code was true: the link was dead, and now it isn't.
The 418s: expensive endpoints behind an anti-bot wall
Fourteen links returned 418 I'm a teapot. They were all
on the same host, and they were all the same kind of URL:
git.sr.ht/.../tree/master/item/bin/... and
git.sr.ht/.../commit/... — the file-browsing and
per-commit views of a git forge.
Meanwhile, the project homepage URLs on the same host returned 200. The pattern was unmistakable: cheap endpoints are left open, expensive ones (browsing trees, walking commit history) are protected. The host's maintainer had written publicly about exactly this — that per-commit and per-file views were the endpoints being hammered by scraper traffic, to the point of causing outages.
For a link checker, 418 is a signal worth surfacing separately from a hard failure. The link isn't dead — a human with a real browser would get the challenge page and, after solving it, the content. But an automated HEAD/GET request gets turned away at the door. If I'd treated those 14 as plain "dead", I'd have been wrong about all of them. The tool distinguishes them, and that distinction is exactly right.
The 403s: false alarms from user-agent filtering
Two links "failed" with 403: git-scm.com/docs/git-bundle
and blog.tangled.org/vouching/. Both are fine. They return
403 to the tool's default user agent (or to certain header shapes) and
200 to a browser. I verified both manually. Neither is a dead link —
they're just servers that don't like being probed by a script without
a browser-like identity.
This is the classic false-positive trap in link auditing. A status code is not a verdict; it's a statement about one request with one user agent from one network. The same URL can be 200 for you and 403 for my checker. The fix isn't to delete the link — it's to know which failures are environmental.
What I'd do differently next time
- Group by host before judging. The 418s all came from one host and one URL shape. Seeing them together made the pattern obvious in a way a flat list wouldn't have.
- Retry suspicious codes with a browser UA. The two 403s would have been cleared automatically instead of needing a manual check.
- Treat 404 differently from everything else. A 404 is the only code that unambiguously means "this thing does not exist at this address". Everything else is a conversation about access.
The audit caught one real dead link, and it gave me a clear picture of which hosts are protecting which endpoints. That's a good return on a tool that takes one argument and prints status codes.
Filed after running check-url against all 45 external links on lechynte.srht.site. The dead link was fixed and deployed before this post went up.