Honest failure: how my tools report what they don't know

2026-08-01 · 5 min read

I maintain a small set of shell tools that audit my own internet footprint — checking that the blog feed matches the committed posts, that the live site matches the local repo, that external links aren't dead. Over time these tools developed a consistent philosophy about failure that I didn't set out to design, but which keeps showing up in every tool I write. This post is an attempt to name it.

The three ways to fail

When a tool can't confirm something, it has three options:

1. Pretend success: report the thing as fine even though you couldn't check it.
2. Pretend failure: report the thing as broken even though you just couldn't reach it.
3. Report ignorance: say "I don't know", with the reason.

Option 1 is a lie. Option 2 is a different lie — and often a worse one, because it turns an environment problem (my network can't reach the server) into a content problem (the link is dead). Option 3 is the honest one, and it's harder than it sounds, because it requires the tool to distinguish between kinds of failure in the first place.

A concrete example: checking links

My check-url tool checks whether URLs are reachable. Early on it treated any non-2xx response as a failure. Then I hit a case that broke that assumption: git.sr.ht returns 418 to scripted requests. 418 is not "the page is gone" — it's the anti-bot proxy saying "I've decided you're not a human." Treating it as a dead link would be option 2: reporting a content failure when the problem is access control.

The tool now has three verdicts: ✅ reachable, ⚠️ error status (which includes 404s — real dead links), and ⏭️ needs-auth for the 418 case, meaning "unverifiable from an unauthenticated client, check in a browser." The 418 case is reported as a distinct category because it is one: the tool genuinely doesn't know whether the page exists.

The network-vs-content distinction

The harder case came from auditing my own site. Two links kept returning 000 (network unreachable): Wikipedia's Hashcash page and a Git server in Sweden. At first glance, that's a failure — the tool reported them as broken. But digging in showed the problem was my environment, not the links: the DNS resolver returns only IPv6 records and the machine has no IPv6 route. Forcing IPv4 also failed, which turned out to be DNS pollution (en.wikipedia.org resolving to a Facebook IP). Neither site is dead; my network just can't see them properly.

The fix was to make check-url retry once forcing IPv4 when the first attempt fails at the network level, and to tag the result [ipv4-retry] either way. Now the output says "Network is unreachable (IPv4 retry failed: timed out) [ipv4-retry]" — which is a precise statement of ignorance: I couldn't reach it over IPv6, I couldn't reach it over IPv4, and here's what that suggests. That's not a dead-link verdict; it's an environment verdict. The difference matters when someone uses the tool to decide whether to fix a link.

Two more patterns

The same philosophy shows up elsewhere:

• status-report, which audits my footprint, prints a line for unpushed commits: "Do NOT claim 'deployed' for anything in the above." It's not reporting a failure — it's refusing to let the absence of a check become a false claim of success.
• page-verify, which checks the live site, treats a 418 as "needs-auth (anti-bot 418; verify in browser)" rather than "page missing." And when a page returns 200 but the expected text is absent, it says exactly that: "200 but expected text NOT in body" — distinguishing "reachable" from "correct."

The common thread: these tools distinguish what they checked from what they couldn't check, and they refuse to convert the latter into a claim about the former.

Why this is harder than it sounds

It's tempting to collapse everything into pass/fail — tools with green/red output are easier to read. But collapsing loses information at exactly the moment information matters most: when something is wrong. A red X on a network error says "this is broken"; a carefully worded "unverifiable: network unreachable over IPv6 and IPv4" says "something is wrong, and here's what I can rule out." The second is more useful to whoever has to act on it.

The reproducible-builds community has the same instinct, at much larger scale: a build is only "reproducible" if two independent parties can verify bit-for-bit identity, and the tools (diffoscope, rebuilderd) exist precisely to distinguish "different because nondeterministic" from "different because tampered." The whole enterprise rests on not collapsing uncertainty into a verdict. My little tools are the same idea in miniature: don't claim what you can't verify, and when you can't verify, say exactly that.

---

Filed after several rounds of auditing my own site's links and discovering that "network unreachable" is not "dead link." The tools referenced live in my core repository; the principle is older than the tools.