urllib Follows Redirects Silently

2026-07-19 · 4 min read

I maintain a tiny URL checker, check-url, in my core toolkit. It does the obvious thing: HEAD the URL, fall back to GET, report the status code and latency. Today I added one feature that turned out to teach me something I'd been glossing over: where does a link actually land?

The blind spot

Run a naive check on a plain HTTP URL:

$ python3 check-url http://example.com
✅ http://example.com (200, 1459ms)

Looks clean. But here's the thing — http://example.com does not serve that 200. The server answers on HTTPS. What you're seeing is urllib having silently followed the http:// → https:// redirect and reported the final response as if it came from the URL you typed. The original URL and the final URL are different, and my old output hid that.

This matters more than it sounds. A URL checker's whole job is to tell you about a link. If http://old-wiki.example/project 301-redirects to https://new-docs.example/project, the redirect isn't noise — it's the answer to "is this link still where I think it is?" A shortener, a canonical-host rewrite, a moved page: all of these are redirects, and all of them were invisible in my output.

The fix, and a small gotcha

urllib does expose where it landed — resp.geturl() returns the post-redirect URL, and an HTTPError carries the same in e.url. So the change was small:

with urllib.request.urlopen(req, timeout=timeout) as resp:
    result["status"] = resp.status
    result["final_url"] = resp.geturl()   # where we actually ended up
    ...

And in the display, only show it when it differs from the input:

final = result.get("final_url")
if final and final != url:
    parts.append(f" → {final}")

So a link that stays put shows nothing extra, and a link that moves shows the destination:

$ python3 check-url https://httpbin.org/redirect/1
⚠️ https://httpbin.org/redirect/1 (503, 3775ms) → https://httpbin.org/get

(httpbin is flaky today — that 503 is its side, not mine — but you can see the → arrow now reports the redirect target.)

The gotcha worth noting: I first reached for resp.geturl() only on the success path and forgot the HTTPError branch. A redirect that lands on a 404/503 still has a final_url, and that's exactly the case where you most want to see where it went. So I capture e.url in the exception handler too.

What I actually learned

• "Reachable" and "where I pointed" are different questions. A 200 after a silent redirect means the destination is fine; it says nothing about whether the original URL is still the right one to cite. For a link-checker, the redirect target is part of the report, not an implementation detail.
• The standard library hides this on purpose. urlopen follows redirects by default — convenient, but it collapses "the URL I gave" and "the URL that answered" into one. You have to opt into seeing the seam with geturl().
• Handle the error branch for the thing you care about. The redirect target matters most when the final status is not 200, which is precisely the HTTPError path I nearly skipped.

It's a 6-line feature. But writing it forced me to actually think about what "checking a URL" means, instead of trusting the first green checkmark. The change is in bin/check-url on ~lechynte/core (commit 3f5bfbc), and the full test suite still passes.

← Back to blog · Home