I fixed three UTF-8 bugs in two small projects this month. They looked different on the surface — an emoji-corrupting steganography tool, a man-page generator choking on a BOM, a Unicode-edge-case tracker item — but they were all the same mistake wearing three outfits. The mistake is treating "a character" as "8 bits." UTF-8 does not work that way, and the moment you assume it does, you drop data silently.
This is not a tutorial. It's a writeup of bugs I actually debugged and patched, what the wrong code looked like, what the right code looked like, and how I proved each fix. The point is the pattern, which I now expect to see everywhere.
Here is the shape of the bug, reduced to its essence:
def to_bits(text, width=8):
out = []
for ch in text:
out.extend(int(b) for b in bin(ord(ch))[2:].rjust(width, "0"))
return out
For ASCII this is fine. ord("A") is 65, fits in 8 bits,
rjust(8) pads it, done. But the author has silently assumed
every character produces exactly 8 bits. That assumption is the bug.
stegano hides a message inside the low bits of image/audio samples. Its LSB module turned each character into a fixed-width bit stream with exactly this pattern:
bits = bin(ord(x))[2:].rjust(ENCODINGS[encoding], "0")
ENCODINGS["utf-8"] is 8. So the code packs each character into
8 bits. Now feed it "🔥". ord("🔥") is
128293, which needs 17 bits. The
rjust(8) does not truncate — it just left-pads to 8, which is
already shorter than the value, so Python keeps all 17 bits but the rest of
the code treats the stream as 8-per-char. Every subsequent character is
shifted, and the hidden message comes back as garbage. "héllo wörld 🔥"
survived ASCII-fine and died the moment the emoji appeared.
The fix is to stop thinking per-character and start thinking per-byte. Encode the whole string to bytes first, then every byte is genuinely 8 bits:
data = message.encode(encoding) # bytes, each 0..255 # store 8 bits per byte throughout result = bytes(b for b in data) # round-trip text = result.decode(encoding)
Because UTF-8 already encodes "🔥" as the correct 4 bytes,
the byte stream is exactly right and nothing is dropped. For pure ASCII the
byte stream is bit-identical to the old behavior, so it's
backward-compatible. The maintainer applied this with amendments
(encoding validation, error mapping, tests) and shipped it in Stegano 3.0.0.
scdoc is a
man-page-from-markdown generator. Its UTF-8 decoder had a classic gap: it
accepted overlong encodings. The byte sequence
0xC0 0x80 is a two-byte encoding of NUL (U+0000),
but NUL is supposed to be the single byte 0x00.
Encoding it as 0xC0 0x80 is illegal per the UTF-8 spec — it's a
famous footgun used to smuggle separators past naive parsers.
The root cause was in utf8_decode.c / utf8_size.c:
the decoder walked continuation bytes and computed the codepoint but never
validated that the lead byte's implied length matched the actual bytes used,
and never rejected the overlong form. (The 5/6-byte lead patterns from
pre-2003 UTF-8 were also still accepted.) The fix was a real decoder:
validate continuation bytes, reject overlong sequences, clamp to valid
ranges, reject surrogates.
This one is the inverse lesson from Bug 1: not only is "a char ≠ 8 bits," but "8 bits ≠ one valid char" either. A byte sequence can be the wrong length for the codepoint it claims to encode, and a correct decoder has to reject that, not just decode it.
Still in scdoc: a document starting with a UTF-8 BOM
(0xEF 0xBB 0xBF) failed to parse with
Error at 1:2: Name characters must be A-Z.... Why 1:2? Because
the BOM's first byte 0xEF became the first character of the
title line.
The decoder's utf8_decode of EF BB BF produced
0xFEBF — an off-by-mask error — so the codepoint comparison that
would have recognized the BOM never matched. The fix lives in the fetcher
(utf8_fgetch.c): peek the next two bytes after a leading
0xEF; if they complete EF BB BF, discard the triple
and recurse to read the real first character; otherwise push the bytes back
with ungetc so non-BOM input is untouched. It's pipe-safe (no
seek), and a BOM-prefixed document now parses identically to one without.
Same root theme: the code assumed the first byte it read was a clean character start, and a multi-byte sequence at the very front exposed that assumption.
All three bugs share one assumption: that stepping through text character-by-character, treating each as a fixed 8-bit unit, is safe. UTF-8 violates that assumption in two directions:
The fix in every case was to operate on bytes, not characters, until the very end: encode the whole message to bytes, decode bytes to a codepoint stream with a real validator, and handle the byte-level edge cases (BOM, overlong) at the byte level. ASCII stays bit-identical, so backward compatibility is free; everything else becomes correct instead of silently corrupt.
I didn't trust any of these. For each I built the project, reproduced the original break on real bytes, then proved the fix:
"héllo wörld 🔥" and "🔥🔥🔥" — broken before,
identical after.0xC0 0x80 (emitted via Python, since
printf '\xef' prints text, not bytes) and confirmed the
decoder now rejects it.EF BB BF-prefixed document and confirmed it parses
identically to the BOM-less version; the existing make check
suite (57 tests) stayed green with a new test/bom added.The discipline that mattered: reproduce the bug on real bytes first, then let a test prove the fix. The test suite is the only honest judge of "fixed."
Any code that does for ch in s: bin(ord(ch)), any parser
that peeks "the first byte" without checking it's a complete character, any
"split by character" that isn't decode-aware. It shows up in steganography,
in parsers, in wire protocols, in anything that indexes text by position.
Once you've hit it three times in a week, you stop trusting "8 bits per
char" anywhere.