Why Your Text Arrives as Question Marks and Garbage

A message is sent and arrives as Grüße. Or as Gr??e. Or with small boxes where the accented letters should be. Three different pictures, three different causes — and only one of them can still be repaired once the message is sitting in the recipient's mailbox.

What a character set actually settles

What flows over the wire is bytes, that is, numbers from 0 to 255. Which character a given sequence of numbers represents does not follow from the sequence itself — it is a convention that has to be communicated separately.

In an email that convention sits in the Content-Type, and separately in every part of the message:

Content-Type: text/plain; charset=UTF-8

If the declaration is missing, the receiving client guesses. And it guesses differently depending on its origin and its defaults.

Where the à comes from

UTF-8 represents ü as two bytes, C3 BC. A program that takes those same bytes for Windows-1252 — an older character set with exactly one character per byte — reads two characters out of them: Ã and ¼.

The whole familiar family arises from this pattern:

The leading Ã is no coincidence: every Western European accented character begins in UTF-8 with the byte C3, and C3 in Windows-1252 is Ã. Seeing that letter sequence tells you immediately what to look for.

Question marks and boxes are something else

The three symptoms from the first paragraph are not three shades of the same fault.

• ü — read wrongly. The bytes are intact, only the interpretation is wrong. This can be undone: set the right character set and the text comes back.
• ? — lost. The text was converted into a character set that does not know the character. ü became a question mark, and nothing can be reconstructed from a question mark. This is the one irreversible case.
• Boxes or diamonds. Character set and bytes are fine, but the font in use has no glyph for that character. The text is undamaged and can be copied; only the display fails.

The distinction decides whether it is worth asking for the original again or not.

The header has its own rules

Subject, display name and attachment filenames sit in the message header, and by the original standard that may only contain ASCII. So there is a separate transcription for it:

Subject: =?UTF-8?Q?Gr=C3=BC=C3=9Fe?=

That is a properly encoded subject reading "Grüße": character set declaration, method, content. A client that understands it displays the text. If a recipient sees that sequence in the clear, the sending system chose the wrong order — encoding where it should not have, or the reverse.

This encoding is independent of the charset of the message body. Hence the common case where the subject is mangled and the body is fine, or the other way round. They are two separate matters. How the message is divided into parts overall is covered in the article on MIME.

Double-encoded

The most stubborn case is when already-mangled text is converted a second time. Grüße then becomes Grüße — the string grows with every pass.

This typically happens in chains of several systems: a form delivers UTF-8, the database is set to latin1, the sending program assumes UTF-8. Each station does the right thing on its own terms, and nonsense comes out at the end.

The only repair is to walk the chain backwards and find the point where the first misinterpretation occurred. Everything after it is consequence.

The rules that make the problem go away

1. UTF-8 everywhere, no exceptions. Form, application, database, sending, message. One single point on a different setting is enough for the whole effect.
2. Declare the character set, do not assume it. charset=UTF-8 belongs in every text part. Without it the recipient guesses, and a client that guessed right for years will eventually guess differently.
3. Leave headers to the library. Assembling encoded words by hand is error-prone, and every serious sending library gets it right.
4. With databases, watch the connection encoding, not just the table's. A utf8mb4 column is no help if the connection is on latin1 — that is exactly where double-encoding is born.
5. Test with real characters. A test run with "Test" proves nothing. A test run with "Grüße, zażółć gęślą jaźń, Ελλάδα" proves a great deal.

The last point is the cheapest and the most reliably forgotten. The fault shows up exclusively on characters that usually do not appear in the test data — and then on a customer who happens to be called Schmidt-Küppers.