Fix Mojibake: Rescue Garbled Vietnamese, Thai & Indonesian Text

You export a CSV of Vietnamese customer names and half of them read Nguyá»…n instead of Nguyễn. A Thai product description comes back as a wall of question marks. An Indonesian address arrives with ’ where an apostrophe should be. The data isn't lost. It was decoded with the wrong charset somewhere in the pipeline, and it can almost always be recovered.

What mojibake actually is

Mojibake is text that was encoded with one character set and then decoded with a different, incompatible one. The bytes on disk are usually fine. What breaks is the interpretation.

There are two dominant failure modes:

• Wrong single decode. Bytes that were written as UTF-8 get read as Windows-1252, TIS-620, or Latin-1. Each byte is now mapped to the wrong glyph.
• Double encoding. Text that was already valid UTF-8 gets treated as Latin-1, then re-encoded to UTF-8 again. Now every non-ASCII character is stored as two or three garbage characters that are themselves valid UTF-8, which is why the file "looks fine" to tooling but reads as nonsense.

The second case is the sneaky one: the file passes every "is this valid UTF-8?" check and still displays garbage, because the corruption happened at the character level, not the byte level.

Why Southeast Asian text gets hit hardest

ASCII survives almost any mistreatment because every ASCII byte is below 0x80 and maps to itself in most legacy encodings. English text rarely shows visible mojibake. The moment you leave ASCII, the wheels come off, and SEA languages leave ASCII constantly.

• Vietnamese stacks diacritics densely. A single ễ (as in Nguyễn) is a multi-byte UTF-8 sequence, and legacy pipelines built around CP1258 or Windows-1252 mangle it in predictable, ugly ways.
• Thai has no legacy single-byte home in the Windows-1252 world at all. It lived in TIS-620 (and its Windows cousin CP874) for decades. Feed TIS-620 bytes to a UTF-8 reader and you get replacement characters or ? soup.
• Indonesian and Malay are mostly ASCII, so the damage is subtler: smart quotes, the rupiah sign, em-dashes, and the occasional é in a loanword turn into ’, â€", and friends after a Windows-1252 round trip.

Legacy Windows exports, old MySQL tables set to latin1, and CSV files with no byte-order mark are the usual crime scenes.

Recognising the tell-tale patterns

You can often name the exact wrong decode from the shape of the garble.

| You see | Real character | Likely cause | |---|---|---| | é | é | UTF-8 read as Windows-1252 / Latin-1 | | ’ | ’ (right single quote) | UTF-8 smart quote read as Windows-1252 | | â€" | — (em dash) | UTF-8 read as Windows-1252 | | â, à | â, à | UTF-8 Vietnamese read as Latin-1 | | á»…, ệ | ễ, ệ | UTF-8 Vietnamese double-encoded | | à¸, à¹ runs | Thai syllables | UTF-8 Thai read as Latin-1 |

Two heuristics catch most cases in the wild:

• A capital Ã or lowercase â immediately followed by another odd character almost always means "UTF-8 was read as a single-byte Western encoding."
• Long runs of à¸/à¹ pairs are UTF-8 Thai that took a trip through Latin-1.

The fix: re-decode through the correct chain

Recovery means reversing the exact sequence of wrong steps. You encode the garbled string back into the bytes it was mistakenly read as, then decode those bytes with the charset that should have been used.

For the classic "UTF-8 read as Latin-1" case, the reversal is one round trip:

// garbled = "Nguyá»…n"  (should be "Nguyễn")
const bytes = Buffer.from(garbled, "latin1"); // undo the wrong decode
const fixed = bytes.toString("utf8");         // apply the right one
// fixed === "Nguyễn"

Before and after, end to end:

Broken:  Nguyá»…n Vän Ân  — công ty TNHH
Fixed:   Nguyễn Văn Ân  — công ty TNHH

If the text was double-encoded, you may need to run the round trip twice. If the source was Thai from a legacy system, the correct decode is tis-620 (or cp874) rather than utf8. The key is identifying which wrong decode happened; from there the reversal is mechanical. Testing a couple of candidate chains and eyeballing which one produces real words is faster than guessing blind, and it is exactly what the mojibake fixer automates: it tries the common SEA charset chains and shows you the readable result.

Prevention: UTF-8 end to end

Fixing mojibake after the fact is recovery. Not creating it is cheaper. The rule is boring and it works: use UTF-8 at every hop, and declare it everywhere.

• Database. Store columns as utf8mb4 (real 4-byte UTF-8), not MySQL's misnamed utf8. A latin1 column will silently mangle anything above ASCII.
• Connection charset. The column being utf8mb4 is not enough. The client connection must also negotiate utf8mb4, or the driver re-encodes on the way in and out.
• Content-Type. Serve Content-Type: text/html; charset=utf-8 (and the equivalent for JSON and CSV). A browser or parser guessing the charset is a browser guessing wrong.
• File I/O. Read and write files with an explicit UTF-8 encoding. Never rely on the platform default, which differs between a Windows CI box and a Linux container.
• CSV exports. If a downstream tool (looking at you, Excel) insists on a legacy default, emit a UTF-8 BOM so it stops guessing.

Key takeaways

• Mojibake is a decode mistake, not data loss. The bytes are usually intact and the text is recoverable.
• Two root causes dominate: bytes decoded with the wrong charset, and already-valid UTF-8 that got double-encoded.
• SEA text is disproportionately affected because Vietnamese diacritics, Thai script, and non-ASCII punctuation all leave the ASCII safe zone.
• The garble pattern tells you the culprit: é and ’ mean "UTF-8 read as Windows-1252"; à¸/à¹ runs mean "UTF-8 Thai read as Latin-1."
• Fix by reversing the wrong decode chain; prevent by keeping UTF-8 unbroken from database to connection to Content-Type.

Stop hand-crafting Buffer.from(...).toString(...) incantations for every broken export. Paste the garbled text into the mojibake fixer, let it walk the common Vietnamese, Thai, and Indonesian charset chains, and copy back the version that reads like real words.