Can “�” and “?” Mojibake Be Recovered? Restoring Lost Characters, and How to Spot Text That’s Gone for Good

“�” stands in for bytes that couldn’t be read

“�” is the Unicode character U+FFFD (the replacement character). Software displays it when it can’t read some bytes as text. When it meets a byte sequence that breaks the encoding’s rules, it substitutes “�” for the unreadable part and moves on.

You often see it when UTF-8 Japanese is opened as Shift_JIS. In UTF-8, hiragana and kanji take 3 bytes per character, while Shift_JIS reads in 2-byte units, so the boundaries drift out of alignment. Wherever that drift produces a byte pair that doesn’t exist in Shift_JIS, or a single byte is left over at the end, you get “�”. The word 文字化 + ke (“mojibake”) becomes “譁�ｭ怜喧縺�”, and 会議 no 資料 (“meeting materials”) becomes “莨夊ｭｰ縺ｮ雉�侭”.

How text with “�” can still be filled in

UTF-8 has a fixed shape: the first byte determines how many bytes the character has, and the following bytes fall in the range 80–BF. Tracing garbled text backward reveals how many bytes are missing and where, such as “this character is missing the second of its three bytes”. Candidates can then be narrowed to only the bytes that really do turn into “�” when read as Shift_JIS.

The Mojibake Fixer & Decoder tries the remaining candidates one by one and picks those that make natural Japanese (common kanji and hiragana). Here’s what happened when we tried it. Results containing hiragana are shown in romaji, with an English gloss.

As the last example shows, a final character has few clues around it and can come out wrong. Filled-in characters are highlighted in yellow, and tapping one swaps it for another candidate, so reselect any that don’t fit the context. The longer the text, the more clues there are and the better the guesses.

How to spot mojibake that can’t be recovered

Inference works only while most of the garbled text still holds the original bytes. In the following cases the original information is gone, so the text can’t be restored.

• All “?”, as in “?????”: when the text was converted and saved, characters the target encoding couldn’t represent were replaced with “?”. Nothing of the originals remains
• All “�”, as in “������”: common when Shift_JIS or UTF-8 text is read as EUC-JP. Nearly every byte becomes “�”, leaving nothing behind
• Letters alternating with “�”, as in “�e�X�g”: the first of each two bytes is lost and only the second survives (this example was the katakana word for “test”)
• “・” mixed in, as in “譁・ｭ怜喧縺・”: software such as Excel displayed “・” in place of “�”, and it was then saved or copied that way. It can’t be told apart from a genuine “・”, so it often can’t be filled in

Characters the tool can’t restore are marked with “〓”. When you see many of them, give up on recovering from the garbled text and get the original file or email, then reopen it in the correct encoding.

Double mojibake and “�”

If already garbled text is saved and then read in yet another encoding, you get mojibake twice over. For example, take “譁�ｭ怜喧縺�”, from UTF-8 read as Shift_JIS, save it as UTF-8, and read that as Latin-1. You get “è­ï¿½ï½­æ€œå–§ç¸ºï¿½”.

“�” is what the bytes of “�” saved in UTF-8 (EF BF BD) look like when read as Latin-1, and it’s the telltale sign of double mojibake. The tool tries two rounds of garbling by brute force, so even this example returns to 文字化 + ta, and choosing the right final character from the candidates restores the original. Text garbled three or more times, or replaced by “?” along the way, may not be recoverable.

How to avoid making mojibake worse

1. 1

   Don’t save over a garbled file

   Saving a file while it displays garbled destroys the original bytes behind each “�”. If you notice the problem, close without saving and reopen the file with the encoding specified.

2. 2

   Keep the original file or email

   Make a copy of the original before converting or editing. With the original in hand, you can restore it reliably without inferring anything from the garbled text.

3. 3

   Paste it exactly as it is, “�” included

   When pasting garbled text into the tool, don’t delete the “�” or extract only the readable parts. The positions of the “�” are the clues used to fill the gaps.

To work out which encoding mix-up caused the garbling, see the Mojibake Patterns Chart. If a CSV came out garbled in Excel, follow How to Fix CSV Garbled in Excel to reopen it, and you can read the original characters without inferring any “�”.

FAQ

What is the “�” in garbled text?

It’s the Unicode replacement character U+FFFD, displayed where software couldn’t read bytes as text. When a file is opened in the wrong encoding, byte sequences that break the rules are replaced with “�”.

Can mojibake containing “�” be restored?

If the “�” came from opening UTF-8 as Shift_JIS, inference can often fill it in, because usually only about one byte per character is missing and UTF-8’s rules narrow down the candidates. Spots with few clues, such as the end of the text, can come out wrong.

Can characters that turned into “?” be restored?

No. When the text was converted and saved, characters that couldn’t be represented were replaced with “?”, so nothing of the originals remains. If you have the original file, reopen that in the correct encoding.

What does “�” mean?

It’s the bytes of “�” saved in UTF-8 (EF BF BD), read as Latin-1 (Windows-1252). It’s a sign that already garbled text was garbled again, and reversing both rounds can sometimes restore it.