01 / Beyond ASCII without losing compatibility
Early text exchange needed agreed letters, digits and control characters. Seven-bit ASCII provided a common basis for English. Other languages needed more characters, so regions developed encodings within compatibility and capacity constraints. The same byte can mean different text under different rules. Exchanging bytes is not enough; the interpretation must be known.
GB2312 addressed early simplified-Chinese exchange; GBK expanded character coverage. GB18030 extended it with one-, two- and four-byte sequences. None means that Chinese always takes two bytes. GBK cannot represent all Unicode characters; rare characters and emoji expose its limits. Quietly replacing them with question marks loses information even if the file appears to export successfully.
Latin-1 and Windows-1252 differ: bytes 0x80–0x9F are controls in Latin-1, while some represent visible punctuation in Windows-1252. Browser compatibility aliases can give the same historical label different meanings across APIs. Here, Latin-1 explicitly means ISO-8859-1.
02 / Unicode assigns numbers; UTF encodes them
Unicode gives characters a shared code space rather than separate regional character sets. A code point is a number; UTF-8 and UTF-16 represent those numbers as code units and bytes. Saying a file is Unicode does not identify its encoding, byte order or boundaries.
UTF-8 preserves ASCII bytes and uses one to four bytes per Unicode scalar value, easing compatibility with existing ASCII protocols. UTF-16 uses one or two 16-bit code units; supplementary characters require surrogate pairs, and files need a byte-order rule. An isolated surrogate is not an independently encodable scalar value. This page rejects it instead of silently replacing it.
JavaScript length counts UTF-16 code units; string iteration uses code points. One perceived character can contain several code points, such as e with a combining accent or a family emoji. An extended grapheme cluster is not a font glyph count either. This page shows four separate measures rather than one ambiguous length.
03 / Wrong interpretation or lost information
A file may be written as UTF-8 and read with another encoding, with further disagreement in database connections, HTTP headers or program conventions. Re-encoding the displayed text can preserve the mistake in another file. Garbled text usually calls for tracing the original bytes through encoding and decoding, not changing the font.
If the original bytes remain, you can try the correct decoding. Once invalid sequences become U+FFFD (�), or unsupported characters become ?, distinct originals may collapse to one replacement and cannot be uniquely recovered. Those characters can also be legitimate original text; their presence alone does not prove corruption.
Strict validation detects invalid UTF-8 and truncated multibyte sequences. But ASCII fits many encodings, and garbled text can still be syntactically valid in another one. Successful decoding does not prove the encoding is correct. This page retains the bytes and provides explicit choices and comparisons, without fabricated confidence scores, automatic repair or normalization.
BOMs, conversion and protocol boundaries
BOMs originally identified Unicode byte order: UTF-16LE often starts FF FE and UTF-16BE FE FF. UTF-8 has no byte-order issue and needs no BOM, though software sometimes treats EF BB BF as a signature. Rules differ by protocol; network JSON senders should not add a BOM. A header clue is not a substitute for a protocol contract.
A matching leading BOM can be consumed as a marker or retained as U+FEFF content. The target BOM is a separate choice. UTF-32 markers are reported, but UTF-32 conversion is not supported. U+FEFF in the body, line breaks, combining characters and whitespace are not automatically removed or changed. Such cleanup can affect hashes, signatures or program behavior.
Unicode decoding is strict; legacy encodings use byte round-trip checks. The latter is conservative and may reject valid bytes with non-unique mappings. The target encoding must reproduce the same text or export is blocked. GB18030 uses the library's mappings; this is not a GB18030-2022 compliance test.
Character encoding turns text into bytes. Hex and Base64 represent existing bytes as readable or transportable text. Base64 does not resolve a source-encoding mismatch or encrypt the data. For byte-container rules and validation, see Binary encoding.