Base64 and Unicode in JavaScript: Fix btoa Errors and Mojibake

Base64 problems usually start with a category error: treating Base64 as a text encoding. Base64 is a way to represent arbitrary bytes using 64 printable characters. It says nothing about whether those bytes are UTF-8, UTF-16, an image, or a compressed archive.

Once you separate bytes from text, the errors become predictable and fixable.

Why btoa fails on non-Latin text

The browser function btoa accepts a "binary string": each character must have a code point from 0 to 255. It converts those characters to bytes and encodes the bytes.

btoa("hello");        // "aGVsbG8="
btoa("你好");          // InvalidCharacterError
btoa("こんにちは");     // InvalidCharacterError
btoa("🚀");           // InvalidCharacterError

Chinese, Japanese, Korean, and emoji characters have code points above 255, so btoa rejects them. The error is not a bug in btoa; the input is outside its contract.

The browser message is Uncaught DOMException: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range. Node's legacy global btoa reports InvalidCharacterError: Invalid character. Neither message includes the index of the offending character, so finding it in a long string means scanning for the first code point above 255 or bisecting the input.

The more dangerous case is Latin-1 text that does not throw at all. é is U+00E9, inside btoa's accepted range, so btoa("é") returns 6Q==, which is the Base64 of the single byte 0xE9. A UTF-8 consumer expects the two bytes 0xC3 0xA9, whose Base64 is w6k=. No exception is raised; the damage appears later when a server or TextDecoder interprets the bytes as UTF-8. The check is therefore not "does btoa throw" but "were the input bytes produced by a known encoding."

The correct UTF-8 pipeline

Encode text to UTF-8 bytes first, then convert those bytes to a binary string that btoa can accept.

function base64EncodeUtf8(text) {
  const bytes = new TextEncoder().encode(text);
  let binary = "";
  for (const byte of bytes) binary += String.fromCharCode(byte);
  return btoa(binary);
}

function base64DecodeUtf8(encoded) {
  const binary = atob(encoded);
  const bytes = Uint8Array.from(binary, (char) => char.charCodeAt(0));
  return new TextDecoder("utf-8", { fatal: true }).decode(bytes);
}

base64EncodeUtf8("你好, 世界");
base64DecodeUtf8("5L2g5aW9LCDkuJbnlYw=");

The round trip works because every step has a clear type:

text -> UTF-8 bytes -> binary string -> Base64 text
Base64 text -> binary string -> UTF-8 bytes -> text

Skipping the byte step is what produces mojibake. If you decode Base64 as Latin-1 and then display it as UTF-8, multi-byte characters are split and replaced with question marks or strange symbols.

TextDecoder has a strict mode worth using at trust boundaries. With fatal: true, invalid UTF-8 throws TypeError: The encoded data was not valid for encoding utf-8 instead of substituting U+FFFD, so a mis-encoded payload fails loudly. The tradeoff is that one bad byte rejects the entire string; without fatal, decoding always succeeds and silently loses information. Display code can tolerate replacement characters, while credential material and anything that will be re-signed cannot.

TextEncoder only produces UTF-8 and takes no encoding argument. It also converts unpaired surrogates to U+FFFD instead of throwing, so a string containing "\uD800" is silently changed before it reaches Base64. If the original code units must survive byte-for-byte, validate them first.

The byte-to-binary-string loop is fine for form fields and headers and wrong for large blobs. Appending one character at a time grows a string proportional to the input, and btoa then allocates the encoded copy. A 10 MB file becomes roughly 13.3 MB of Base64 plus the intermediate strings. For large data, chunk the byte array at multiples of three bytes and encode each chunk separately: three bytes map to exactly four Base64 characters, so concatenating per-chunk output stays valid. Newer runtimes expose Uint8Array.prototype.toBase64 and Uint8Array.fromBase64, which operate on bytes directly and skip the binary-string step; check runtime support before relying on them.

The Node.js equivalent

In Node.js, Buffer handles the conversion directly:

const encoded = Buffer.from("你好", "utf8").toString("base64");
const decoded = Buffer.from(encoded, "base64").toString("utf8");

This is the same pipeline with a shorter API. Do not pass a JavaScript string to Buffer.from(text, "base64") unless you actually have Base64 text; the encoding argument describes the input format.

Buffer also supports "base64url", so buf.toString("base64url") produces the -/_ alphabet without padding and needs no manual replacement.

Node's decoder is deliberately lenient. Buffer.from("abc!!!", "base64") ignores characters outside the alphabet, and decoding stops at the = that terminates a padded payload. Leniency is convenient and risky: corrupted input often decodes to some bytes instead of throwing. buf.toString("utf8") is equally forgiving and replaces invalid sequences with U+FFFD, so it cannot tell you whether the bytes were valid. To verify, re-encode the decoded bytes and compare with the normalized input, or compare a SHA-256 digest against the producer's.

Emoji and surrogate pairs

Emoji often live outside the Basic Multilingual Plane. JavaScript strings represent them as surrogate pairs, which is another reason per-character loops fail.

"🚀".length; // 2 UTF-16 code units
[... "🚀"].length; // 1 code point

TextEncoder handles code points correctly. If you write your own encoder, iterate over code points with for...of or use a well-tested library. Do not assume one JavaScript character equals one byte.

A family emoji sharpens the point. "👨‍👩‍👧‍👦" is seven code points — four emoji joined by three zero-width joiners — stored as eleven UTF-16 code units, and [...value].length returns 7. Base64 never sees characters or grapheme clusters: TextEncoder emits the UTF-8 bytes and the encoder does not need to know how many people the picture contains. That is another argument for deleting per-character conversion code. If a manual loop is unavoidable, iterate code points and hand each to TextEncoder; mixing UTF-16 code-unit counts with byte counts is where the off-by-one bugs live.

Mojibake: the decode-side failure

Suppose a UTF-8 string is decoded as Latin-1. The bytes for 你 (three bytes) become three separate characters. When displayed as UTF-8, the result looks like ä½ .

The repair steps are:

1. Confirm the original encoding. It is usually UTF-8.
2. Decode the Base64 to bytes.
3. Decode the bytes with the correct character set.
4. If the bytes were already damaged by a lossy conversion, the original text may be unrecoverable.

// Wrong: assuming one byte per character
const broken = atob(encoded);

// Right: decode bytes as UTF-8
const fixed = base64DecodeUtf8(encoded);

For one-off inspection, paste the value into a Base64 encoder/decoder and compare the byte output with a UTF-8 encoder. If the byte sequences match but the text differs, the problem is on the decode side.

Repair is mechanical when the damage is a single mis-decode. Take the mojibake string, reinterpret its characters as Latin-1 bytes, and decode those bytes as UTF-8: Buffer.from(mojibake, "latin1").toString("utf8"). The string ä½  becomes 你 if and only if no byte was replaced with ? along the way. A question mark, a U+FFFD, or a charset conversion that dropped bytes destroys the information, and no later step can recover it.

Count the layers. UTF-8 decoded as Latin-1 once yields ä½ ; twice yields ä½Â. Every extra layer means another round of reinterpretation and another chance that some tool already normalized the text. Find the producer that declared the wrong charset instead of stacking repair functions, or the next payload will break the same way.

The same bytes decoded as GBK, Shift_JIS, or Windows-1252 produce different-looking garbage. Guessing the source encoding from appearance is unreliable; the Content-Type charset, the database connection charset, and the file's declared encoding are the evidence to check first.

URL-safe Base64

Standard Base64 uses + and /, which have special meanings in URLs. URL-safe Base64 uses - and _ instead and often drops the = padding.

function toUrlSafe(base64) {
  return base64.replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
}

function fromUrlSafe(value) {
  const base64 = value.replace(/-/g, "+").replace(/_/g, "/");
  return base64.padEnd(Math.ceil(base64.length / 4) * 4, "=");
}

JWT uses URL-safe Base64 without padding for its header and payload. This is why a JWT segment may not decode with a strict standard-Base64 decoder. The JWT decoder handles this automatically.

A URL-safe value still needs care when it travels in a query string. URL-encoded form parsing turns + into a space, but URL-safe Base64 contains no +; the reverse hazard is padding = and, in some linkifiers, _ handling. The practical rule is to treat the segment as opaque: parse it raw, avoid round-tripping it through URLSearchParams, and compare at the byte level.

Padding arithmetic has one invalid state. A Base64 string whose length is one more than a multiple of four cannot be produced by any byte sequence. padEnd will append three = characters, but a strict decoder should still reject the value. If you generate identifiers, ensure the encoder's output length is never 4n+1 before stripping padding.

JWT is the most common place developers meet URL-safe Base64, but not the only one. JWK fields such as n, e, and d, and the OAuth PKCE code_challenge, use the same alphabet. A helper written for one of them works for the others as long as it tolerates missing padding.

Base64 is not encryption

Base64 is reversible by anyone. It provides no confidentiality and no integrity. If you need to protect data:

• Use AES-GCM for authenticated encryption.
• Use HTTPS for transport.
• Store secrets in a secret manager, not in encoded strings.

The AES-GCM tool can help you test an encryption workflow, but production key management belongs in your infrastructure, not in a browser tab.

Base64 has a measurable cost. The output is 4/3 the input byte length, plus line breaks if the producer inserts them; a 10 MB upload becomes 13.3 MB on the wire, and the raw and encoded buffers coexist in memory during conversion. For large files, multipart upload or a presigned object-storage URL avoids the expansion. For a short JSON field, the cost is irrelevant.

Base64 also leaks. A value placed in a URL ends up in browser history, proxy logs, and Referer headers, and Base64 does not make it unreadable. Putting a token or an email address in a query parameter is a privacy decision, not an encoding decision.

Common mistakes checklist

Verify with a known vector

A useful test vector is the UTF-8 encoding of 你好:

text:   你好
UTF-8:  E4 BD A0 E5 A5 BD
Base64: 5L2g5aW9

If your function produces 5L2g5aW9, the UTF-8 pipeline is correct. If it produces something else, inspect the intermediate bytes. The hex encoder shows the bytes directly, which makes the mismatch obvious.

Base64 is simple once you keep the layers separate: text to bytes, bytes to Base64, and the reverse. Most "Base64 bugs" are really text-encoding bugs wearing a different label.

A five-case round-trip suite catches nearly every regression:

1. hello encodes to aGVsbG8=.
2. é encodes to w6k=; a result of 6Q== means the UTF-8 step was skipped.
3. 你好 encodes to 5L2g5aW9.
4. 🚀 encodes to 8J+agA==; a different value usually means a surrogate was encoded separately.
5. A byte sequence whose Base64 contains + and / survives toUrlSafe and fromUrlSafe unchanged after normalization.

If one case fails, compare the intermediate bytes instead of the Base64 strings; the hex encoder linked earlier in this guide shows exactly where the pipeline diverged.

Use Base64 when a text-only channel must carry bytes: a JSON field, an email attachment, a small data URL. Do not use it for compression, secrecy, or large binary transfers where multipart and object storage exist. It is an adapter between bytes and text, and nothing more.