Native API: Wtf8Decoder drops U+FEFF from string literals and template text #65458

Search terms

Wtf8Decoder, ignoreBOM, U+FEFF, BOM, string literal, template literal, native API.

Version and regression information

Reproduces with both sync and async APIs in typescript@7.0.2, 7.1.0-dev.20261110.1, and today's typescript@next, 7.1.0-dev.20261112.1. Tested with Node.js v24.21.0. The first affected version is unknown.

Code

Install typescript@7.1.0-dev.20261112.1, save this as repro.mjs in that directory, and run node repro.mjs. The script creates input.ts and tsconfig.json. No other dependencies are needed.

import { API } from "typescript/async";
import { writeFileSync } from "node:fs";
import { resolve } from "node:path";

writeFileSync("input.ts", [
  String.raw`const leading = "\uFEFFx";`,
  String.raw`const only = "\uFEFF";`,
  String.raw`const afterSurrogate = "\uD800\uFEFF";`,
  String.raw`const middle = "a\uFEFFb";`,
  'const template = `\\uFEFFtail`;',
].join("\n"));
writeFileSync("tsconfig.json", JSON.stringify({
  compilerOptions: { noEmit: true, noLib: true, types: [] },
  files: ["input.ts"],
}));
const config = resolve("tsconfig.json");
const api = new API({ cwd: process.cwd() });
try {
  const snapshot = await api.createSnapshot({ openProjects: [config] });
  const project = snapshot.getConfiguredProject(config);
  const file = await project.program.getSourceFile(resolve("input.ts"));
  for (const statement of file.statements) {
    const declaration = statement.declarationList.declarations[0];
    const units = Array.from(declaration.initializer.text,
      c => c.codePointAt(0).toString(16));
    console.log(declaration.name.text, JSON.stringify(units));
  }
  await snapshot.dispose();
} finally {
  await api.close();
}

The input file contains no file-level BOM. Its literal escape sequences produce U+FEFF inside the literal values.

Actual behavior

leading ["78"]
only []
afterSurrogate ["d800"]
middle ["61","feff","62"]
template ["74","61","69","6c"]

U+FEFF is lost at the start of string and template values. It is also lost after a lone surrogate, while an ordinary middle occurrence is preserved.

Expected behavior

leading ["feff","78"]
only ["feff"]
afterSurrogate ["d800","feff"]
middle ["61","feff","62"]
template ["feff","74","61","69","6c"]

AST literal values should retain these characters. Removing them changes the data returned to compiler API consumers.

Additional information

The published API clients construct new Wtf8Decoder() without ignoreBOM: true, including the source-file decoder in dist/api/node/node.js and the decoder instances in dist/api/async/api.js and dist/api/sync/api.js.

Wtf8Decoder inherits TextDecoder's default BOM handling. Its handling of lone surrogates decodes the surrounding segments separately, which explains the loss after a surrogate too. Constructing payload decoders with { ignoreBOM: true } preserves U+FEFF; the WASM transport already uses that option.

Related: #63850 discusses source-file text and positions around an initial BOM. This reproduction has no initial BOM and also demonstrates corruption of string and template values.