TypeScript/JavaScript lexer.
Disambiguation rests on three rules: previous-token tracking separates a
regex literal from division, template literals nest their ${…}
interpolations, and matching is position-major (a string wins over a comment
that opens inside it).
Nesting (template interpolations, generic argument lists, : type
annotations) runs on an explicit pooled frame stack, so arbitrarily deep
input tokenizes fully without touching the JS call stack.
Type positions: a : is an annotation or an operator by where it is. The
machine tracks the bracket level it is in (block, object literal, parens,
array, class body) and each level's ?s still waiting for a :, so
(x: T), let x: T; and ): T { are annotations and {a: b}, a ? b : c,
case x: and labels are not. The type then lexes in type mode and ends at
the first token that can't continue it. A type alias's body and the tail of
an as assertion lex the same way. It is a heuristic over tokens, not a
parse: a name alone on its line before a : reads as a label.
Resilience: unterminated strings extend to end of line; unterminated
templates, block comments, and interpolations extend to end of window.
Interpolation bodies discover their own closing } during real tokenization
(nothing is prescanned), so constructs that contain a } — regex literals,
strings, comments — never end an interpolation early. The tradeoff is that
damage propagates rather than being contained: a malformed interior that
consumes the closing } (say an unterminated block comment) extends the
interpolation past it, editor-style.
SyntaxLang import {lexer_ts} from '@fuzdev/fuz_code/lexer_ts.js'; The TypeScript language registration for the lexer engine.
JavaScript reuses this lexer via the js/javascript aliases: TypeScript is
a syntactic superset, and the TS-only constructs the lexer recognizes (type
annotations, as, generics before a call) can't appear in valid JS, so
running the full TS lexer on JS is a no-op for those paths. There is no
separate JS lexer.