lexer.ts

Returns the cached next occurrence of ch at or after from, re-probing with indexOf only when the cached position has fallen behind. PROBE_NOT_FOUND when the text has no further occurrence — a monotonic probe that keeps delimiter scans linear across a construct that is dense in ch.

Copies a lexed event stream out of the buffer lex_syntax reuses, sized to events_len, for a result kept past the next lex.

Returned by Lexer.embed_to_brace when it lexed nothing.

A \w word char — ASCII letters, digits, and _.

Lexes text with lang, returning the flat event stream.

The result is valid until the next lex_syntax call: every call writes its events into one reused buffer, which is also why events runs past events_len. Use the result right away, or keep a copy with copy_lexed_syntax. A buffer allocated per call would be off-heap memory, zero-filled up front and freed only at a later GC, which on small inputs costs more than the lexing.

The result of lexing: the source text plus its flat event stream.

Shared lexing context passed to language lex functions. Holds the text window, the event buffer, and the language registry for embedding.

Case-insensitive ASCII match of word (must be lowercase) at text[from..], via code | 0x20 folding — never allocates, unlike toLowerCase() comparisons.

What advance_probe returns when the text has no further occurrence: past the end of any string, since V8 caps string length below it. It is a small integer rather than Infinity so probe-cache fields, which start at -1, never change representation: a double stored there gives the cache a hidden class that only live caches hold, so a major GC between lex calls would discard every optimized lexer function that embeds it.

Renders a lexed event stream to HTML in one forward pass. Gap text is copy-escaped; token spans use the precomputed open tags.

The result is built by appending, which in V8 makes a rope the first read copies into one string, at a cost per piece. So the loop appends few pieces and flat ones: an open tag is a single string rather than the tree + would have left at intern time, which every span would otherwise carry into the rope, and a one-char leaf, the most common kind in code, appends its whole span from TokenTypeInfo.char_spans. Text is sliced without a scan while the next character needing an escape, tracked by monotonic probes, lies past it.

Finds the matching } for the { at i, skipping js-style strings, templates, and comments. Returns -1 when unbalanced within the window.

Scans an identifier starting at from (assumed to be an identifier start), returning the exclusive end index.

Returns the index of the next \n at or after i (exclusive end of the line's content, excluding a preceding \r), or end when there is none. Uses native indexOf — the fast path for line-oriented scans.

Skips a js-style quoted span from the quote at from (used inside balanced scans), returning the index after the closing quote. Unterminated '/" strings stop at the newline; templates (`` `) span lines.

Skips whitespace (including newlines) from from, returning the next non-space index.

Flattens a lexed event stream to SyntaxEventTokens in document order (containers precede their children).

A flattened token span, in document order with containers before their children. Used by fixtures, tests, and range building.

A lexer-based language registration.

Interns a token type into token_types_global — the zero-config authoring path used by the built-in lexers' module-load type constants.

The shared default registry — the single id space used by the built-in lexers' module-load token_type constants and by any SyntaxStyler not given its own registry. The type vocabulary is global by design, mirroring the global .token_* CSS namespace; per-registry isolation exists for fully-custom stylers and tests, whose lexers must intern into the same registry they're rendered against.

Interned metadata for a token type.

An id space of interned token types with precomputed CSS classes and HTML open tags. Ids are only meaningful against the registry that interned them — a lexed event stream resolves back through the registry stamped on its LexedSyntax.

Trims trailing whitespace from a [from, to) span, returning the new exclusive end.

Validates a lexed event stream's structural invariants, returning a list of human-readable issues (empty when valid): records well-formed, offsets monotonic and in-bounds, containers balanced.

A fixed table of ASCII words, each carrying an integer, that a span of text is looked up in without slicing it out. A lexer classifying every identifier through map.get(text.slice(start, end)) allocates and hashes a string per identifier; find compares char codes against the few words of the span's length and first char and returns the matching word's integer, so the lexer branches on a number. The integer is the lexer's to define: a kind, or a kind with flag bits beside it.

Builds a word→kind classification map from [kind, words] entries, where words is space-separated — the shared shape of the lexers' keyword tables.