api #

syntax styling utilities and components for TypeScript, Svelte, Markdown, and more

25 modules · 46 declarations

Modules
#

add_grammar_bash
#

grammar_bash.ts view source

(syntax_styler: SyntaxStyler): void import {add_grammar_bash} from '@fuzdev/fuz_code/grammar_bash.js';

Bash/shell grammar for syntax highlighting.

Standalone grammar (no base extension). Covers core bash syntax: comments, strings, variables, functions, keywords, builtins, operators, and redirections.

Based on Prism (https://github.com/PrismJS/prism) by Lea Verou (https://lea.verou.me/)

MIT license

syntax_styler

type SyntaxStyler

returns

void

see also

  • LICENSE

add_grammar_clike
#

add_grammar_css
#

add_grammar_js
#

add_grammar_json
#

add_grammar_markdown
#

grammar_markdown.ts view source

(syntax_styler: SyntaxStyler): void import {add_grammar_markdown} from '@fuzdev/fuz_code/grammar_markdown.js';

Markdown grammar extending markup. Supports: headings, fenced code blocks (3/4/5 backticks with nesting), lists, blockquotes, bold, italic, strikethrough, inline code, and links.

syntax_styler

type SyntaxStyler

returns

void

add_grammar_markup
#

add_grammar_svelte
#

add_grammar_ts
#

AddSyntaxGrammar
#

syntax_styler.ts view source

AddSyntaxGrammar import type {AddSyntaxGrammar} from '@fuzdev/fuz_code/syntax_styler.js';

(call)

type (syntax_styler: SyntaxStyler): void

syntax_styler

type SyntaxStyler
returns void

class_keywords
#

grammar_clike.ts view source

"class|extends|implements|instanceof|interface|new" import {class_keywords} from '@fuzdev/fuz_code/grammar_clike.js';

Code
#

Code.svelte view source

accepts children

import Code from '@fuzdev/fuz_code/Code.svelte';

lang?

type string
optional default 'svelte'

children?

type Snippet<[]> & Snippet<[markup: string]>
optional

content?

The source code to syntax highlight.

type string
optional

dangerous_raw_html?

type string
optional

grammar?

Optional custom SyntaxGrammar object for syntax tokenization.

When to use:

  • To provide a custom language definition not registered in syntax_styler.langs
  • To use a modified/extended version of an existing grammar
  • For one-off grammar variations without registering globally

Behavior:

  • When provided, this grammar is used for tokenization instead of looking up via lang
  • Enables highlighting even if lang is not in the registry (useful for custom languages)
  • The lang parameter is still used for metadata (data-lang attribute)
  • When undefined, the grammar is automatically looked up via syntax_styler.get_lang(lang)
type SyntaxGrammar
optional default undefined (uses grammar from `syntax_styler.langs[lang]`)

inline?

Whether to render as inline code or block code. Controls display via CSS classes.

type boolean
optional default false

wrap?

Whether to wrap long lines in block code. Sets white-space: pre-wrap instead of white-space: pre.

Behavior:

  • Wraps at whitespace (spaces, newlines)
  • Long tokens without spaces (URLs, hashes) will still scroll horizontally
  • Default false provides traditional code block behavior

Only affects block code (ignored for inline mode).

type boolean
optional default false

nomargin?

Whether to disable the default margin-bottom on block code. Block code has margin-bottom: var(--space_lg) by default when not :last-child.

type boolean
optional default false

syntax_styler?

Custom SyntaxStyler instance to use for highlighting. Allows using a different styler with custom grammars or configuration.

type SyntaxStyler
optional default syntax_styler_global

intersects

SvelteHTMLElements['code']

CodeHighlight
#

CodeHighlight.svelte view source

accepts children

import CodeHighlight from '@fuzdev/fuz_code/CodeHighlight.svelte';

lang?

type string
optional default 'svelte'

children?

type Snippet<[]> & Snippet<[markup: string]>
optional

content

The source code to syntax highlight.

type string

mode?

Highlighting mode for this component.

Options:

  • 'auto' - Uses CSS Custom Highlight API if supported, falls back to HTML mode
  • 'ranges' - Forces CSS Custom Highlight API (requires browser support)
  • 'html' - Forces HTML generation with CSS classes

Note: CSS Custom Highlight API has limitations and limited browser support. Requires importing theme_highlight.css instead of theme.css.

type HighlightMode
optional default 'auto'

grammar?

Optional custom SyntaxGrammar object for syntax tokenization.

When to use:

  • To provide a custom language definition not registered in syntax_styler.langs
  • To use a modified/extended version of an existing grammar
  • For one-off grammar variations without registering globally

Behavior:

  • When provided, this grammar is used for tokenization instead of looking up via lang
  • Enables highlighting even if lang is not in the registry (useful for custom languages)
  • The lang parameter is still used for metadata (data-lang attribute)
  • When undefined, the grammar is automatically looked up via syntax_styler.get_lang(lang)
type SyntaxGrammar
optional default undefined (uses grammar from `syntax_styler.langs[lang]`)

inline?

Whether to render as inline code or block code. Controls display via CSS classes.

type boolean
optional default false

wrap?

Whether to wrap long lines in block code. Sets white-space: pre-wrap instead of white-space: pre.

Behavior:

  • Wraps at whitespace (spaces, newlines)
  • Long tokens without spaces (URLs, hashes) will still scroll horizontally
  • Default false provides traditional code block behavior

Only affects block code (ignored for inline mode).

type boolean
optional default false

syntax_styler?

Custom SyntaxStyler instance to use for highlighting. Allows using a different styler with custom grammars or configuration.

type SyntaxStyler
optional default syntax_styler_global

intersects

SvelteHTMLElements['code']

CodeSample
#

code_sample.ts view source

CodeSample import type {CodeSample} from '@fuzdev/fuz_code/code_sample.js';

name

type string

lang

type string

content

type string

CodeTextarea
#

CodeTextarea.svelte view source

accepts children

import CodeTextarea from '@fuzdev/fuz_code/CodeTextarea.svelte';

value?

type string | (string[] & string)
optional bindable default ''

lang?

type string
optional default 'svelte'

grammar?

Optional custom grammar; takes precedence over lang for tokenization.

type SyntaxGrammar
optional

syntax_styler?

Custom SyntaxStyler instance (defaults to the global one).

type SyntaxStyler
optional default syntax_styler_global

wrapper_attrs?

Attributes for the wrapper <div> — the layout box that the textarea fills and resize grows. Use it for sizing/layout classes, style, id, or container-level handlers. Its class is merged with the internal code_textarea class; data-lang stays component-controlled. (...rest spreads onto the <textarea>; the backdrop <pre> is internal and intentionally not exposed.)

type HTMLAttributes<HTMLDivElement>
optional

intersects

SvelteHTMLElements['textarea']

create_range_highlighting
#

range_highlighting.svelte.ts view source

(options: RangeHighlightingOptions): RangeHighlighting import {create_range_highlighting} from '@fuzdev/fuz_code/range_highlighting.svelte.js';

Wires up CSS Custom Highlight API range highlighting for a single element's text node, shared by CodeHighlight and CodeTextarea. Creates a HighlightManager, memoizes tokenization, applies/clears ranges in an effect, emits DEV warnings for unsupported languages, and tears down on destroy.

Must be called during component initialization (it uses $effect/onDestroy).

options

type RangeHighlightingOptions

returns

RangeHighlighting

grammar_markup_add_attribute
#

grammar_markup.ts view source

(syntax_styler: SyntaxStyler, attr_name: string, lang: string): void import {grammar_markup_add_attribute} from '@fuzdev/fuz_code/grammar_markup.js';

Adds a pattern to style languages embedded in HTML attributes.

An example of an inlined language is CSS with style attributes.

syntax_styler

the SyntaxStyler instance to modify

type SyntaxStyler

attr_name

the name of the attribute that contains the inlined language, treated as case insensitive

type string

lang

the language key

type string

returns

void

grammar_markup_add_inlined
#

grammar_markup.ts view source

(syntax_styler: SyntaxStyler, tag_name: string, lang: string, inside_lang?: string): void import {grammar_markup_add_inlined} from '@fuzdev/fuz_code/grammar_markup.js';

Adds an inlined language to markup.

An example of an inlined language is CSS with <style> tags.

syntax_styler

the SyntaxStyler instance to modify

type SyntaxStyler

tag_name

the name of the tag that contains the inlined language, treated as case insensitive

type string

lang

the language key

type string

inside_lang

the language to insert into, defaults to 'markup'

type string
default 'markup'

returns

void

grammar_svelte_add_inlined
#

grammar_svelte.ts view source

(syntax_styler: SyntaxStyler, tag_name: string, lang: string): void import {grammar_svelte_add_inlined} from '@fuzdev/fuz_code/grammar_svelte.js';

syntax_styler

type SyntaxStyler

tag_name

type string

lang

type string

returns

void

highlight_priorities
#

HighlightManager
#

highlight_manager.ts view source

import {HighlightManager} from '@fuzdev/fuz_code/highlight_manager.js';

Manages CSS Custom Highlight API ranges for a single element's text node. Tracks ranges per element and only removes its own ranges when clearing, cooperating with other managers that share the global CSS.highlights registry.

Experimental — limited browser support. Use Code.svelte for production block code; this powers the experimental CodeHighlight and CodeTextarea.

examples

const manager = new HighlightManager(); manager.highlight_from_syntax_tokens(element, tokens);

element_ranges

This manager's ranges, keyed by prefixed highlight name (e.g. token_keyword). A single range object may be shared across several names (a token type plus its aliases), since one range can belong to multiple Highlight sets.

type Map<string, Array<AbstractRange>>

constructor

type new (): HighlightManager

highlight_from_syntax_tokens

Highlights element's text node from a SyntaxTokenStream produced by tokenize_syntax. Clears this manager's previous ranges first.

In production this never throws on a tokenizer/DOM mismatch: out-of-bounds tokens are clamped and a missing text node is a no-op. In DEV the same conditions throw loudly to surface grammar bugs.

type (element: Element, tokens: SyntaxTokenStream): void

element

type Element

tokens

type SyntaxTokenStream
returns void

clear_element_ranges

Clears only this manager's ranges from the shared highlights. Defensive: a highlight may already be gone (e.g. another manager removed the last range, or HMR reset the registry), which is a valid state, not an error.

type (): void

returns void

destroy

type (): void

returns void

HighlightMode
#

HighlightTokenName
#

HookAfterTokenizeCallback
#

HookAfterTokenizeCallbackContext
#

HookBeforeTokenizeCallback
#

HookBeforeTokenizeCallbackContext
#

syntax_styler.ts view source

HookBeforeTokenizeCallbackContext import type {HookBeforeTokenizeCallbackContext} from '@fuzdev/fuz_code/syntax_styler.js';

code

type string

grammar

type SyntaxGrammar

lang

type string

tokens

type undefined

HookWrapCallback
#

HookWrapCallbackContext
#

syntax_styler.ts view source

HookWrapCallbackContext import type {HookWrapCallbackContext} from '@fuzdev/fuz_code/syntax_styler.js';

type

type string

content

type string

tag

type string

classes

type Array<string>

attributes

type Record<string, string>

lang

type string

PreprocessFuzCodeOptions
#

svelte_preprocess_fuz_code.ts view source

PreprocessFuzCodeOptions import type {PreprocessFuzCodeOptions} from '@fuzdev/fuz_code/svelte_preprocess_fuz_code.js';

exclude?

File patterns to exclude.

type Array<string | RegExp>

syntax_styler?

Custom SyntaxStyler instance.

type SyntaxStyler

default syntax_styler_global

cache?

Enable in-memory caching.

type boolean

default true

component_imports?

Import sources that resolve to the Code component. Used to verify that <Code> in templates actually refers to Code.svelte.

type Array<string>

default ['@fuzdev/fuz_code/Code.svelte']

on_error?

How to handle errors.

type 'log' | 'throw'

default 'throw' in CI, 'log' otherwise

RangeHighlighting
#

RangeHighlightingOptions
#

range_highlighting.svelte.ts view source

RangeHighlightingOptions import type {RangeHighlightingOptions} from '@fuzdev/fuz_code/range_highlighting.svelte.js';

Reactive inputs for create_range_highlighting. All values are getters so the helper can track the consuming component's reactive state across the call boundary (the Svelte 5 getter-injection pattern).

element

The element whose first text node receives the highlight ranges.

type () => Element | undefined

text

The text to tokenize. Must match the element's text node exactly (e.g. a textarea backdrop includes its trailing newline here too).

type () => string

lang

Language id; null disables highlighting.

type () => string | null

grammar

Optional custom grammar; takes precedence over lang for tokenization.

type () => SyntaxGrammar | undefined

syntax_styler

The syntax styler whose registered grammars back lang lookups.

type () => SyntaxStyler

enabled?

Extra gate — ranges are only applied when this returns true. Defaults to always-on.

type () => boolean

dev_label

Component name used in DEV warnings.

type string

sample_langs
#

code_sample.ts view source

readonly ["json", "css", "ts", "html", "svelte", "md", "bash"] import {sample_langs} from '@fuzdev/fuz_code/code_sample.js';

SampleLang
#

code_sample.ts view source

"html" | "json" | "css" | "ts" | "svelte" | "md" | "bash" import type {SampleLang} from '@fuzdev/fuz_code/code_sample.js';

supports_css_highlight_api
#

highlight_manager.ts view source

(): boolean import {supports_css_highlight_api} from '@fuzdev/fuz_code/highlight_manager.js';

Checks for CSS Highlights API support.

returns

boolean

svelte_preprocess_fuz_code
#

svelte_preprocess_fuz_code.ts view source

(options?: PreprocessFuzCodeOptions): PreprocessorGroup import {svelte_preprocess_fuz_code} from '@fuzdev/fuz_code/svelte_preprocess_fuz_code.js';

Svelte preprocessor that compiles static Code component content at build time, replacing runtime syntax highlighting with pre-rendered HTML.

options

type PreprocessFuzCodeOptions
default {}

returns

PreprocessorGroup

a Svelte preprocessor group

examples

// svelte.config.js import {svelte_preprocess_fuz_code} from '@fuzdev/fuz_code/svelte_preprocess_fuz_code.js'; export default { preprocess: [svelte_preprocess_fuz_code(), vitePreprocess()], };

syntax_styler_global
#

syntax_styler_global.ts view source

SyntaxStyler import {syntax_styler_global} from '@fuzdev/fuz_code/syntax_styler_global.js';

Pre-configured SyntaxStyler instance with all built-in grammars registered.

examples

import {syntax_styler_global} from '@fuzdev/fuz_code/syntax_styler_global.js'; const html = syntax_styler_global.stylize('const x = 1;', 'ts');

SyntaxGrammar
#

syntax_styler.ts view source

SyntaxGrammar import type {SyntaxGrammar} from '@fuzdev/fuz_code/syntax_styler.js';

A grammar after normalization. All values are arrays of normalized SyntaxGrammarToken with consistent shapes.

[key: string]

type SyntaxGrammarToken[]

SyntaxGrammarRaw
#

SyntaxGrammarToken
#

syntax_styler.ts view source

SyntaxGrammarToken import type {SyntaxGrammarToken} from '@fuzdev/fuz_code/syntax_styler.js';

Grammar token with all properties required. This is the normalized representation of SyntaxGrammarTokenRaw used at runtime.

pattern

type RegExp

lookbehind

type boolean

greedy

type boolean

alias

type Array<string>

inside

type SyntaxGrammar | null

SyntaxGrammarTokenRaw
#

syntax_styler.ts view source

SyntaxGrammarTokenRaw import type {SyntaxGrammarTokenRaw} from '@fuzdev/fuz_code/syntax_styler.js';

The expansion of a simple RegExp literal to support additional properties.

The inside grammar will be used to tokenize the text value of each token of this kind.

This can be used to make nested and even recursive language definitions.

Note: This can cause infinite recursion. Be careful when you embed different languages or even the same language into each another.

Note: Grammar authors can use optional properties, but they will be normalized to required properties at registration time for optimal performance.

pattern

The regular expression of the token.

type RegExp

lookbehind?

If true, then the first capturing group of pattern will (effectively) behave as a lookbehind group meaning that the captured text will not be part of the matched text of the new token.

type boolean

default false

greedy?

Whether the token is greedy.

type boolean

default false

alias?

An optional alias or list of aliases.

type string | Array<string>

inside?

The nested SyntaxGrammarRaw of this token.

type SyntaxGrammarRaw | null

SyntaxGrammarValueRaw
#

SyntaxStyler
#

syntax_styler.ts view source

import {SyntaxStyler} from '@fuzdev/fuz_code/syntax_styler.js';

Based on Prism (https://github.com/PrismJS/prism) by Lea Verou (https://lea.verou.me/)

MIT license

see also

  • LICENSE

langs

type Record<string, SyntaxGrammar | undefined>

add_lang

type (id: string, grammar: SyntaxGrammarRaw, aliases?: string[] | undefined): void

id

type string

grammar

type SyntaxGrammarRaw

aliases?

type string[] | undefined
optional
returns void

add_extended_lang

type (base_id: string, extension_id: string, extension: SyntaxGrammarRaw, aliases?: string[] | undefined): SyntaxGrammar

base_id

type string

extension_id

type string

extension

type SyntaxGrammarRaw

aliases?

type string[] | undefined
optional
returns SyntaxGrammar

get_lang

type (id: string): SyntaxGrammar

id

type string
returns SyntaxGrammar

stylize

Generates HTML with syntax highlighting from source code.

Process:

  1. Runs before_tokenize hook
  2. Tokenizes code using the provided or looked-up grammar
  3. Runs after_tokenize hook
  4. Runs wrap hook on each token
  5. Converts tokens to HTML with CSS classes

Parameter Relationship:

  • lang is ALWAYS required for hook context and identification
  • grammar is optional; when undefined, automatically looks up via this.get_lang(lang)
  • When both are provided, grammar is used for tokenization, lang for metadata

Use cases:

  • Standard usage: stylize(code, 'ts') - uses registered TypeScript grammar
  • Custom grammar: stylize(code, 'ts', customGrammar) - uses custom grammar but keeps 'ts' label
  • Extended grammar: stylize(code, 'custom', this.extend_grammar('ts', extension)) - new language variant

type (text: string, lang: string, grammar?: SyntaxGrammar | undefined): string

text

the source code to syntax highlight

type string

lang

language identifier (e.g., 'ts', 'css', 'html'), used for:

  • grammar lookup when grammar is undefined
  • hook context (lang field passed to hooks)
  • language identification in output
type string

grammar

optional custom SyntaxGrammar object; when undefined, automatically looks up the grammar via this.get_lang(lang); provide this to use a custom or modified grammar instead of the registered one

type SyntaxGrammar | undefined
default this.get_lang(lang)
returns string

HTML string with syntax highlighting using CSS classes (.token_*)

// Standard usage - uses registered grammar stylize('var foo = true;', 'ts');
// Custom grammar - overrides registered grammar const customGrammar = { keyword: [...], string: [...] }; stylize('var foo = false;', 'ts', customGrammar);
// Extended grammar - builds on existing grammar const extended = this.extend_grammar('ts', { customToken: [...] }); stylize('var foo = 42;', 'ts-extended', extended);

tokenize

Tokenizes text into a SyntaxTokenStream, running the before_tokenize and after_tokenize hooks. This is the tokenization half of stylize — use it when you need the token stream itself (e.g. CSS Custom Highlight API range highlighting) rather than HTML.

type (text: string, lang: string, grammar?: SyntaxGrammar | undefined): SyntaxTokenStream

text

source to tokenize

type string

lang

language identifier; passed to the tokenize hooks

type string

grammar

grammar to tokenize with; defaults to this.get_lang(lang)

type SyntaxGrammar | undefined
default this.get_lang(lang)
returns SyntaxTokenStream

the resulting token stream

grammar_insert_before

Inserts tokens before another token in a language definition or any other grammar.

Usage

This helper method makes it easy to modify existing languages. For example, the CSS language definition not only defines CSS styling for CSS documents, but also needs to define styling for CSS embedded in HTML through <style> elements. To do this, it needs to modify syntax_styler.get_lang('markup') and add the appropriate tokens. However, syntax_styler.get_lang('markup') is a regular JS object literal, so if you do this:

syntax_styler.get_lang('markup').style = { // token };

then the style token will be added (and processed) at the end. grammar_insert_before allows you to insert tokens before existing tokens. For the CSS example above, you would use it like this:

grammar_insert_before('markup', 'cdata', { 'style': { // token } });

Special cases

If the grammars of inside and insert have tokens with the same name, the tokens in inside's grammar will be ignored.

This behavior can be used to insert tokens after before:

grammar_insert_before('markup', 'comment', { 'comment': syntax_styler.get_lang('markup').comment, // tokens after 'comment' });

Limitations

The main problem grammar_insert_before has to solve is iteration order. Since ES2015, the iteration order for object properties is guaranteed to be the insertion order (except for integer keys) but some browsers behave differently when keys are deleted and re-inserted. So grammar_insert_before can't be implemented by temporarily deleting properties which is necessary to insert at arbitrary positions.

To solve this problem, grammar_insert_before doesn't actually insert the given tokens into the target object. Instead, it will create a new object and replace all references to the target object with the new one. This can be done without temporarily deleting properties, so the iteration order is well-defined.

However, only references that can be reached from syntax_styler.langs or insert will be replaced. I.e. if you hold the target object in a variable, then the value of the variable will not change.

var oldMarkup = syntax_styler.get_lang('markup'); var newMarkup = grammar_insert_before('markup', 'comment', { ... }); assert(oldMarkup !== syntax_styler.get_lang('markup')); assert(newMarkup === syntax_styler.get_lang('markup'));

type (inside: string, before: string, insert: SyntaxGrammarRaw, root?: Record<string, any>): SyntaxGrammar

inside

the property of root (e.g. a language id in syntax_styler.langs) that contains the object to be modified

type string

before

the key to insert before

type string

insert

an object containing the key-value pairs to be inserted

type SyntaxGrammarRaw

root

the object containing inside, i.e. the object that contains the object to be modified; defaults to syntax_styler.langs

type Record<string, any>
default this.langs
returns SyntaxGrammar

the new SyntaxGrammar object

stringify_token

Converts the given token or token stream to an HTML representation.

Runs the wrap hook on each SyntaxToken.

type (o: string | SyntaxTokenStream | SyntaxToken, lang: string): string

o

the token or SyntaxTokenStream to be converted

type string | SyntaxTokenStream | SyntaxToken

lang

the name of current language

type string
returns string

HTML representation of the token or token stream

extend_grammar

Creates a deep copy of the language with the given id and appends the given tokens.

If a token in extension also appears in the copied language, then the existing token in the copied language will be overwritten at its original position.

Best practices

Since the position of overwriting tokens (token in extension that overwrite tokens in the copied language) doesn't matter, they can technically be in any order. However, this can be confusing to others that trying to understand the language definition because, normally, the order of tokens matters in the grammars.

Therefore, it is encouraged to order overwriting tokens according to the positions of the overwritten tokens. Furthermore, all non-overwriting tokens should be placed after the overwriting ones.

type (base_id: string, extension: SyntaxGrammarRaw): SyntaxGrammar

base_id

the id of the language to extend, must be a key in syntax_styler.langs

type string

extension

the new tokens to append

type SyntaxGrammarRaw
returns SyntaxGrammar

plugins

type Record<string, any>

hooks_before_tokenize

type Array<HookBeforeTokenizeCallback>

hooks_after_tokenize

type Array<HookAfterTokenizeCallback>

hooks_wrap

type Array<HookWrapCallback>

add_hook_before_tokenize

type (cb: HookBeforeTokenizeCallback): void

cb

type HookBeforeTokenizeCallback
returns void

add_hook_after_tokenize

type (cb: HookAfterTokenizeCallback): void

cb

type HookAfterTokenizeCallback
returns void

add_hook_wrap

type (cb: HookWrapCallback): void

cb

type HookWrapCallback
returns void

run_hook_before_tokenize

type (ctx: HookBeforeTokenizeCallbackContext): void

ctx

type HookBeforeTokenizeCallbackContext
returns void

run_hook_after_tokenize

type (ctx: HookAfterTokenizeCallbackContext): void

ctx

type HookAfterTokenizeCallbackContext
returns void

run_hook_wrap

type (ctx: HookWrapCallbackContext): void

ctx

type HookWrapCallbackContext
returns void

SyntaxToken
#

syntax_token.ts view source

import {SyntaxToken} from '@fuzdev/fuz_code/syntax_token.js';

type

The type of the token.

This is usually the key of a pattern in a SyntaxGrammar.

type string

content

The strings or tokens contained by this token.

This will be a SyntaxTokenStream if the pattern matched also defined an inside grammar.

type string | SyntaxTokenStream

alias

The alias(es) of the token. Always an array, even if empty or single value.

type Array<string>

length

type number

constructor

type new (type: string, content: string | SyntaxTokenStream, alias: string | string[] | undefined, matched_str?: string): SyntaxToken

type

type string

content

type string | SyntaxTokenStream

alias

type string | string[] | undefined

matched_str

type string
default ''

SyntaxTokenStream
#

syntax_token.ts view source

SyntaxTokenStream import type {SyntaxTokenStream} from '@fuzdev/fuz_code/syntax_token.js';

A token stream is an array of strings and SyntaxToken objects.

SyntaxTokenStream values have to fulfill a few properties that are assumed by most functions (mostly internal ones) that process them.

  1. No adjacent strings.
  2. No empty strings.

    The only exception here is the token stream that only contains the empty string and nothing else.

tokenize_syntax
#

tokenize_syntax.ts view source

(text: string, grammar: SyntaxGrammar): SyntaxTokenStream import {tokenize_syntax} from '@fuzdev/fuz_code/tokenize_syntax.js';

Accepts a string of text as input and the language definitions to use, and returns an array with the tokenized code.

When the language definition includes nested tokens, the function is called recursively on each of these tokens.

This method could be useful in other contexts as well, as a very crude parser.

text

a string with the code to be styled

type string

grammar

a SyntaxGrammar object containing the tokens to use. Usually a language definition like syntax_styler.get_lang('markup').

type SyntaxGrammar

returns

SyntaxTokenStream

a SyntaxTokenStream array of strings and tokens

examples

var code = `var foo = 0;`; var tokens = tokenize_syntax(code, SyntaxStyler.langs.js); for (var token of tokens) { if (token instanceof SyntaxToken && token.type === 'number') { console.log(`Found numeric literal: ${token.content}`); } }