tokenize_syntax.ts view source
(text: string, grammar: SyntaxGrammar): SyntaxTokenStream Accepts a string of text as input and the language definitions to use, and returns an array with the tokenized code.
When the language definition includes nested tokens, the function is called recursively on each of these tokens.
This method could be useful in other contexts as well, as a very crude parser.
text
- a string with the code to be styled
type
stringgrammar
- an object containing the tokens to use
Usually a language definition like syntax_styler.get_lang('markup').
type SyntaxGrammar
returns
SyntaxTokenStream an array of strings and tokens, a token stream