Grammar language

Escape hatches

Some lexical decisions depend on more than the longest match, and some languages change their own syntax as they go. Three pragmas cover these cases, each naming exactly what it touches.

Ichor 0.3

Keywords

@keywords renames a token after lexing when its text is in a table. The new names need no declaration of their own:

WORD := [a-zA-Z_][a-zA-Z0-9_]*
@keywords WORD { "if" -> KEYWORD_IF, "return" -> KEYWORD_RETURN }

Every WORD spelled if reaches the parser as KEYWORD_IF; every other WORD stays a WORD. It is the table-driven alternative to declaring each keyword as a token ahead of the identifier.

Refining a token

@refine attaches code to a token definition. After the module and function name, it lists every name the token may become; the function sees the token’s text, its position and all tokens before it. That is what tells JavaScript’s / as division from the start of a regex literal:

SLASH := "/" @refine("JS.SlashDisambiguator", "refine", DIV, REGEX_START)

A token may have one @keywords table or one @refine, not both.

Native tokens and rules

@native("Module", "function", dep, …) replaces the body of one token or rule with hand-written code — for what no fixed grammar can say: heredocs whose terminator is read from the input, string interpolation, operator tables that change while a file is read. The names after the function are the only rules the code may call back into, so the grammar’s own checks still see them.

RESOLVER_REF_RAW := @native("Cooper.Native.ResolverRef", "scan")

expr := @native("Prolog.Grammar", "parse_term", primary)
        @hint(nullable: false, leading: (primary))

The first is from CASC: a !{…} reference may nest braces, which no fixed token can count. A token-level @native must be the whole definition. @hint gives the facts the grammar cannot work out from code: nullable: whether it can match nothing, and, for rules, leading: which dependencies can come first.