joetjen.net
EN DE
Grammar language

Grammar

Both implementations read .aether files with a hand-written lexer and reader. The grammar below states the same syntax in Aether itself, and both test their readers against it.

Ichor 0.7

Pragmas and tokens

This grammar was written for these pages from Aether.Lexer and Aether.Reader in Ichor 0.3.0. Since Ichor 0.3.1 the same grammar, character for character, is part of Ichor’s documentation as guides/aether/AETHER_GRAMMAR.md, and the Praxis port carries it in its tutorial; the text below is quoted from Ichor 0.7.3. In each, a test compiles it and checks that it accepts exactly what the reader accepts, on every .aether file in the repository and on a set of broken ones.

Checked again with Ichor 0.7.3, it agrees with Ichor’s reader on all 55 Aether grammars in the Ichor, Cooper, dextrin, logos, xeger, aletheia and Praxis repositories.

aether.aether
@grammar "aether"
@root grammar_file
@skip TRIVIA

COMMENT := ";" (!"\n" .)*
TRIVIA  := (SPACE | COMMENT)*

UPPER_NAME := [A-Z_] [A-Z0-9_]*
LOWER_NAME := [a-z] [a-z0-9_\-]*
NUMBER     := DIGIT+

ESCAPE     := "\\" ("x" HEX HEX | "u{" HEX+ "}" | [nrt] | [^a-zA-Z0-9])
STRING     := "\"" (ESCAPE | !"\"" .)* "\"" (("cs" | "i") ![a-zA-Z0-9_] | !"cs" !"i")
POSIX      := "[:" ("alpha" | "alnum" | "digit" | "space" | "hex") ":]"
CHAR_CLASS := "[" "^"? (POSIX | ESCAPE | !"]" .)* "]"
REGEX      := "/" ("\\" . | "[" ("\\" . | !"]" .)* "]" | ![/\[] .)* "/"

Header and definitions

grammar_file := header definition*

header := "@grammar" name:STRING "@root" root:LOWER_NAME pragma*
pragma := "@skip" UPPER_NAME | "@noskip" | "@case_insensitive" | "@engine" LOWER_NAME

definition   := token_def | rule_def | keywords_def
token_def    := UPPER_NAME ":=" choice refine?
rule_def     := LOWER_NAME ":=" choice
keywords_def := "@keywords" UPPER_NAME "{" keyword ("," keyword)* "}"
keyword      := STRING "->" UPPER_NAME
refine       := "@refine" "(" STRING "," STRING ("," UPPER_NAME)* ")"

sequence stops in front of NAME :=. That lookahead is the whole reason a definition needs no terminator.

grammar_file has no TRIVIA? at either end, though a file may begin with a comment and end with a newline: every engine drops skip tokens before the first token and after the last. Up to Ichor 0.5 the grammar, and these pages, still had TRIVIA? header definition* TRIVIA?; Ichor 0.6 dropped it. Both forms agree with the reader on the same 55 files.

Expressions

choice     := sequence ("|" sequence)*
sequence   := (!def_head item)+
def_head   := (UPPER_NAME | LOWER_NAME) ":="
item       := "~"? term
term       := (LOWER_NAME ":")? postfix
postfix    := ("&" | "!") primary | primary quantifier?
quantifier := "*" | "+" | "?" | "{" NUMBER ("," NUMBER?)? "}"
primary    := STRING | CHAR_CLASS | REGEX | "." | UPPER_NAME | LOWER_NAME
            | "(" choice ")" | layout | native

layout     := ("@indent" | "@samecol") ("(" choice ")" | postfix)
native     := "@native" "(" STRING "," STRING ("," LOWER_NAME)* ")" hint?
hint       := "@hint" "(" hint_entry ("," hint_entry)* ")"
hint_entry := LOWER_NAME ":" (LOWER_NAME | "(" (LOWER_NAME ("," LOWER_NAME)*)? ")")

Beyond syntax

Both readers check more than this grammar can express. They also reject:

  • a character class, . or /regex/ in a rule, and a rule name in a token;
  • a name: capture in a token;
  • ~ anywhere but directly before a bare name, in a token, or under @noskip;
  • a pragma given twice, @skip together with @noskip, and an @engine other than peg, lr or glr;
  • a name declared twice, and both @keywords and @refine on one token;
  • in @hint, keys other than nullable and leading, leading: on a token, and a leading: name that is not a dependency.

Undefined references, redeclared predefined tokens and left recursion are checked once the whole grammar has been read.

Where the implementation differs

Since Ichor 0.6, in nothing these pages know of. Up to Ichor 0.5.1 a table here listed one difference: the reference said both backends run every @engine, while Grammar.VM runs only peg and refuses the rest. That is still how Ichor behaves, and since 0.6.0 the reference says so — interpreted, Grammar.LR and Grammar.GLR run the other two; generated, the parser takes its engine from @engine, see engines.

Up to Ichor 0.3.0 the table had a second row: a capture in a token body was accepted and then crashed the matcher. Since 0.3.1 the reader rejects it, as the reference always said.