Grammar
The DXN text format is defined by a PEG grammar written in Aether, the grammar language of Ichor. dextrin’s parser is generated from it rather than written by hand.
Pragmas and tokens
This is the grammar dextrin’s parser is generated from, priv/grammar/dxn.aether as released with dextrin 0.3.2, with its comments removed. Upper-case names are tokens, matched longest-first with ties going to the earlier declaration; lower-case names are rules, matched over the tokens by ordered choice.
@grammar "dxn"
@root document
@skip TRIVIA
SPACE := [ \t\r\n,]+
COMMENT := "#" (!"\n" .)* "\n"?
TRIVIA := (SPACE | COMMENT)*
NIL := "nil"
TRUE := "true"
FALSE := "false"
EXPONENT := [eE] [+-]? DIGIT+
FLOAT := ("-"? DIGIT+ (("." DIGIT+ EXPONENT?) | EXPONENT)) | "NaN" | "Infinity" | "-Infinity"
DECIMAL := "-"? DIGIT+ ("." DIGIT+)? "M"
RATIONAL := "-"? DIGIT+ "/" DIGIT+
INTEGER := "-"? DIGIT+
IDENT_START := [_\u{41}-\u{5A}\u{61}-\u{7A}\u{AA} … \u{30000}-\u{3134A}\u{31350}-\u{33479}]
IDENT_CONT := IDENT_START | [\u{30}-\u{39}\u{41}-\u{5A}\u{5F}\u{61}-\u{7A} … \u{E0100}-\u{E01EF}\-?!]
IDENTIFIER := IDENT_START IDENT_CONT* ("." IDENT_START IDENT_CONT*)* ("/" IDENT_START IDENT_CONT* ("." IDENT_START IDENT_CONT*)*)?
MAP_KEY := IDENTIFIER ":"
KEYWORD := ":" (IDENTIFIER | STRING)
AT_DISCARD := "@_"
AT := "@"
ESCAPE := "\\" ("\"" | "\\" | "n" | "t" | "r" | "0" | "a" | "b" | "f" | "v" | ("x{" HEX+ "}"))
STRING := "\"" (ESCAPE | (![\"\\] .))* "\""
CHAR := "?" (ESCAPE | "s" | (![ \t\r\n] .))Declaration order does real work here. NIL, TRUE and FALSE come before IDENTIFIER so they win the tie. MAP_KEY fuses a name and its colon into one token, so name:value without a space can never be read as a name followed by the keyword :value. AT_DISCARD beats AT by length, so @_ is always a discard and never a custom tag named _.
Rules
document := header? value TRIVIA?
header := "@dxn" version:STRING
value := discard* value_body
discard := AT_DISCARD value
value_body := nil | boolean | number | string | char | keyword | symbol
| list | tuple | map_lit | struct_lit | set_lit
| tag_form
nil := NIL
boolean := TRUE | FALSE
number := DECIMAL | RATIONAL | FLOAT | INTEGER
string := STRING
char := CHAR
symbol := IDENTIFIER
keyword := KEYWORD
list := "[" value* "]"
tuple := "{" value* "}"
set_lit := AT "{" value* "}"
map_lit := "%" "{" map_entry* "}"
map_entry := (short_key:MAP_KEY short_val:value) | (arrow_key:value "=>" arrow_val:value)
struct_lit := "%" name:IDENTIFIER ("(" version:INTEGER ")")? (keyed:struct_keyed | positional:struct_positional)
struct_keyed := "{" map_entry* "}"
struct_positional := "[" (value !"=>")* map_entry* "]"
tag_form := AT name:IDENTIFIER ("(" flags:IDENTIFIER ")")? valueEvery date, time and regular expression is a tag_form with a string, and the string is checked after parsing: dextrin hands it to the host language’s own parsers and refuses one that is not the form its tag names. Because "@dxn" is a longer token than AT everywhere in a document, not only at its start, @dxn can never be a custom tag.
Three things changed with dextrin 0.3. The four sigil tokens and their rule are gone, so ~ begins no token and a sigil fails in the lexer. IDENTIFIER takes segments joined by .. And tag_form takes an optional header in parentheses, the way struct_lit takes a version: @regex(im). ( begins nothing else in DXN, so it needs no token of its own; whether the tag is regex and the letters are flags is checked after parsing.
Where the implementation differs
In a few places dextrin 0.3.2 accepts more than the specification, writes something other than it says, or settles something it leaves open. These are the portable choices:
| Specification | dextrin 0.3.2 | Write |
|---|---|---|
A reader must accept any 1.x header and may refuse another major version | Any version string is accepted unchecked, "2.0" and "banana" included | @dxn "1.0" |
@uri holds an RFC 3986 URI | Any string is accepted: @uri "not a uri" reads | Only real URIs |
| Binary structs are positional, in schema order | With a schema loaded, so they are. Without one, a keyed struct is written in source order and loses its names: %Point{y: 2, x: 1} comes back as %Point[2 1] | Positional form, or encode with the schema loaded |
| Tag 29 refers to “the n-th marked item” | Numbers marked items as they finish decoding, innermost first; the registered CBOR extension numbers them as they open | Leave sharing off for files that other CBOR tools read |
Three rows of the dextrin 0.2.0 list are gone. Sigils are no longer DXN, so ~U[…] with an offset no longer arises. @datetime takes Z by the specification now. And :nil, :true and :false stay keywords, also in a trusted decode. A fourth went with dextrin 0.3.2: a duration’s canonical form leaves out zero fields again, in text and in .dxnb, so @duration "P0D" is @duration "PT0S".