joetjen.net
EN DE
Data format

Grammar

The DXN text format is defined by a PEG grammar written in Aether, the grammar language of Ichor. dextrin’s parser is generated from it rather than written by hand.

Format 1.0

Pragmas and tokens

This is the grammar dextrin’s parser is generated from, priv/grammar/dxn.aether as released with dextrin 0.3.2, with its comments removed. Upper-case names are tokens, matched longest-first with ties going to the earlier declaration; lower-case names are rules, matched over the tokens by ordered choice.

dxn.aether
@grammar "dxn"
@root document
@skip TRIVIA

SPACE   := [ \t\r\n,]+
COMMENT := "#" (!"\n" .)* "\n"?
TRIVIA  := (SPACE | COMMENT)*

NIL   := "nil"
TRUE  := "true"
FALSE := "false"

EXPONENT := [eE] [+-]? DIGIT+
FLOAT    := ("-"? DIGIT+ (("." DIGIT+ EXPONENT?) | EXPONENT)) | "NaN" | "Infinity" | "-Infinity"
DECIMAL  := "-"? DIGIT+ ("." DIGIT+)? "M"
RATIONAL := "-"? DIGIT+ "/" DIGIT+
INTEGER  := "-"? DIGIT+

IDENT_START := [_\u{41}-\u{5A}\u{61}-\u{7A}\u{AA} … \u{30000}-\u{3134A}\u{31350}-\u{33479}]
IDENT_CONT  := IDENT_START | [\u{30}-\u{39}\u{41}-\u{5A}\u{5F}\u{61}-\u{7A} … \u{E0100}-\u{E01EF}\-?!]
IDENTIFIER  := IDENT_START IDENT_CONT* ("." IDENT_START IDENT_CONT*)* ("/" IDENT_START IDENT_CONT* ("." IDENT_START IDENT_CONT*)*)?

MAP_KEY := IDENTIFIER ":"
KEYWORD := ":" (IDENTIFIER | STRING)

AT_DISCARD := "@_"
AT         := "@"

ESCAPE := "\\" ("\"" | "\\" | "n" | "t" | "r" | "0" | "a" | "b" | "f" | "v" | ("x{" HEX+ "}"))
STRING := "\"" (ESCAPE | (![\"\\] .))* "\""
CHAR   := "?" (ESCAPE | "s" | (![ \t\r\n] .))

Declaration order does real work here. NIL, TRUE and FALSE come before IDENTIFIER so they win the tie. MAP_KEY fuses a name and its colon into one token, so name:value without a space can never be read as a name followed by the keyword :value. AT_DISCARD beats AT by length, so @_ is always a discard and never a custom tag named _.

Rules

document := header? value TRIVIA?
header   := "@dxn" version:STRING

value   := discard* value_body
discard := AT_DISCARD value

value_body := nil | boolean | number | string | char | keyword | symbol
            | list | tuple | map_lit | struct_lit | set_lit
            | tag_form

nil     := NIL
boolean := TRUE | FALSE
number  := DECIMAL | RATIONAL | FLOAT | INTEGER
string  := STRING
char    := CHAR
symbol  := IDENTIFIER
keyword := KEYWORD

list    := "[" value* "]"
tuple   := "{" value* "}"
set_lit := AT "{" value* "}"

map_lit   := "%" "{" map_entry* "}"
map_entry := (short_key:MAP_KEY short_val:value) | (arrow_key:value "=>" arrow_val:value)

struct_lit        := "%" name:IDENTIFIER ("(" version:INTEGER ")")? (keyed:struct_keyed | positional:struct_positional)
struct_keyed      := "{" map_entry* "}"
struct_positional := "[" (value !"=>")* map_entry* "]"

tag_form := AT name:IDENTIFIER ("(" flags:IDENTIFIER ")")? value

Every date, time and regular expression is a tag_form with a string, and the string is checked after parsing: dextrin hands it to the host language’s own parsers and refuses one that is not the form its tag names. Because "@dxn" is a longer token than AT everywhere in a document, not only at its start, @dxn can never be a custom tag.

Three things changed with dextrin 0.3. The four sigil tokens and their rule are gone, so ~ begins no token and a sigil fails in the lexer. IDENTIFIER takes segments joined by .. And tag_form takes an optional header in parentheses, the way struct_lit takes a version: @regex(im). ( begins nothing else in DXN, so it needs no token of its own; whether the tag is regex and the letters are flags is checked after parsing.

Where the implementation differs

In a few places dextrin 0.3.2 accepts more than the specification, writes something other than it says, or settles something it leaves open. These are the portable choices:

Specificationdextrin 0.3.2Write
A reader must accept any 1.x header and may refuse another major versionAny version string is accepted unchecked, "2.0" and "banana" included@dxn "1.0"
@uri holds an RFC 3986 URIAny string is accepted: @uri "not a uri" readsOnly real URIs
Binary structs are positional, in schema orderWith a schema loaded, so they are. Without one, a keyed struct is written in source order and loses its names: %Point{y: 2, x: 1} comes back as %Point[2 1]Positional form, or encode with the schema loaded
Tag 29 refers to “the n-th marked item”Numbers marked items as they finish decoding, innermost first; the registered CBOR extension numbers them as they openLeave sharing off for files that other CBOR tools read

Three rows of the dextrin 0.2.0 list are gone. Sigils are no longer DXN, so ~U[…] with an offset no longer arises. @datetime takes Z by the specification now. And :nil, :true and :false stay keywords, also in a trusted decode. A fourth went with dextrin 0.3.2: a duration’s canonical form leaves out zero fields again, in text and in .dxnb, so @duration "P0D" is @duration "PT0S".