joetjen.net
EN DE
Data format

Schemas

A .dxns schema is an ordinary DXN document: a map from names to struct schemas, shapes and named types. It tells a reader what a struct’s fields are called, in which order they come, what they may hold, which may be left out — and which version of the schema it is. The schema language is called DXNS.

Format 1.0

A schema document

customer.dxns
%{
  Percentage: {:refine :float %{min: 0.0, max: 100.0}}
  Tag:        {:refine :string %{min-length: 1, max-length: 20}}

  Address: %schema{
    street: :string
    city:   :string
    zip?:   :string
  }

  Customer: %schema{
    name:    :string
    address: Address
    loyalty: Percentage
    tags?:   {:list-of Tag}
  }
}

Address and Customer are struct schemas. Percentage and Tag are named types — reusable names for a type expression, referred to exactly like a struct name.

Read with customer.dxns loaded
%Customer{name: "Ada", address: %Address{street: "1 Main St", city: "Springfield"}, loyalty: 12.5}
%Customer(1){name:"Ada",address:%Address(1){street:"1 Main St",city:"Springfield"},loyalty:12.5}

Type forms

A bare keyword names a type: one per DXN type, :integer, :string, :datetime, :objectid and so on, plus :any for any value and :never for none. :bits takes bits and bytes, :bytes only whole bytes. A few more keywords are shorthand: :int32, :int64, :uint32 and :uint64 for integers in that range, :float32 for a float single precision holds exactly, :number for any number, and :timestamp for a datetime in UTC. Everything else is a tuple or a struct:

FormMatches
Address, ns/Namea struct schema, shape or named type, by name
{:list-of T}, {:set-of T}, {:sorted-set-of T}, {:array-of T}that collection, every element a T
{:list-items A {:repeat B 0 nil}}a list item by item: an A, then any number of B — see below
{:tuple-of A B C}a tuple of exactly these slots
{:tuple-of-all T}a tuple of any length, every element a T
{:map-of K V}, {:ordered-map-of K V}every key a K, every value a V
{:enum a b …}, {:const v}one of the given literals; exactly v
{:one-of A B}exactly one variant matches — none, or two, is a refusal
{:any-of A B}, {:all-of A B}one or more variants; every variant
{:not T}anything that is no T
{:nilable T}nil or T, short for {:any-of :nil T}
{:if I T E}a value that is an I must be a T, any other an E
{:refine T %{…}}a T that meets the constraints — see below
{:tagged acme.money T}the custom tag @acme.money around a T
{:case field %{v => T, :else => U}}the type a sibling field’s value chooses — see shapes
{:doc T %{description: "…"}}a T, annotated with a title, description, examples, deprecation or free metadata; changes nothing it takes
{:generic [t] body}, {:of Name T}a named type with parameters, and one use of it — see named types
%shape{…}a plain map with these fields
%schema(N){…}a struct schema, version N

{:list-items …} describes a list in order. Each part is a type for one item, {:repeat T min max} for between min and max items — max nil for no bound — or {:repeat {:group A B} min max} for several items together. The list matches when every item is taken. A repeat takes as many items as it can and gives them back one at a time, so every implementation makes the same choices.

job.dxns
%{
  Job: %schema{
    id:      {:any-of :integer :uuid}
    retries: {:one-of {:refine :integer %{max: 3}} {:refine :integer %{min: 2}}}
    command: {:list-items :string {:repeat :string 0 nil}}
    tags:    {:refine {:list-of :string} %{unique: true, min-count: 1}}
    host?:   {:not {:enum "localhost"}}
  }
}
Read with job.dxns loaded
%Job{id: 7, retries: 1, command: ["ls" "-l"], tags: ["a"]}
%Job{id: 7, retries: 3, command: ["ls"], tags: ["a"]}
%Job{id: 7, retries: 1, command: [], tags: ["a"]}
%Job{id: 7, retries: 1, command: ["ls"], tags: ["a" "a"]}
%Job{id: 7, retries: 1, command: ["ls"], tags: ["a"], host: "localhost"}
%Job{id: "7", retries: 1, command: ["ls"], tags: ["a"]}
%Job(1){id:7,retries:1,command:["ls" "-l"],tags:["a"]}
schema/type-mismatch: field "retries" does not match its declared type
schema/type-mismatch: field "command" does not match its declared type
schema/type-mismatch: field "tags" does not match its declared type
schema/type-mismatch: field "host" does not match its declared type
schema/type-mismatch: field "id" does not match its declared type

3 is refused because both variants of the one-of take it. The empty command lacks the item list-items asks for first. ["a" "a"] is not unique, "localhost" is what the not excludes, and "7" is neither an integer nor a UUID — a string is never converted to make it match.

Constraints

A {:refine T %{…}} value must match T and every constraint in the map. The keys, by the kind of value they apply to:

Applies toKeys
numbers, dates, times, durationsmin, max, exclusive-min, exclusive-max — against a value of the same kind; NaN lies outside every range
numbersmultiple-of; for floats finite and float32; for decimals min-precision, max-precision, min-exponent, max-exponent
strings, keywords, symbols, bytes, bitsmin-length, max-length — code points of a text, bytes of bytes, bits under :bits; min-bytes, max-bytes (UTF-8); pattern
stringsformat; encoding (base64, base64url, base32, base16), media-type and content — a string holding a DXN document of a given type
integers, bytesbits — only the bits numbered in the list may be set; for bytes also encodes and encodes-seq, one or several .dxnb documents of a given type
collectionsmin-count, max-count; contains with min-contains, max-contains; unique, unique-by
datetimesoffsets; before-now, after-now and within, which ask the clock
times, datetimes, durationsresolution — no part finer than :second, :millisecond, :microsecond or :nanosecond
IP addressesversion 4 or 6; prefix true or false

format takes one of a closed set of 22 names: date, time, datetime, local-datetime, duration, uuid, ipv4, ipv6, hostname, idn-hostname, email, idn-email, uri, uri-reference, iri, iri-reference, uri-template, json-pointer, relative-json-pointer, semver, mac and host-port. A format DXN has a tag for is read by that tag’s reader; every other one by a grammar the specification gives, so every implementation checks it the same way.

A constraint refuses a value it does not apply to — a length on an integer, a range on a string — and never raises. Everything else is caught when the schema compiles: an unknown key, a misspelt one, an argument of the wrong kind, min-contains without contains, and a named type that reaches itself with nothing between.

%{Name: {:refine :string %{min-lenght: 1}}}
%{Name: {:refine :string %{min-length: -1}}}
%{Name: {:refine {:list-of :integer} %{min-contains: 2}}}
%{A: {:nilable A}}
schema/compile: named type "Name": min-lenght is no refine constraint (DXNS.md §2)
schema/compile: named type "Name": min-length takes a non-negative integer, not -1
schema/compile: named type "Name": min-contains and max-contains count the items a contains matches
schema/compile: named type "A": reaches itself with no field or collection between (A -> A); it describes no value

An enum’s literal is the value itself, compared as its canonical form: {:enum 1} takes 1 and not 1.0.

With %{E: %schema{v: {:enum 1}}} loaded
%E{v: 1}
%E{v: 1.0}
%E(1){v:1}
schema/type-mismatch: field "v" does not match its declared type

Struct schemas

The body of a %schema{…} lists the fields directly, and the order they are written in is the order of the fields — the one positional structs and the binary form use. A field named twice is an error. A key ending in ? marks an optional field; the ? is not part of the name. A field that needs more than a type uses %field{type: …, default: …, description: …}. The header holds only the version: %schema(3){…} is version 3, and a schema without one is version 1.

User: %schema(3){
  name:  :string
  email: %field{type: :string, description: "where mail goes"}
  role?: %field{type: {:enum :admin :staff :guest}, default: :guest}
  id?:   :uuid
}

Optional means the field may be absent, not that it may be nil: a present optional field must match its type like any other, so nil is refused unless the type is {:nilable T}. Text simply leaves an absent field out. The binary form fills every position, so an encoder writes an absent field’s default where it has one and otherwise refuses, schema/missing-field, rather than write a nil the reader would refuse.

Shapes and cases

A %shape{…} is a named, unversioned type for a plain map. Its body is written like a %schema body — fields in order, ? for optional, %field{} for a default — but it matches a plain map only, never a struct. A declared field is found by name whatever spelling its key has; other keys are extras and are kept. A missing field takes its default when the map is decoded.

{:case field %{value => type, :else => type}} is the type a sibling field chooses: the value of field in the same schema or shape is compared with each key, as an enum compares, and :else applies when none is equal.

envelope.dxns
%{
  Envelope: %schema{
    type:    :string
    payload: {:case type %{"order.placed" => OrderPlaced, :else => :any}}
  }
  OrderPlaced: %shape{
    order-id: :integer
    currency: %field{type: :string, default: "EUR"}
    gift?:    :boolean
  }
}
Read with envelope.dxns loaded
%Envelope{type: "order.placed", payload: %{order-id: 42}}
%Envelope{type: "order.placed", payload: %{order-id: "42"}}
%Envelope{type: "ping", payload: [1 2 3]}
%Envelope(1){type:"order.placed",payload:%{currency:"EUR",order-id:42}}
schema/type-mismatch: field "payload" does not match its declared type
%Envelope(1){type:"ping",payload:[1 2 3]}

Named types and other files

Every top-level entry that is not a %schema{} is a named type. A named type may refer to the others in the same document, in any order: a reader compiles them all before it resolves any reference between them. It may also refer to itself, as long as every such cycle passes a field or a collection’s elements. {:nilable A} inside A checks the same value again and describes nothing, so it does not compile.

A named type may take parameters: {:generic [t] body} declares it, and {:of Name T} uses it with T in place of t. A %schema cannot be generic, since a struct travels under one name; a schema may use a generic in its fields.

thread.dxns
%{
  Thread:  %schema{comments: {:of Page Comment}}
  Page:    {:generic [t] %shape{items: {:list-of t}, next?: :string}}
  Comment: %shape{text: :string, replies?: {:list-of Comment}}
}
Read with thread.dxns loaded
%Thread{comments: %{items: [%{text: "hi", replies: [%{text: "yes"}]}]}}
%Thread{comments: %{items: [%{text: "hi", replies: [%{text: 1}]}]}}
%Thread(1){comments:%{items:[%{replies:[%{text:"yes"}],text:"hi"}]}}
schema/type-mismatch: field "comments" does not match its declared type

ns/Name refers to a schema or type in another file, and a field typed geo/Address takes a struct named geo/Address, not one named Address. How a reader finds the file is its own business, but one that resolves names to files follows one convention, so a set of schema files serves every implementation. For Foo.Bar/Baz at version N, on each search path in turn:

LayoutVersion 1Version N
directoriesFoo/Bar/Baz.dxns, Foo/Bar/Baz-v1.dxnsFoo/Bar/Baz-vN.dxns
a directory per nameFoo/Bar/Baz/v1.dxnsFoo/Bar/Baz/vN.dxns
flatFoo.Bar.Baz.dxns, Foo.Bar.Baz-v1.dxnsFoo.Bar.Baz-vN.dxns
namespace filethe entry Baz in Foo.Bar.dxnsthe same file, by its %schema(N)

The first search path that has the name decides. A file’s version is its schema’s, one version found in two files is refused, naming both, and every version found is loaded, so an older struct reaches the materializer.

customer.dxns, with geo.dxns holding %{Address: %schema{city: :string}} on the search path
%{Customer: %schema{name: :string, home: geo/Address}}

%Customer{name: "Ada", home: %geo/Address{city: "Dresden"}}
%Customer{name: "Ada", home: %Address{city: "Dresden"}}
%Customer(1){name:"Ada",home:%geo/Address(1){city:"Dresden"}}
schema/type-mismatch: field "home" does not match its declared type

Enforcement

Without a schema, a struct is an opaque value, never an error. Once a reader knows the schema, every value of that struct must have its required fields, and every present field a value of its declared type. A value matches only its declared type, and nothing is converted to make it match: an integer is not a string, and a map is not a struct — a field typed Address takes an Address struct and nothing else. A violation is an error of its own kind, distinct from a parse error.

Read with %{Money: %schema{amount: :decimal, currency: {:enum :usd :eur :gbp}}} loaded
%Money{amount: 9.99M, currency: :eur}
%Money[9.99M :eur]
%Money{amount: 9.99M, currency: :chf}
%Money{currency: :eur}
%Money[9.99M]
%Money(1){amount:9.99M,currency::eur}
%Money(1){amount:9.99M,currency::eur}
schema/type-mismatch: field "currency" does not match its declared type
schema/missing-field: missing required field "amount"
schema/field-count: expected 2 positional field(s), got 1

Extra fields

A struct guarantees its declared fields and may carry more. Code that relies on a struct can count only on the declared ones; the rest are its extra fields, which a reader keeps and writes back rather than drops. That is what lets a newer producer’s fields pass through an older service unharmed.

  • In text they are entries: a keyed struct’s undeclared entries, and a positional struct’s entries after its items — see structs.
  • In .dxnb they travel as one trailing map under tag 215 — see private tags.
  • An extra field never shadows a declared one: the same name given both ways is schema/shadowed-field. Its key follows the map-key rule.
  • Every reader must offer a strict decode, which refuses extra fields on a struct whose schema it knows: schema/undeclared-field. A config loader reads strictly, to catch a typo in a hand-written file; a message consumer does not.
With the Money schema above
%Money{amount: 9.99M, currency: :eur, note: "gift"}
%Money[9.99M :eur note: "gift"]
%Money{amount: 9.99M, currency: :eur, "amount" => 1}
%Money{amount: 9.99M, currency: :eur, note: "gift"}   # strict
%Money(1){amount:9.99M,currency::eur,note:"gift"}
%Money(1){amount:9.99M,currency::eur,note:"gift"}
schema/shadowed-field: struct "Money" names "amount" twice: an extra field never shadows a declared one
schema/undeclared-field: struct "Money" carries "note", which its schema does not declare

That is why a DXNS type is never closed. Whether unknown fields are refused is the reader’s choice, not the type’s, so the same schema serves a config loader and a message consumer. And a closed type would turn every added field into a breaking change: each older reader would refuse each newer message.

Schema versions

Each .dxns document describes one version of each schema in it. A reader loads any number of documents and keeps every schema by name and version, so Point versions 1, 2 and 3 can be loaded side by side. Loading a version that is already loaded is an error, schema/duplicate-version: two documents that both claim to be Point version 2 are a mistake, not an update. Replacing a loaded version — the hot swap of a running system — is a separate operation, and naming a version that is not loaded there is schema/not-loaded.

# point-v1.dxns
%{Point: %schema{x: :integer, y: :integer}}

# point-v2.dxns
%{Point: %schema(2){x: :integer, y: :integer, label?: :string}}

Every struct an encoder writes against a schema carries the latest loaded version, %Point(2){…} — and %Point(1){…} when version 1 is the only one — so a reader never has to guess. A struct written without a version, which mostly means text written by hand, is read as the latest. A struct reference in a schema names no version: a field typed Point accepts any version, and the nested struct’s own tag decides how it is read.

The struct saysThe reader
no version, or the latest loadedreads it with the latest schema, exactly; in .dxnb the item count must match
a newer version than any loadedreads it against the latest it has: text by name, keeping unknown fields as extras; .dxnb by position, dropping items beyond its own fields
an older version that is loadedreads it with that version’s schema and hands it to the materializer; without one, schema/no-materializer
an older version that is not loadedschema/unknown-version — without the writer’s schema there are no field names
An older reader, with only point-v1.dxns loaded
%Point(2){x: 1, y: 2, label: "origin"}
%Point(1){x:1,y:2,label:"origin"}

A reader may register one materializer per schema: a function that builds the application’s value from a decoded struct. It is called for every struct of its schema, whatever the version, with the name, the version the fields were read with, and the fields; it upgrades older versions, and what it returns is checked against the latest version — a refusal or a rejected result is schema/materializer.

Every implementation provides a version discipline check for CI. It compares the previous release’s schemas with the current ones, fails when a version’s content changed without a new version or a schema’s latest version went down, and lists what the materializer has to upgrade. In dextrin it is mix dextrin.compat:

$ mix dextrin.compat schemas-old/ schemas/
Point 1 -> 2: added label
OK: no version discipline problems