Schemas
A .dxns schema is an ordinary DXN document: a map from names to struct schemas, shapes and named types. It tells a reader what a struct’s fields are called, in which order they come, what they may hold, which may be left out — and which version of the schema it is. The schema language is called DXNS.
A schema document
%{
Percentage: {:refine :float %{min: 0.0, max: 100.0}}
Tag: {:refine :string %{min-length: 1, max-length: 20}}
Address: %schema{
street: :string
city: :string
zip?: :string
}
Customer: %schema{
name: :string
address: Address
loyalty: Percentage
tags?: {:list-of Tag}
}
}Address and Customer are struct schemas. Percentage and Tag are named types — reusable names for a type expression, referred to exactly like a struct name.
%Customer{name: "Ada", address: %Address{street: "1 Main St", city: "Springfield"}, loyalty: 12.5}%Customer(1){name:"Ada",address:%Address(1){street:"1 Main St",city:"Springfield"},loyalty:12.5}Type forms
A bare keyword names a type: one per DXN type, :integer, :string, :datetime, :objectid and so on, plus :any for any value and :never for none. :bits takes bits and bytes, :bytes only whole bytes. A few more keywords are shorthand: :int32, :int64, :uint32 and :uint64 for integers in that range, :float32 for a float single precision holds exactly, :number for any number, and :timestamp for a datetime in UTC. Everything else is a tuple or a struct:
| Form | Matches |
|---|---|
Address, ns/Name | a struct schema, shape or named type, by name |
{:list-of T}, {:set-of T}, {:sorted-set-of T}, {:array-of T} | that collection, every element a T |
{:list-items A {:repeat B 0 nil}} | a list item by item: an A, then any number of B — see below |
{:tuple-of A B C} | a tuple of exactly these slots |
{:tuple-of-all T} | a tuple of any length, every element a T |
{:map-of K V}, {:ordered-map-of K V} | every key a K, every value a V |
{:enum a b …}, {:const v} | one of the given literals; exactly v |
{:one-of A B} | exactly one variant matches — none, or two, is a refusal |
{:any-of A B}, {:all-of A B} | one or more variants; every variant |
{:not T} | anything that is no T |
{:nilable T} | nil or T, short for {:any-of :nil T} |
{:if I T E} | a value that is an I must be a T, any other an E |
{:refine T %{…}} | a T that meets the constraints — see below |
{:tagged acme.money T} | the custom tag @acme.money around a T |
{:case field %{v => T, :else => U}} | the type a sibling field’s value chooses — see shapes |
{:doc T %{description: "…"}} | a T, annotated with a title, description, examples, deprecation or free metadata; changes nothing it takes |
{:generic [t] body}, {:of Name T} | a named type with parameters, and one use of it — see named types |
%shape{…} | a plain map with these fields |
%schema(N){…} | a struct schema, version N |
{:list-items …} describes a list in order. Each part is a type for one item, {:repeat T min max} for between min and max items — max nil for no bound — or {:repeat {:group A B} min max} for several items together. The list matches when every item is taken. A repeat takes as many items as it can and gives them back one at a time, so every implementation makes the same choices.
%{
Job: %schema{
id: {:any-of :integer :uuid}
retries: {:one-of {:refine :integer %{max: 3}} {:refine :integer %{min: 2}}}
command: {:list-items :string {:repeat :string 0 nil}}
tags: {:refine {:list-of :string} %{unique: true, min-count: 1}}
host?: {:not {:enum "localhost"}}
}
}%Job{id: 7, retries: 1, command: ["ls" "-l"], tags: ["a"]}
%Job{id: 7, retries: 3, command: ["ls"], tags: ["a"]}
%Job{id: 7, retries: 1, command: [], tags: ["a"]}
%Job{id: 7, retries: 1, command: ["ls"], tags: ["a" "a"]}
%Job{id: 7, retries: 1, command: ["ls"], tags: ["a"], host: "localhost"}
%Job{id: "7", retries: 1, command: ["ls"], tags: ["a"]}%Job(1){id:7,retries:1,command:["ls" "-l"],tags:["a"]}
schema/type-mismatch: field "retries" does not match its declared type
schema/type-mismatch: field "command" does not match its declared type
schema/type-mismatch: field "tags" does not match its declared type
schema/type-mismatch: field "host" does not match its declared type
schema/type-mismatch: field "id" does not match its declared type3 is refused because both variants of the one-of take it. The empty command lacks the item list-items asks for first. ["a" "a"] is not unique, "localhost" is what the not excludes, and "7" is neither an integer nor a UUID — a string is never converted to make it match.
Constraints
A {:refine T %{…}} value must match T and every constraint in the map. The keys, by the kind of value they apply to:
| Applies to | Keys |
|---|---|
| numbers, dates, times, durations | min, max, exclusive-min, exclusive-max — against a value of the same kind; NaN lies outside every range |
| numbers | multiple-of; for floats finite and float32; for decimals min-precision, max-precision, min-exponent, max-exponent |
| strings, keywords, symbols, bytes, bits | min-length, max-length — code points of a text, bytes of bytes, bits under :bits; min-bytes, max-bytes (UTF-8); pattern |
| strings | format; encoding (base64, base64url, base32, base16), media-type and content — a string holding a DXN document of a given type |
| integers, bytes | bits — only the bits numbered in the list may be set; for bytes also encodes and encodes-seq, one or several .dxnb documents of a given type |
| collections | min-count, max-count; contains with min-contains, max-contains; unique, unique-by |
| datetimes | offsets; before-now, after-now and within, which ask the clock |
| times, datetimes, durations | resolution — no part finer than :second, :millisecond, :microsecond or :nanosecond |
| IP addresses | version 4 or 6; prefix true or false |
format takes one of a closed set of 22 names: date, time, datetime, local-datetime, duration, uuid, ipv4, ipv6, hostname, idn-hostname, email, idn-email, uri, uri-reference, iri, iri-reference, uri-template, json-pointer, relative-json-pointer, semver, mac and host-port. A format DXN has a tag for is read by that tag’s reader; every other one by a grammar the specification gives, so every implementation checks it the same way.
A constraint refuses a value it does not apply to — a length on an integer, a range on a string — and never raises. Everything else is caught when the schema compiles: an unknown key, a misspelt one, an argument of the wrong kind, min-contains without contains, and a named type that reaches itself with nothing between.
%{Name: {:refine :string %{min-lenght: 1}}}
%{Name: {:refine :string %{min-length: -1}}}
%{Name: {:refine {:list-of :integer} %{min-contains: 2}}}
%{A: {:nilable A}}schema/compile: named type "Name": min-lenght is no refine constraint (DXNS.md §2) schema/compile: named type "Name": min-length takes a non-negative integer, not -1 schema/compile: named type "Name": min-contains and max-contains count the items a contains matches schema/compile: named type "A": reaches itself with no field or collection between (A -> A); it describes no value
An enum’s literal is the value itself, compared as its canonical form: {:enum 1} takes 1 and not 1.0.
%E{v: 1}
%E{v: 1.0}%E(1){v:1}
schema/type-mismatch: field "v" does not match its declared typeStruct schemas
The body of a %schema{…} lists the fields directly, and the order they are written in is the order of the fields — the one positional structs and the binary form use. A field named twice is an error. A key ending in ? marks an optional field; the ? is not part of the name. A field that needs more than a type uses %field{type: …, default: …, description: …}. The header holds only the version: %schema(3){…} is version 3, and a schema without one is version 1.
User: %schema(3){
name: :string
email: %field{type: :string, description: "where mail goes"}
role?: %field{type: {:enum :admin :staff :guest}, default: :guest}
id?: :uuid
}Optional means the field may be absent, not that it may be nil: a present optional field must match its type like any other, so nil is refused unless the type is {:nilable T}. Text simply leaves an absent field out. The binary form fills every position, so an encoder writes an absent field’s default where it has one and otherwise refuses, schema/missing-field, rather than write a nil the reader would refuse.
Shapes and cases
A %shape{…} is a named, unversioned type for a plain map. Its body is written like a %schema body — fields in order, ? for optional, %field{} for a default — but it matches a plain map only, never a struct. A declared field is found by name whatever spelling its key has; other keys are extras and are kept. A missing field takes its default when the map is decoded.
{:case field %{value => type, :else => type}} is the type a sibling field chooses: the value of field in the same schema or shape is compared with each key, as an enum compares, and :else applies when none is equal.
%{
Envelope: %schema{
type: :string
payload: {:case type %{"order.placed" => OrderPlaced, :else => :any}}
}
OrderPlaced: %shape{
order-id: :integer
currency: %field{type: :string, default: "EUR"}
gift?: :boolean
}
}%Envelope{type: "order.placed", payload: %{order-id: 42}}
%Envelope{type: "order.placed", payload: %{order-id: "42"}}
%Envelope{type: "ping", payload: [1 2 3]}%Envelope(1){type:"order.placed",payload:%{currency:"EUR",order-id:42}}
schema/type-mismatch: field "payload" does not match its declared type
%Envelope(1){type:"ping",payload:[1 2 3]}Named types and other files
Every top-level entry that is not a %schema{} is a named type. A named type may refer to the others in the same document, in any order: a reader compiles them all before it resolves any reference between them. It may also refer to itself, as long as every such cycle passes a field or a collection’s elements. {:nilable A} inside A checks the same value again and describes nothing, so it does not compile.
A named type may take parameters: {:generic [t] body} declares it, and {:of Name T} uses it with T in place of t. A %schema cannot be generic, since a struct travels under one name; a schema may use a generic in its fields.
%{
Thread: %schema{comments: {:of Page Comment}}
Page: {:generic [t] %shape{items: {:list-of t}, next?: :string}}
Comment: %shape{text: :string, replies?: {:list-of Comment}}
}%Thread{comments: %{items: [%{text: "hi", replies: [%{text: "yes"}]}]}}
%Thread{comments: %{items: [%{text: "hi", replies: [%{text: 1}]}]}}%Thread(1){comments:%{items:[%{replies:[%{text:"yes"}],text:"hi"}]}}
schema/type-mismatch: field "comments" does not match its declared typens/Name refers to a schema or type in another file, and a field typed geo/Address takes a struct named geo/Address, not one named Address. How a reader finds the file is its own business, but one that resolves names to files follows one convention, so a set of schema files serves every implementation. For Foo.Bar/Baz at version N, on each search path in turn:
| Layout | Version 1 | Version N |
|---|---|---|
| directories | Foo/Bar/Baz.dxns, Foo/Bar/Baz-v1.dxns | Foo/Bar/Baz-vN.dxns |
| a directory per name | Foo/Bar/Baz/v1.dxns | Foo/Bar/Baz/vN.dxns |
| flat | Foo.Bar.Baz.dxns, Foo.Bar.Baz-v1.dxns | Foo.Bar.Baz-vN.dxns |
| namespace file | the entry Baz in Foo.Bar.dxns | the same file, by its %schema(N) |
The first search path that has the name decides. A file’s version is its schema’s, one version found in two files is refused, naming both, and every version found is loaded, so an older struct reaches the materializer.
%{Customer: %schema{name: :string, home: geo/Address}}
%Customer{name: "Ada", home: %geo/Address{city: "Dresden"}}
%Customer{name: "Ada", home: %Address{city: "Dresden"}}%Customer(1){name:"Ada",home:%geo/Address(1){city:"Dresden"}}
schema/type-mismatch: field "home" does not match its declared typeEnforcement
Without a schema, a struct is an opaque value, never an error. Once a reader knows the schema, every value of that struct must have its required fields, and every present field a value of its declared type. A value matches only its declared type, and nothing is converted to make it match: an integer is not a string, and a map is not a struct — a field typed Address takes an Address struct and nothing else. A violation is an error of its own kind, distinct from a parse error.
%Money{amount: 9.99M, currency: :eur}
%Money[9.99M :eur]
%Money{amount: 9.99M, currency: :chf}
%Money{currency: :eur}
%Money[9.99M]%Money(1){amount:9.99M,currency::eur}
%Money(1){amount:9.99M,currency::eur}
schema/type-mismatch: field "currency" does not match its declared type
schema/missing-field: missing required field "amount"
schema/field-count: expected 2 positional field(s), got 1Extra fields
A struct guarantees its declared fields and may carry more. Code that relies on a struct can count only on the declared ones; the rest are its extra fields, which a reader keeps and writes back rather than drops. That is what lets a newer producer’s fields pass through an older service unharmed.
- In text they are entries: a keyed struct’s undeclared entries, and a positional struct’s entries after its items — see structs.
- In
.dxnbthey travel as one trailing map under tag 215 — see private tags. - An extra field never shadows a declared one: the same name given both ways is
schema/shadowed-field. Its key follows the map-key rule. - Every reader must offer a strict decode, which refuses extra fields on a struct whose schema it knows:
schema/undeclared-field. A config loader reads strictly, to catch a typo in a hand-written file; a message consumer does not.
%Money{amount: 9.99M, currency: :eur, note: "gift"}
%Money[9.99M :eur note: "gift"]
%Money{amount: 9.99M, currency: :eur, "amount" => 1}
%Money{amount: 9.99M, currency: :eur, note: "gift"} # strict%Money(1){amount:9.99M,currency::eur,note:"gift"}
%Money(1){amount:9.99M,currency::eur,note:"gift"}
schema/shadowed-field: struct "Money" names "amount" twice: an extra field never shadows a declared one
schema/undeclared-field: struct "Money" carries "note", which its schema does not declareThat is why a DXNS type is never closed. Whether unknown fields are refused is the reader’s choice, not the type’s, so the same schema serves a config loader and a message consumer. And a closed type would turn every added field into a breaking change: each older reader would refuse each newer message.
Schema versions
Each .dxns document describes one version of each schema in it. A reader loads any number of documents and keeps every schema by name and version, so Point versions 1, 2 and 3 can be loaded side by side. Loading a version that is already loaded is an error, schema/duplicate-version: two documents that both claim to be Point version 2 are a mistake, not an update. Replacing a loaded version — the hot swap of a running system — is a separate operation, and naming a version that is not loaded there is schema/not-loaded.
# point-v1.dxns
%{Point: %schema{x: :integer, y: :integer}}
# point-v2.dxns
%{Point: %schema(2){x: :integer, y: :integer, label?: :string}}Every struct an encoder writes against a schema carries the latest loaded version, %Point(2){…} — and %Point(1){…} when version 1 is the only one — so a reader never has to guess. A struct written without a version, which mostly means text written by hand, is read as the latest. A struct reference in a schema names no version: a field typed Point accepts any version, and the nested struct’s own tag decides how it is read.
| The struct says | The reader |
|---|---|
| no version, or the latest loaded | reads it with the latest schema, exactly; in .dxnb the item count must match |
| a newer version than any loaded | reads it against the latest it has: text by name, keeping unknown fields as extras; .dxnb by position, dropping items beyond its own fields |
| an older version that is loaded | reads it with that version’s schema and hands it to the materializer; without one, schema/no-materializer |
| an older version that is not loaded | schema/unknown-version — without the writer’s schema there are no field names |
%Point(2){x: 1, y: 2, label: "origin"}%Point(1){x:1,y:2,label:"origin"}A reader may register one materializer per schema: a function that builds the application’s value from a decoded struct. It is called for every struct of its schema, whatever the version, with the name, the version the fields were read with, and the fields; it upgrades older versions, and what it returns is checked against the latest version — a refusal or a rejected result is schema/materializer.
Every implementation provides a version discipline check for CI. It compares the previous release’s schemas with the current ones, fails when a version’s content changed without a new version or a schema’s latest version went down, and lists what the materializer has to upgrade. In dextrin it is mix dextrin.compat:
$ mix dextrin.compat schemas-old/ schemas/
Point 1 -> 2: added label OK: no version discipline problems