php-dextrin

Lexer
in package

FinalYes

Hand-written `.dxn` tokenizer, directly off `dxn/DXN.md` ยง1.1's lexical grammar. No parser-generator dependency -- the grammar is small and fully normative.

ws = { " " | "\t" | "\n" | "\r" | "," } is "fully insignificant, everywhere" per the spec -- this lexer skips it (and # comments) between every token rather than emitting whitespace tokens at all.

DXN 0.2's sigils, ~D[...], ~T[...], ~U[...] and ~r/.../, are gone (Jan, 2026-10-20): every such value is a tag with a string, @date "...", so ~ begins no token and is refused where it stands.

A few tokens bind tighter than plain "insignificant whitespace everywhere" would suggest:

  • identifier ":" (the map-entry colon-shorthand key) is lexed as one MAP_KEY token when the : immediately follows the identifier with no whitespace between them; otherwise the : starts a fresh KEYWORD token instead (:name in value position), and the identifier is left as a separate token.
  • keyword (:name / :"...") similarly requires no whitespace between : and what follows.

Operates on the source split into an array of single-codepoint strings (see Token's own doc) rather than raw byte-indexed PHP strings -- \p{ID_Start}/\p{ID_Continue} regex tests then always run against one well-formed codepoint at a time.

Table of Contents

Constants

DIGIT  : mixed = '/^[0-9]$/'
HEX_DIGIT  : mixed = '/^[0-9a-fA-F]$/'
ID_CONTINUE  : mixed = '/^[\p{ID_Continue}\-?!]$/u'
ID_START  : mixed = '/^[\p{ID_Start}_]$/u'
RESERVED  : mixed = ['nil' => true, 'true' => true, 'false' => true...

Properties

$cp  : array<int, string>
$i  : int
$len  : int

Methods

tokenize()  : array<int, Token>
__construct()  : mixed
arrow()  : Token
isDigit()  : bool
isIdContinue()  : bool
isIdStart()  : bool
looksLikeExponent()  : bool
nextToken()  : Token
readChar()  : Token
readColon()  : Token
readIdentifierOrKeywordLike()  : Token
readIdentifierRun()  : string
`identifier = (letter | "_"), {ident_char}, ["/", (letter | "_"), {ident_char}]`. First char already validated present at $this->i.
readNumber()  : Token
readSegmentsRest()  : void
The rest of a run of segments joined by ".", the first character of the first one already read. A "." not followed by a segment's start is left where it is, and refused by whatever reads it next.
readString()  : Token
reservedToken()  : Token
run()  : array<int, Token>
single()  : Token
skipTrivia()  : void
slice()  : string
startsWith()  : bool

Constants

DIGIT

private mixed DIGIT = '/^[0-9]$/'

HEX_DIGIT

private mixed HEX_DIGIT = '/^[0-9a-fA-F]$/'

ID_CONTINUE

private mixed ID_CONTINUE = '/^[\p{ID_Continue}\-?!]$/u'

ID_START

private mixed ID_START = '/^[\p{ID_Start}_]$/u'

RESERVED

private mixed RESERVED = ['nil' => true, 'true' => true, 'false' => true, 'NaN' => true, 'Infinity' => true]

Properties

$cp read-only

private array<int, string> $cp

$len read-only

private int $len

Methods

tokenize()

public static tokenize(string $src) : array<int, Token>
Parameters
$src : string
Return values
array<int, Token>

__construct()

private __construct(string $src) : mixed
Parameters
$src : string

isDigit()

private isDigit(string|null $c) : bool
Parameters
$c : string|null
Return values
bool

isIdContinue()

private isIdContinue(string|null $c) : bool
Parameters
$c : string|null
Return values
bool

isIdStart()

private isIdStart(string $c) : bool
Parameters
$c : string
Return values
bool

looksLikeExponent()

private looksLikeExponent(int $at) : bool
Parameters
$at : int
Return values
bool

readIdentifierOrKeywordLike()

private readIdentifierOrKeywordLike(int $start) : Token
Parameters
$start : int
Return values
Token

readIdentifierRun()

`identifier = (letter | "_"), {ident_char}, ["/", (letter | "_"), {ident_char}]`. First char already validated present at $this->i.

private readIdentifierRun() : string
Return values
string

readSegmentsRest()

The rest of a run of segments joined by ".", the first character of the first one already read. A "." not followed by a segment's start is left where it is, and refused by whatever reads it next.

private readSegmentsRest() : void

reservedToken()

private reservedToken(string $name, int $start, int $end) : Token
Parameters
$name : string
$start : int
$end : int
Return values
Token

single()

private single(string $type) : Token
Parameters
$type : string
Return values
Token

skipTrivia()

private skipTrivia() : void

slice()

private slice(int $start, int $end) : string
Parameters
$start : int
$end : int
Return values
string

startsWith()

private startsWith(int $at, string $literal) : bool
Parameters
$at : int
$literal : string
Return values
bool
On this page

Search results