# tabnas — full documentation

> tabnas is a parsing engine that can handle any language. Grammars are data, so you extend one that already works instead of starting over — and an agent can write one directly.

Describes @tabnas/parser 0.8.10. Exact package pins: https://tabnas.dev/versions.json
Index: https://tabnas.dev/llms.txt · Error codes: https://tabnas.dev/errors

13 documentation pages and 12 how-to guides follow, in site order.

---

# Introduction

Source: https://tabnas.dev/docs/introduction
What tabnas is, and how a grammar becomes a parser.

**tabnas** is a parsing engine that can handle any language. You describe a
language as a grammar and get a working parser that builds a small, uniform
syntax tree.

There is no code-generation step. The grammar *is* the parser: a table of
rules and token lookups the engine walks at runtime. That has two consequences,
and they are the reason the project exists.

**Extension is cheap.** A new language is usually an existing one plus some
rules — JSONC is JSON with comments, jsonic is JSONC with relaxed quoting,
and so on. You add rules and token alternates to a grammar that already
parses, rather than forking a parser and owning the fork.

**The engine is a good compile target.** ABNF compiles into it, so a human can
write a grammar in the notation the RFCs already use. And because the grammar
is flat declarative data with no control flow, a language model can emit one
directly — and you can inspect it, print it back as ABNF, or draw it as a
railroad diagram before you run it.

See [why tabnas](/why) for where this came from, and
[agents](/agents) for the second point in full.

## The syntax tree

Every parse yields the same shape:

Because the shape never changes, a walker, action, or tool you write once
applies to every grammar you define.

## Runtimes

The engine is implemented twice, in TypeScript and in Go, so it runs wherever
you do. TypeScript is the reference implementation; the Go port tracks it and
is verified against the same fixtures. A grammar parses the same way in both.

## Where to go next

- [Quickstart](/docs/quickstart) — parse your first input in five minutes.
- [Your first grammar](/docs/first-grammar) — build a language from nothing.
- [A grammar with plugins](/docs/grammar-with-plugins) — build one by composing
  what already exists. Usually the cheaper route.
- [The rule table](/docs/rule-table) — the grammar format, in full.
- [How it works](/docs/how-it-works) — the ideas behind the engine.

---

# Quickstart

Source: https://tabnas.dev/docs/quickstart
Parse your first input with tabnas in five minutes.

Install the engine, define a small grammar in ABNF, parse a string, then attach
actions to compute a value. TypeScript here; the Go path mirrors it exactly.

## 1 · Install

```bash
npm install @tabnas/parser @tabnas/abnf
```

## 2 · Define a grammar

An addition grammar, written in ABNF — `NR` is the built-in number token,
`[ … ]` is optional, and the rule refers to itself to handle a whole chain.

`val` wraps the chain — it's the rule that will hold the running total.

## 3 · Parse

That grammar recognises the input and builds a tree. Every parse has the same
`{ rule, src, kids }` shape:

Each repetition of `add` is a sibling — the compiler turns the tail
self-reference `[ PL add ]` into a same-depth repeat, not a nested push.
(`PL` compiles to a token, so it never appears in `kids`.)

## 4 · Add actions

Recognising isn't computing. To get a total, attach actions by reference — the
grammar text stays untouched.

Actions attach by **alternate mark** — a rule's alternate, named by its
leading discriminator. `'@val:o:add'` is the `val` rule's alternate that
pushes `add`; `'@add:o:NR'` is the `add` rule's alternate on an `NR` token.
`r.o` holds the tokens that alternate matched, so `r.o[0].val` is the number
just read — already a number, courtesy of the lexer.

`r.parent` is `val` for **every** repetition — that's the same-depth repeat
again — so the total accumulates in one place, on `val`'s node, where
`parse` returns it. The instance carries no state between calls.

These are the same two actions a hand-written rule table uses for this
grammar (see [the home page](/#a-grammar-end-to-end), steps 3 and 4):
ABNF and the rule table aren't just equivalent notations, they compile to
the same machine.

Mark names come from each alternate's leading discriminator, so ask the
compiler rather than guessing:

```bash
tabnas-abnf --marks -f grammar.abnf
```

```
val  o:add  p:add
val  c:_  (empty)
add  o:NR  s:#NR
add  c:PL  s:#PL
add  c:_  (empty)
```

`add c:PL` is the repeat itself — a close-phase alternate you can attach an
action to with `'@add:c:PL'`.

## Next steps

- Open the [playground](/playground) and edit this grammar live.
- Read about [ABNF grammars](/docs/abnf-grammars) — repetition, groups, left
  recursion, and `@ref` actions in full.
- The [home page](/#a-grammar-end-to-end) shows this grammar four ways,
  including how to carry the total on the parse itself instead of in an outer
  variable.
- Building in Go? The same grammar and tree are available via
  `github.com/tabnas/parser/go`.

---

# Your first grammar

Source: https://tabnas.dev/docs/first-grammar
Build a small language from nothing — recognise it, reject bad input, then read the tree.

The [quickstart](/docs/quickstart) parsed arithmetic. This one builds a
grammar that depends on nothing but the engine — no existing language
underneath it — and takes it as far as a usable tree. It should take about ten
minutes.

We'll parse a comma-separated list: `a,bc,def`.

## 1 · Install

```bash
npm install @tabnas/parser @tabnas/abnf
```

## 2 · Write the grammar

Three rules, in ABNF:

Reading it: a `list` is an `item`, followed by zero or more (`*`) groups of a
comma and another item. An `item` is one or more (`1*`) letters. `ALPHA` is one
of the RFC 5234 core rules, included automatically when you refer to it.

`COMMA` is declared as its own rule so the comma becomes a named token rather
than an anonymous literal — which means it shows up in the tree, and you can
attach behaviour to it later.

## 3 · Parse

Every parse returns the same shape — `{ rule, src, kids }` — whatever the
grammar. Walk it once and the walker works for every language you define.

## 4 · Check that it rejects

This is the step people skip, and it's the one that catches a broken grammar.
A grammar that accepts everything looks exactly like a grammar that works.

If `a,,b` had parsed, the `*( COMMA item )` group would be matching a comma
without requiring an item after it. Always try the malformed input.

## 5 · See inside it

Two packages read a live grammar, which is possible because the grammar is
still data at runtime rather than generated code:

```bash
npm install @tabnas/debug @tabnas/railroad
```

`@tabnas/debug` can describe the grammar and print it back as ABNF — if the
round-trip isn't what you wrote, the grammar isn't what you meant.
`@tabnas/railroad` draws it as a syntax diagram.

You can also paste the grammar straight into the
[playground](/playground) and watch the tree change as you type.

## What you skipped

Two things this grammar quietly relies on, worth knowing before you write a
bigger one:

- **The lexer.** `ALPHA` and the number token `NR` are built in, and `COMMA` is
  a fixed token. Beyond that you configure or write matchers yourself — tabnas
  separates lexing from parsing and is mostly about the parsing half.
- **Repetition is desugared.** `*( … )` compiles to generated group rules, so
  the runtime rule names aren't only the ones you wrote. That matters when you
  attach actions — see [attaching actions](/docs/actions).

## Next

- [A grammar with plugins](/docs/grammar-with-plugins) — the other way to
  build, and usually the cheaper one.
- [Attaching actions](/docs/actions) — turn a tree into a value.
- [The rule table](/docs/rule-table) — what the ABNF compiled into.

---

# A grammar with plugins

Source: https://tabnas.dev/docs/grammar-with-plugins
Build a configuration language by composing existing grammars instead of writing one.

[Your first grammar](/docs/first-grammar) built a language from nothing. This
is the other way, and usually the cheaper one: start from a grammar that
already parses something close, and add the pieces you need.

We'll build a config format that takes arithmetic in its values — so
`width: 2+3*4` yields `14`, not the string `"2+3*4"`. Writing that from
scratch means an expression parser with operator precedence. Composing it takes
about six lines.

## 1 · Install

```bash
npm install @tabnas/jsonic @tabnas/expr
```

[jsonic](https://github.com/tabnas/jsonic) is a relaxed JSON — unquoted keys,
implicit objects, comments, trailing commas.
[expr](https://github.com/tabnas/expr) adds Pratt-parser expressions with a
configurable precedence scale.

## 2 · Start from jsonic

That is already a usable config format. What it doesn't do is arithmetic:
`x: 1+2` gives you a string.

## 3 · Add the expression plugin

`Jsonic.make()` derives a fresh instance so the base parser is left alone, and
`.use()` layers a plugin onto it.

Values are now expression trees, and precedence is already handled — `1+2*3`
groups the multiplication first, without you saying so. Parentheses work too:

(The first element is an operator node; the readable part is its `src`. The
shapes above are abbreviated for clarity.)

## 4 · Evaluate

`expr` did the parsing, so evaluation is a short recursive walk:

And that's the language:

## What just happened

You didn't write a parser. You picked one that was close, added a plugin for
the part it was missing, and wrote ten lines of evaluation. No grammar file, no
generated code, and no fork of jsonic — the base instance is untouched, so
other code using it is unaffected.

This is the normal way to build with tabnas, and it composes further:

| Add | For |
|---|---|
| [directive](https://github.com/tabnas/directive) | `@name` and `add<1,2>` forms |
| [hoover](https://github.com/tabnas/hoover) | Block strings with unquoted spaces |
| [multisource](https://github.com/tabnas/multisource) | One document pulling in others |
| [path](https://github.com/tabnas/path) | Knowing where each value sits in the tree |

[aontu](/examples) is the same trick at full size: a CUE-like configuration
language built from five of these plugins and no parser of its own.

## Next

- [Extending a grammar](/docs/extending) — adding and removing rules directly.
- [The rule table](/docs/rule-table) — what a plugin is actually doing.
- [Packages](/docs/packages) — everything available to compose.

---

# The rule instance

Source: https://tabnas.dev/docs/rule-instance
What an action receives — nodes, matched tokens, the rules around it, and the two kinds of scratch data.

Every action, condition and error function is handed the same thing: `r`, the
rule instance. It is the rule *as it is running* — one per activation, not one
per rule — and it is where all the state of a parse lives.

This walks through it with a two-rule grammar you can run. Ten minutes.

## A grammar to look at

`name = value`, and nothing else:

Two rules, four fields of `r` between them. Now the rest.

## Where you are

| Field | What it is |
|---|---|
| `r.name` | the rule's name — `'pair'` |
| `r.state` | `'o'` in the open phase, `'c'` in the close phase |
| `r.d` | stack depth; `0` for the start rule |
| `r.i` | a serial number, unique per rule activation in this parse |

`r.state` matters more than it looks. The same action can be attached to both
phases, and "am I going down or coming back up" is usually the question it
needs answered.

## What you matched

The open phase collects tokens into `r.o`, the close phase into `r.c`:

| Field | What it is |
|---|---|
| `r.o` | tokens matched in the **open** phase |
| `r.o0` `r.o1` | shorthand for `r.o[0]` and `r.o[1]` |
| `r.os` | how many open tokens matched |
| `r.c` | tokens matched in the **close** phase |
| `r.c0` `r.c1` | shorthand for `r.c[0]` and `r.c[1]` |
| `r.cs` | how many close tokens matched |

For the alternate `{ s: ['#TX', '#EQ'] }` on input `port = 8080`:

`r.o0` is the one you will reach for most: it is the token that decided which
alternate you are in, which makes it the right thing to point an
[error](/how-to/parse-errors) at.

### Inside a token

| Field | What it is |
|---|---|
| `src` | the source text, exactly as written |
| `val` | the resolved value — a real number for `#NR`, the unquoted string for `#ST` |
| `name` `tin` | the token's name and its numeric id |
| `rI` `cI` | row and column, 1-based |

**`src` and `val` are not the same thing**, and picking the wrong one is a
common bug. `r.o0.src` for a number is the string `'42'`; `r.o0.val` is the
number `42`.

## The rules around you

| Field | What it is |
|---|---|
| `r.parent` | the enclosing rule instance |
| `r.child` | the rule instance that just closed beneath this one |
| `r.prev` | the previous instance when a rule repeats |

`r.child` is only meaningful in the close phase — going down, nothing has
closed yet. That is why the `pair` grammar above reads `r.child.node` in
`close` and not in `open`.

Both are always *something*: when there is no parent or child, they are a
sentinel rule whose `name` is empty, not `undefined`. So `r.child.name` is safe
to read; `r.child.name === 'val'` is the check to write, rather than a null
test.

## The value: `r.node`

`r.node` is what the rule carries, and the start rule's `r.node` is what
`parse()` returns.

A pushed rule's node is **seeded from its parent**. That is why
`@setval$` can write into `r.node` from inside `pair` and have the result land
in the enclosing map — they are the same object. It is also why
[`@reset$`](/docs/builtin-actions) exists: a rule that needs its own scalar
value has to clear the inherited one first.

## Scratch data: `u` and `k`

Two bags for your own use. They differ in exactly one way, and it is the
important one:

| Field | Scope |
|---|---|
| `r.u` | this rule instance only |
| `r.k` | this rule **and every rule pushed or repeated below it** |

Use `u` for something the rule needs between its own open and close phases —
the captured key in a `key = value` pair. Use `k` for configuration that a
whole subtree should see.

## Counters: `r.n`

`n` is a third bag, for numbers, and it propagates like `k`. It has comparison
helpers, and a trap:

| Helper | True when |
|---|---|
| `r.eq('k', n)` | the counter equals `n` |
| `r.lt` `r.lte` | below / at most `n` |
| `r.gt` `r.gte` | above / at least `n` |

**An unset counter reads as `0`** — it has counted nothing. So `r.lt('depth', 3)`
is true before anything is counted, `r.gt('depth', 99)` is false, and exactly
one of `<`, `=`, `>` holds. `r.exist('k')` asks whether the counter was set at
all, which the comparisons cannot: one set to `0` and one never set compare
identically. [Conditions and counters](/docs/conditions) covers it properly.

> Before 0.6 an unset counter compared as **true against every one of them**,
> so `lt` and `gt` were both true and a guard could fire on the first token.

## Reading it live

You do not have to guess at any of this:

Two `add` instances at the same depth is a repeat; increasing depth is a push.
See [debugging a grammar](/how-to/debug-a-grammar) for the full trace.

## Next

- [Builtin actions](/docs/builtin-actions) — building `r.node` without writing
  any code.
- [Conditions and counters](/docs/conditions) — using `r.n` and `r.u` to pick
  an alternate.
- [The rule table](/docs/rule-table) — the alternate fields that populate all
  of the above.

---

# Builtin actions

Source: https://tabnas.dev/docs/builtin-actions
Build a value with `$`-suffixed action names instead of code — and keep the grammar as pure data.

An action is normally a function. Functions are the reason a grammar stops
being data: it can no longer be serialised, diffed, stored, or accepted from
somewhere you don't trust.

The engine ships a set of actions referenced **by name**. A grammar using only
those is pure JSON with no code in it — and still builds a real value. This
tutorial builds a small JSON-shaped parser that way, with not one function in
the grammar.

## The naming rule

A ref is a string starting with `@`. A trailing `$` marks it as an engine
builtin:

The `$` namespace is **reserved** — `grammar()` refuses a user ref containing
`$`, so a builtin can never be shadowed by something a grammar brought with it.

## The value builders

Seven of them. This is the whole set you need for JSON-shaped data:

| Builtin | What it does |
|---|---|
| `@object$` | put a fresh empty object in `r.node` |
| `@array$` | put a fresh empty array in `r.node` |
| `@value$` | resolve the matched scalar token into `r.node` |
| `@key$` | capture the matched key token into `r.u.key` |
| `@setval$` | `r.node[r.u.key] = r.child.node` |
| `@push$` | append `r.child.node` to the array in `r.node` |
| `@reset$` | clear `r.node` back to "no value" |

`@key$` and `@setval$` are a pair: the first stores the key on the way in, the
second uses it on the way out, once the child has produced a value.

## A whole grammar, as data

Search that object for the word `function`. There isn't one.

## Why it works: the seeded node

The part that looks like magic is `@setval$` writing into `r.node` from inside
`pair`, and the result appearing in the map.

A pushed rule's node is **seeded from its parent**. `@object$` runs on `val`
and puts an object in `val.node`; `val` pushes `map`, which inherits it;
`map` pushes `pair`, which inherits it too. All three names refer to the same
object, so `@setval$` writing a property from `pair` is writing into the object
`val` will return.

Once that clicks, the rest of the table reads straightforwardly:

- `@key$` stores the key on `r.u` — the **non**-propagating bag, so a nested
  pair cannot clobber an outer one's key.
- `@setval$` runs in the *close* phase, because `r.child.node` does not exist
  until the child has closed.
- `@push$` skips a child with no value, so a trailing comma adds nothing.

## Configuring a builtin

Config rides on the alternate's `k`, keyed by the builtin's name:

`from` is which open token to read (default `0`), `slot` is the `r.u` key to
store under (default `'key'`). Both defaults are what the grammar above relies
on, which is why it doesn't mention them.

## Several actions on one alternate

`a` takes an array, run in order:

That is the idiom for a rule that must not inherit its parent's node: clear
first, then build.

## The catch: install mutates the spec

`grammar()` resolves ref strings **in place**. After installing, the `a` fields
hold functions, not strings:

```ts
typeof spec.rule.val.open[0].a       // => 'function'
JSON.stringify(spec).includes('@object$')   // => false
```

So serialise **before** you install, or install a copy:

Getting this the wrong way round produces a spec that silently loses every
action — the grammar still installs, still parses, and returns `undefined`.

## The other families

Two more sets exist, and you are unlikely to write either by hand:

- **Tree builders** — `@node$`, `@capture$`, `@bubble$`, `@fold$` build the
  `{ rule, src, kids }` AST. This is what `@tabnas/abnf` emits, and it is why
  an ABNF grammar can compile to pure data too.
- **Probe dispatch** — `@probeInit$`, `@probeDecide$`, `@probePhase0$/1$/2$`
  implement the optional-prefix disambiguation the ABNF compiler needs.

Both are versioned by `BUILTIN_SCHEMA_VERSION`. A serialized grammar can
declare the schema it was compiled against as `v`, and the engine refuses one
that needs a newer version than it implements — so an old engine fails loudly
rather than mis-parsing.

## When to use code instead

Builtins cover building values. They do not cover *computing* them. The moment
you need arithmetic, validation, or a call into your own code, write a
function — and accept that the grammar is now code.

The useful middle ground is [named refs](/docs/actions#named-refs--code-bound-out-of-band):
the grammar keeps ref *names*, and the functions are supplied separately at
install time. The grammar text stays declarative; only the binding is code.

## Next

- [The rule instance](/docs/rule-instance) — what `r.node`, `r.u` and
  `r.child` are.
- [Attaching actions](/docs/actions) — the three ways, and when each is right.
- [The rule table](/docs/rule-table) — every alternate field.

---

# Conditions and counters

Source: https://tabnas.dev/docs/conditions
Pick an alternate on state rather than tokens — with a predicate, a counter, or a declarative check.

Alternates are chosen by their tokens. When two cases look identical to the
lexer, the difference has to come from somewhere else — how deep you are, what
the enclosing rule is, whether something has already been seen.

That is what `c` is for. This tutorial covers the three ways to write one, and
how counters read before anything has been counted.

## A condition is a predicate

`c` is checked when an alternate's tokens match. If it returns false, the
alternate is skipped and the next one is tried:

Everything on [the rule instance](/docs/rule-instance) is available: `r.o0` for
the token that opened this rule, `r.parent`, `r.child`, `r.n`, `r.u`.

## Counters

`n` on an alternate sets or increments a named counter, and counters
**propagate to pushed and repeated rules** — so a count made at the top is
visible all the way down:

Setting `0` **resets**; any other number adds. `n: { pk: 0 }` in the JSON
grammar is a reset, and reading it as a no-op will mislead you.

The rule instance has comparison helpers:

| Helper | True when |
|---|---|
| `r.eq('k', n)` | the counter equals `n` |
| `r.lt('k', n)` `r.lte` | below / at most `n` |
| `r.gt('k', n)` `r.gte` | above / at least `n` |

## An unset counter reads as zero

A counter that has never been incremented has counted nothing, so it compares
as `0`:

That keeps the permissive direction you want — a rule that never counts is not
blocked by a limit it knows nothing about — while leaving exactly one of `<`,
`=`, `>` true, so a guard means what it says wherever you put it.

`eq('k', 0)` is therefore true both for a counter set to `0` and for one never
set. When the difference matters, ask directly:

> **Changed in 0.6.** Previously an unset counter compared as **true against
> every helper**, so `lt('depth',3)` and `gt('depth',3)` were both true and a
> `$gte` guard fired on the very first token, before anything had been counted.
>
> An alternate gated on `{ 'n.k': { $gt: 0 } }` used to match on **two**
> grounds: the counter was positive, *or* it was unset and the comparison
> passed regardless. Only the second is gone. If you relied on both, add the
> unset case as its own alternate — do not swap the condition, or you lose the
> positive-counter branch you actually wanted:
>
> ```ts
> { s: '#CB', c: { 'n.k': { $gt: 0 } },        b: 1 }   // keep this
> { s: '#CB', c: { 'n.k': { $exist: false } }, b: 1 }   // add this if you
>                                                        // relied on fail-open
> ```
>
> Paths that are not counters are unaffected: an absent `o0` or a `u.*` you
> never set is genuine absence, not zero.

## A depth limit, done properly

Nesting is allowed while `depth` is below the limit. Past it, neither push
alternate matches, and the unconditional guard behind them is reached:

Note the order: the two push alternates carry the condition, and the guard is
*after* them. Alternates are tried in order, so the first one that matches
wins — put the guard first and it claims the token before the push alternates
are ever considered.

A `$gte` guard first now also works, since an unset `depth` reads as `0` and
`0 >= MAX` is false at the opening brace. Before 0.6 it did not: `$gte` passed
unconditionally while the counter was unset, so every parse failed on the first
token. Ordering the permissive alternates first is still the clearer habit —
it does not depend on how the counter reads when nothing has been counted.

## Declarative conditions

`c` also takes an **object**, which is a check against a path on the rule
instance. This keeps a grammar as data — no closure, so it still serialises:

The key is a dot-path resolved against `r`, so `n.depth` is the counter,
`u.mode` is your own scratch value, `o0.tin` is the opening token.

| Form | Meaning |
|---|---|
| `{ 'n.depth': { $lt: 3 } }` | below |
| `$lte` `$gt` `$gte` | at most / above / at least |
| `{ 'u.mode': 'strict' }` | a bare value is `$eq` |
| `{ 'u.mode': { $ne: 'loose' } }` | not equal |

Several keys are **ANDed** — every one must hold:

### The two halves behave differently

This is worth pinning down, because the asymmetry is invisible in the syntax:

So counters compare as numbers from zero, `$eq` on a non-counter path fails
closed, and the ordered ops fail open only for paths that are not counters.
Use `$exist` when you mean "only if this was explicitly set" — on a counter
that is the sole way to tell "never counted" from "counted zero".

## Which form to use

| If you want | Use |
|---|---|
| A grammar that stays serialisable | the object form |
| A grammar an agent emitted, checkable before running | the object form |
| Anything the object form can't express | a function |

The object form covers comparisons against rule state, which is most guards.
Anything involving two paths, arithmetic, or a call into your own code needs a
function — and the grammar becomes code at that point.

## Next

- [The rule instance](/docs/rule-instance) — every path the object form can
  reach.
- [Choose between alternates](/how-to/choose-between-alternates) — order,
  lookahead and group tags, the other ways to pick a branch.
- [Give good parse errors](/how-to/parse-errors) — the `e` alternate the guard
  above uses.

---

# ABNF grammars

Source: https://tabnas.dev/docs/abnf-grammars
Define a language fast using the RFC 5234 ABNF dialect.

The `@tabnas/abnf` plugin compiles [RFC 5234](https://www.rfc-editor.org/rfc/rfc5234)
ABNF straight into a working grammar. It's the fastest way to define a language.

## Dialect

tabnas uses the RFC 5234 dialect: `=` for definitions and `/` for
alternatives (not `::=` or `|`).

- **Literals**: `"+"`, case-insensitive by default; `%s"Hi"` is case-sensitive.
- **Optional**: `[ … ]`.
- **Repetition**: `*element` (zero or more), `1*element` (one or more),
  `2*4element` (bounded).
- **Grouping**: `( … )`.
- **Char ranges**: `%x30-39`.
- **Built-in tokens** by bareword: `NR` (number), plus core rules like `ALPHA`
  and `DIGIT`.

## Left recursion

Left-recursive rules are accepted directly. A left-recursion pass ([Paull's](https://en.wikipedia.org/wiki/Left_recursion#Removing_all_left_recursion) — Wikipedia describes the method without using the name)
algorithm) rewrites both direct (`P = P a / b`) and indirect recursion into the
iterative form the engine runs without re-entering a rule at the same position:

```
P = P a / b     →     P = b *(a)
```

Because it's a rewrite, the tree is flat rather than left-nested, and a
**purely** left-recursive rule (no base branch) is an error. See the
[@tabnas/abnf README](https://github.com/tabnas/abnf#readme) for the full
details and caveats.

## Actions

Attach behaviour with `@ref` action references, passed as `actions` and keyed
by rule and phase. There are two forms:

- `'@add:o:NR'` — an **alternate** action: runs when the `add` rule opens on an
  `NR` token. The trailing mark is the alternate's leading discriminator.
- `'@add:bo'` — a **rule-phase** hook: before-open. Also `ao`, `bc` and `ac`
  for after-open, before-close and after-close.

`r.parent` is `val` for every repetition: a tail self-reference like
`[ PL add ]` compiles to a same-depth repeat of `add`, not a nested push.

`r` is the rule instance and `r.o` the tokens matched in the open phase, so
`r.o[0].val` is the value of the first — a real number, courtesy of the lexer.

### Finding the marks

An alternate mark comes from the alternate's leading discriminator, which the
compiler assigns — so don't guess at the name, ask for it:

```bash
tabnas-abnf --marks -f grammar.abnf
```

```
val  o:add  p:add
val  c:_  (empty)
add  o:NR  s:#NR
add  c:PL  s:#PL
add  c:_  (empty)
```

`markListing(spec)` gives the same listing from code. Reading it also tells
you the shape the compiler produced: `val o:add p:add` says `val` pushes
`add` as a child rule, and `add c:PL s:#PL` says `add` repeats itself from
its close phase when a `+` follows — the tail self-reference `[ PL add ]`
compiled to a same-depth repeat, exactly what a hand-written grammar
declares as `{ s: '#PL', r: 'add' }`.

Requires `@tabnas/abnf` 0.3.0 or later (and `@tabnas/parser` 0.5.0). Earlier
versions compiled the tail into generated option/group rules, nesting each
repetition; 0.2.x additionally dissolved pure aliases like `val = add`.

### Parents and generated rules

For a tail self-reference, `r.parent` is the wrapping rule for **every**
repetition — accumulating onto `r.parent.node` is the intended idiom, the
same as in a hand-written rule table.

Other sugar still desugars into generated group rules: inside `( … )` or
`*( … )` a rule's `parent` at runtime may be a `_gen*` rule you didn't
write. The mark listing shows the compiled rule set when in doubt.

---

# Attaching actions

Source: https://tabnas.dev/docs/actions
Turn a parse into a value — with builtins, named refs, or inline functions.

A grammar that only recognises input returns nothing useful. Actions build the
result. There are three ways to attach them; prefer them in this order.

## Builtins — no code at all

The engine ships `$`-suffixed action builtins. Referenced by name, they let a
grammar stay pure data:

Nothing in that grammar is a function, so it round-trips through JSON. See
[the rule table](/docs/rule-table#builtin-actions) for the full set.

## Named refs — code, bound out of band

The grammar text stays untouched; behaviour binds through names the compiler
assigns. This is the form to use with ABNF, because it keeps the ABNF valid
RFC 5234.

`r.parent` is `val` for **every** repetition, because the compiler turns the
tail self-reference `[ PL add ]` into a same-depth repeat — the same shape as
the inline example below. The total lives in one place, on `val`'s node,
where `parse` returns it; the instance carries no state between calls.

Two kinds of name:

- **Alternate marks** — `@<rule>:<phase>:<mark>`, where the mark comes from the
  alternate's leading discriminator. Open-phase marks fire as tokens are
  matched; close-phase marks (like the repeat's own `@add:c:PL`) fire on the
  way back up.
- **Rule-phase hooks** — `@<rule>:bo`, `:ao`, `:bc`, `:ac` for before/after
  open and close.

### Finding the mark names

Marks are assigned by the compiler. Don't guess:

```bash
tabnas-abnf --marks -f grammar.abnf
```

```
val  o:add  p:add
val  c:_  (empty)
add  o:NR  s:#NR
add  c:PL  s:#PL
add  c:_  (empty)
```

If you attach an action to a mark that doesn't exist, the compiler says so
rather than failing silently:

```
AbnfActionError: abnf: action ref '@item:o:ALPHA' matches no open alt
with mark 'ALPHA' in rule 'item'
```

That error is usually caused by desugaring: `( … )` and `*( … )` compile to
generated group rules, so their marks belong to rules you didn't write. (A
tail self-reference like `[ PL add ]` is the exception — it compiles to a
repeat on the rule itself, which is why `add` owns the `c:PL` mark above.)
The listing is the authority.

## Inline functions — the most direct

Written straight onto the alternate as `a`:

Two one-line actions. It works because `r` **repeats** `add` at the same stack
depth rather than pushing it, so every `add` shares one parent and the total
lives in one place. The result rides on the parse rather than an outer
variable, so `parse()` returns it and concurrent parses can't collide.

The cost: the grammar is now code. It can't be serialised, printed back as
ABNF, or safely accepted from anywhere you don't trust.

## Which to use

| If you want | Use |
|---|---|
| A grammar that survives JSON round-tripping | Builtins |
| A grammar an agent wrote, that you want to check before running | Builtins |
| ABNF that stays valid RFC 5234 | Named refs |
| Full control of the rule table | Inline functions |

## A note on `r.parent`

For a tail self-reference (`add = NR [ PL add ]`) the compiler emits a
same-depth repeat, so `r.parent` is the wrapping rule for every repetition
and accumulating onto `r.parent.node` is the intended idiom — in ABNF and
hand-written grammars alike.

Other sugar is different: `( … )` and `*( … )` still compile to generated
group rules, so inside those a rule's `parent` may be a rule you didn't
write. When in doubt, `tabnas-abnf --marks` shows the compiled rule set.

---

# Extending a grammar

Source: https://tabnas.dev/docs/extending
Add to a language that already parses, instead of forking it.

Extension is the point of the engine, and the cheapest way to get a parser. The
question to ask before writing a grammar is always: *what already parses
something close to this?*

## Derive, don't mutate

`make()` produces a fresh instance. Changes to it leave the original alone, so
other code using the base parser is unaffected.

## Add a plugin

The common case. A plugin is a function that modifies a grammar — adding rules,
adding alternates to existing rules, registering tokens.

[A grammar with plugins](/docs/grammar-with-plugins) walks through this in
full. The available plugins are on the [packages](/docs/packages) page.

## Inspect what you started with

Before changing a grammar, look at it. `rule()` with no arguments lists the
rules on an instance:

Five rules — that is the whole of JSON's structure, and the reason extending it
is tractable. For more, [@tabnas/debug](https://github.com/tabnas/debug)
describes a live grammar and prints it back as ABNF, and
[@tabnas/railroad](https://github.com/tabnas/railroad) draws it.

## Remove a rule

Passing `null` prunes a rule. This is how a stricter dialect is built from a
looser one — the JSON grammar is partly jsonic with the relaxations removed.

## What extension looks like in practice

The published grammars are the worked examples, and most are short because each
builds on the last:

| Package | Extends | By adding |
|---|---|---|
| [json](https://github.com/tabnas/json) | the engine | the five rules of JSON |
| [jsonc](https://github.com/tabnas/jsonc) | json | comments |
| [jsonic](https://github.com/tabnas/jsonic) | json | unquoted keys, implicit structure, trailing commas |
| [csv](https://github.com/tabnas/csv) | jsonic | record and field rules |
| [ini](https://github.com/tabnas/ini) | jsonic | sections and `key=value` |

Reading one of these diffs is the fastest way to understand the mechanism.

## When not to extend

If the thing you're parsing shares no structure with anything published, start
from [your first grammar](/docs/first-grammar) instead. Extension is cheap when
there's a real relationship and confusing when there isn't — a grammar
pretending to descend from JSON because JSON was nearby will fight you.

## See also

- [The rule table](/docs/rule-table) — what a plugin manipulates.
- [Attaching actions](/docs/actions) — adding behaviour to rules you didn't write.
- [Examples](/examples) — two languages built this way.

---

# The rule table

Source: https://tabnas.dev/docs/rule-table
The grammar format the engine walks — rules, phases, alternates, and the fields on each.

A grammar is a table. ABNF compiles to it, plugins build it programmatically,
and you can write it directly as data. This page is the format.

## Shape

Each rule has an **open** phase (on the way down) and a **close** phase (on the
way back up). Each phase holds a list of **alternates**, tried in order. The
first one whose token pattern matches wins.

## Alternate fields

| Field | Meaning |
|---|---|
| `s` | Match this token sequence. One token, or several for lookahead. |
| `p` | **Push** a child rule. It nests: the child's `parent` is this rule. |
| `r` | **Repeat** a rule at the same stack depth. No nesting; the parent is unchanged. |
| `a` | Action — a function, a `@ref` name, a `$`-builtin, or an array of them. |
| `c` | Condition; the alternate only applies when it holds. |
| `{}` | The empty alternate. Ends the phase. |

`p` versus `r` is the distinction worth internalising. Push builds depth, so
`a+b+c` nests three levels. Repeat stays flat, so every repetition shares one
parent — which is what lets an accumulator be a single value in a single place.

**Every phase needs a way out.** If no alternate matches, that is a parse
error. `{}` matches anything and does nothing, which is how a rule ends.

## The rule instance

Actions receive `r`, the current rule instance:

| Property | What it is |
|---|---|
| `r.node` | The value this rule carries. What `parse()` returns for the start rule. |
| `r.o` | Tokens matched in the open phase. `r.o[0].val` is the first token's value, `r.o[0].src` its source text. |
| `r.parent` | The enclosing rule instance. |
| `r.child` | The rule instance just closed beneath this one. |

## Tokens

Some tokens exist before you declare anything:

| Token | Matches |
|---|---|
| `#NR` | A number. `val` is a real number, not a string. |
| `#TX` | Bare text. |
| `#ST` | A quoted string. |
| `#CA` | `,` |
| `#CL` | `:` |
| `#OB` `#CB` | `{` `}` |
| `#OS` `#CS` | `[` `]` |
| `#SP` `#LN` | Space, newline. |
| `#ZZ` | End of source. |

Declare your own under `options.fixed.token`. Pick a name that isn't taken —
`#CM` is *comment*, not comma, and silently redefining it will cost you an
afternoon.

## Builtin actions

The engine ships `$`-suffixed action builtins, merged into the ref map when the
grammar loads. Because they are referenced by name, a grammar using only these
is **pure JSON with no functions in it** — serialisable, diffable, and safe to
accept from somewhere else.

| Builtin | Effect |
|---|---|
| `@object$` | `r.node = {}` |
| `@array$` | `r.node = []` |
| `@key$` | Capture the matched key token. |
| `@setval$` | Assign the child's node as an object property. |
| `@push$` | Append the child's node to an array. |
| `@value$` | Resolve the matched scalar token. |
| `@reset$` | Clear the parent-seeded node. |
| `@node$` `@capture$` `@bubble$` | Rebuild the `{ rule, src, kids }` tree. Used by the ABNF compiler. |

## Constraints

These are properties of the machine, not gaps to be filled later:

- **Deterministic dispatch.** Alternates are tried in order and the first
  match wins, so two alternates that can't be told apart from their leading
  tokens will resolve to whichever comes first.
- **No backtracking.** One path through, or an error.
- **No ambiguity.** One parse per input.
- **A hand-written grammar has no end-of-source check.** The ABNF compiler adds
  a `__start__` wrapper that consumes `#ZZ`; if you write the table yourself,
  trailing input can be silently ignored unless you handle it.

## See also

- [ABNF grammars](/docs/abnf-grammars) — the notation that compiles to this.
- [Attaching actions](/docs/actions) — the three ways to bind behaviour.
- [How it works](/docs/how-it-works) — why the machine has this shape.

---

# Packages

Source: https://tabnas.dev/docs/packages
Every package in the project — the engine, the grammar tooling, and all the ready-made grammars.

Every package is published to npm under the `@tabnas/*` scope and, where
applicable, as a Go module at `github.com/tabnas/<name>/go`. Each ships
four-quadrant [Diátaxis](https://diataxis.fr) docs in both languages.

Grammars are plugins over other grammars, so the list below is closer to a
dependency order than a catalogue: JSON is a handful of rules, JSONC adds
comments to them, JSON5 and jsonic relax the quoting, and so on. Extending one
of these is usually cheaper than starting a new grammar.

<PackageTable />

Health and CI status for every repository is on the
[org status dashboard](https://tabnas.github.io/status/).

---

# How it works

Source: https://tabnas.dev/docs/how-it-works
The ideas behind the engine — grammar as parser, a uniform tree, two runtimes.

tabnas is built on a few deliberate choices. Understanding them explains why the
API is small and why the same grammar behaves identically across languages.

## The grammar is the parser

Most parser tools generate code from a grammar, then ask you to keep the
generated code in sync. tabnas doesn't generate anything: a grammar is a data
structure the engine executes directly. That means:

- No build step and nothing to fall out of sync.
- You can build grammars at runtime (for example, compiled from ABNF).
- You can inspect a live grammar — describe it, render it back to ABNF, or draw
  it as a railroad diagram — because it's just data.

## A uniform tree

Parsing always produces the same node shape:

`rule` is the grammar rule that matched, `src` is the source it covered, and
`kids` are the child nodes. Because the shape is invariant, tooling written for
one grammar works for all of them.

## One grammar, two runtimes

The engine has two implementations: TypeScript (canonical) and Go (tracking).
They compile the **same** grammar fixtures and must produce the **same** trees —
this is enforced in CI. Practically, you can define a language once and run it
in a Node service and a Go binary with confidence they agree.

Some differences are intentional where a runtime's API can't mirror the other;
those are documented per package.

## Scope

tabnas is the shortest path from "I have a grammar" to "I have a parser I can
trust, everywhere I run." It is not a full compiler toolchain, and it isn't the
fastest parser for every workload — it optimises for correctness, portability,
and a grammar you can read.

---

# Include one source from another

Source: https://tabnas.dev/how-to/include-other-sources
Splice a file, a package or an in-memory string into a parse at the point it is referenced.

Almost every configuration format grows an include statement. The requirement
is always the same shape: a mark in the source names another source, and the
value it parses to is spliced in at that point.

`@tabnas/multisource` is that feature, finished. It is worth reading how it is
built, because the mechanism underneath — `@tabnas/directive` — is how you
would add *any* statement that triggers custom parsing.

## The finished answer

The plugin needs a **resolver**: a function that turns a path into source text.
Two ship with the package, and the in-memory one is the easiest to see:

```ts

const tn = new Tabnas().use(jsonic).use(MultiSource, {
  resolver: makeMemResolver({
    'base.jsonic': 'port: 8080, host: localhost',
  }),
})

tn.parse('@"base.jsonic"')
// => { port: 8080, host: 'localhost' }
```

The mark is `@` by default (`markchar` changes it), and the path is an ordinary
value in the host grammar — so it obeys that grammar's quoting rules.

## Where the mark can go

Anywhere a value can go, plus the top of a document:

```ts
tn.parse('cfg: @"base.jsonic"')
// => { cfg: { port: 8080, host: 'localhost' } }

tn.parse('[ @"base.jsonic", 2 ]')
// => [ { port: 8080, host: 'localhost' }, 2 ]

tn.parse('{ @"base.jsonic", extra: 1 }')
// => { port: 8080, host: 'localhost', extra: 1 }
```

The third form is the interesting one. In *key* position the included map's
keys are merged into the enclosing map rather than nested under a key, which is
what makes an include statement feel like an include statement.

## Later wins

Merging is ordinary object merging in source order, so an include followed by a
key overrides that key. This is the whole of "environment overlays":

```ts
const tn = new Tabnas().use(jsonic).use(MultiSource, {
  resolver: makeMemResolver({
    'base.jsonic': 'port: 8080\nhost: localhost',
    'dev.jsonic': '@"base.jsonic"\nport: 3000',
  }),
})

tn.parse('@"dev.jsonic"')
// => { port: 3000, host: 'localhost' }
```

Includes nest: `dev.jsonic` pulls in `base.jsonic` while itself being pulled in.

## Reading real files

`makeFileResolver()` reads from disk, and `path` sets the base directory that
relative references resolve against:

```ts

const tn = new Tabnas().use(jsonic).use(MultiSource, {
  resolver: makeFileResolver(),
  path: 'cfg',
})

tn.parse('@"dev.jsonic"')   // reads cfg/dev.jsonic
```

A nested include resolves against *its own* file's directory, not the entry
point's — the usual expectation, and the reason `base` is tracked per source
rather than globally. `makePkgResolver()` does the same job through
`require.resolve`, so a reference can name a published package.

The extension may be left off. `@"base"` searches `base.jsonic`, `base.jsc`,
`base.json` and `base.js`, then the same four as folder index files —
`base/index.*` and `base/index.base.*`.

## Deciding what a file means

The extension picks a **processor**, and processors are just functions that set
`res.val`. Adding a kind is adding a key:

```ts
const tn = new Tabnas().use(jsonic).use(MultiSource, {
  resolver: makeMemResolver({
    'notes.txt': '  hello  ',
    'rows.csv': 'a,b\n1,2',
  }),
  processor: {
    txt: (res) => { res.val = res.src.trim() },
    csv: (res) => { res.val = res.src.trim().split('\n').map((l) => l.split(',')) },
  },
})

tn.parse('note: @"notes.txt"')
// => { note: 'hello' }

tn.parse('rows: @"rows.csv"')
// => { rows: [ [ 'a', 'b' ], [ '1', '2' ] ] }
```

Out of the box: `.jsonic` and `.jsc` parse with the host instance, `.json`
parses as strict JSON, `.js` is evaluated as a module, and anything else is
inserted as a raw string.

## Knowing what was read

Pass a `deps` object in the parse metadata and it comes back filled in — which
is how a build tool knows what to watch:

```ts
const deps = {}
tn.parse('@"dev.jsonic"', { multisource: { deps } })

Object.keys(deps['dev.jsonic'])
// => [ 'base.jsonic' ]
```

The map is keyed by the *including* source's full path. The outermost entry is
keyed by an exported `TOP` symbol rather than a string, so
`Object.keys(deps)` will not show it; import `TOP` from the package and index
with it.

**There is no cycle detection.** Two files that include each other will recurse
until the stack overflows, with a `RangeError` rather than a parse error. If
your inputs are not trusted to be acyclic, walk `deps` yourself before or
during the parse.

## A missing source is a parse error

Not an exception from the file system — an error positioned at the reference,
listing where it looked:

```
[jsonic/multisource_not_found]: source not found: nope.jsonic
  --> <no-file>:1:1
```

## Building your own

Underneath, `MultiSource` is a `Directive`: a token that opens a rule, parses a
value, and hands you the result. The whole of a `$NAME` environment lookup is
one call.

```ts

const env = { HOME: '/home/dev', PORT: '8080' }

const tn = new Tabnas().use(jsonic).use(Directive, {
  name: 'env',
  open: '$',
  action: (rule) => { rule.node = env[String(rule.child.node)] },
})

tn.parse('home: $HOME, port: $PORT')
// => { home: '/home/dev', port: '8080' }
```

`open` is the token that starts the directive. `rule.child.node` is the value
that was parsed after it, and whatever you assign to `rule.node` is the value
the directive produced. Directives compose — a second `use(Directive, …)` adds
another, and they nest:

```ts
const tn = new Tabnas().use(jsonic)
  .use(Directive, { name: 'env',   open: '$', action: (r) => { r.node = env[String(r.child.node)] } })
  .use(Directive, { name: 'upper', open: '^', action: (r) => { r.node = String(r.child.node).toUpperCase() } })

tn.parse('a: ^$HOME')
// => { a: '/HOME/DEV' }
```

A directive can also be **bracketed**, with a `close` token, which is what lets
it take a list of arguments:

```ts
const tn = new Tabnas().use(jsonic).use(Directive, {
  name: 'sum',
  open: 'sum<',
  close: '>',
  action: (rule) => { rule.node = rule.child.node.reduce((a, b) => a + b, 0) },
})

tn.parse('a: sum<1,2,3>, b: 9')
// => { a: 6, b: 9 }
```

By default a directive is accepted wherever a value is accepted. The `rules`
option narrows or widens that — `MultiSource` uses it to also allow the mark in
key position, which is how the merge-into-the-enclosing-map form works.

An include of your own is the `env` example with a file read and a recursive
`parse` in place of the object lookup, plus an error when the path is unknown:

```ts
const FILES = { 'base.jsonic': 'port: 8080, host: localhost' }

const tn = new Tabnas().use(jsonic).use(Directive, {
  name: 'include',
  open: '@',
  action: (rule, ctx) => {
    const path = String(rule.child.node)
    const src = FILES[path]
    if (null == src) return rule.parent.o0.bad('include_not_found', { path })
    rule.node = ctx.inst().parse(src)
  },
})

tn.options({
  error: { include_not_found: 'no such include: {path}' },
  hint: { include_not_found: 'Known includes: base.jsonic' },
})

tn.parse('cfg: @"base.jsonic"')
// => { cfg: { port: 8080, host: 'localhost' } }
```

Returning a token from `bad()` is how an action reports a parse error rather
than throwing — see [giving good parse errors](/how-to/parse-errors).

Having written that, use `MultiSource` instead. Resolution order, base paths,
implicit extensions, key-position merging and dependency tracking are the parts
that take the time, and they are already done.

## See also

- [Write a parameterised parser](/how-to/parameterised-parsers) — how
  `Directive` takes its options, and how to do the same.
- [Extending a grammar](/docs/extending) — the general form of adding to a
  grammar you didn't write.
- [@tabnas/multisource](https://github.com/tabnas/multisource) — resolvers,
  processors, preloading.

---

# Parse expressions with precedence

Source: https://tabnas.dev/how-to/expressions-with-precedence
Add infix, prefix, suffix and ternary operators to a grammar, with a binding-power scale you control.

Precedence is the one part of parsing that rule tables are bad at. Expressing
`1+2*3` as nested rules means one rule per precedence level, and adding a level
means rewriting the chain. `@tabnas/expr` does it with a
[Pratt parser](https://matklad.github.io/2020/04/13/simple-but-powerful-pratt-parsing.html)
instead: every operator carries two numbers, and precedence falls out of
comparing them.

Register it on a grammar that already defines values, and values become
expressions:

```ts

const tn = new Tabnas().use(jsonic).use(Expr)

tn.parse('1+2*3')
```

## What comes back

An S-expression: an array whose first element is the operator and whose
remaining elements are the terms. The operator is an `Op` object — it carries
the source text, the binding powers, and which fixity matched — so printing a
tree usually means replacing it with `op.src` first:

```ts
// Replace each Op with its source text, for display.
const S = (x) =>
  Array.isArray(x) && x.length
    ? [x[0].src || x[0].osrc || S(x[0]), ...x.slice(1).map(S)]
    : x

S(tn.parse('1+2*3'))     // => [ '+', 1, [ '*', 2, 3 ] ]
S(tn.parse('-1+2'))      // => [ '+', [ '-', 1 ], 2 ]
S(tn.parse('2+3+4'))     // => [ '+', [ '+', 2, 3 ], 4 ]
S(tn.parse('(1+2)*3'))   // => [ '*', [ '(', [ '+', 1, 2 ] ], 3 ]
```

Two things worth noticing. `2+3+4` groups to the left, because addition's
binding powers say so. And the parenthesis is **kept** in the tree as an
operator of its own rather than dissolved — the tree records that the source
was written with brackets, which matters if you are formatting it back out.

Expressions live wherever values live, so this needs nothing extra:

```ts
S(tn.parse('[1+2, 3*4]'))
// => [ [ '+', 1, 2 ], [ '*', 3, 4 ] ]
```

## The binding-power scale

Each operator gets a `left` and a `right` number. When two operators compete
for a term, the larger binding power wins. The default table is spaced in
millions so there is room to insert between levels:

| Operator | `left` | `right` |
|---|---|---|
| `+` `-` prefix | — | 4000000 |
| `*` `/` `%` | 3000000 | 3100000 |
| `+` `-` infix | 2000000 | 2100000 |

**Associativity is the relationship between the two numbers on one operator.**
`left < right` binds to the left, which is what you want for arithmetic.
`left > right` binds to the right, which is what you want for exponentiation
and assignment.

## Adding operators

`op` is a map of definitions merged over the defaults, so you only name what
you are adding:

```ts
const tn = new Tabnas().use(jsonic).use(Expr, {
  op: {
    exponent:  { infix: true,  left: 5000000, right: 4900000, src: '**' },
    lt:        { infix: true,  left: 1400000, right: 1500000, src: '<' },
    and:       { infix: true,  left: 1000000, right: 1100000, src: '&&' },
    factorial: { suffix: true, left: 6000000,                 src: '!' },
  },
})

S(tn.parse('2**3**2'))     // => [ '**', 2, [ '**', 3, 2 ] ]
S(tn.parse('1<2 && 3<4'))  // => [ '&&', [ '<', 1, 2 ], [ '<', 3, 4 ] ]
S(tn.parse('3! + 1'))      // => [ '+', [ '!', 3 ], 1 ]
```

`**` binds right (`left > right`), so `2**3**2` is `2**(3**2)`. `&&` sits below
comparison, so comparisons bind tighter and become its operands. `!` is a
suffix at the top of the scale, so it takes `3` before `+` sees it.

Setting an operator to `null` removes it. Be aware of what that means: the
symbol is no longer an operator, but it is still lexable text, so in a relaxed
grammar like jsonic `1 % 2` becomes the implicit list `[1, '%', 2]` rather than
a syntax error. Removing an operator is not the same as forbidding it.

## Function-call and index syntax

A `paren` operator with a **preval** takes the value written immediately before
it as its first term. That is all a function call is:

```ts
const tn = new Tabnas().use(jsonic).use(Expr, {
  op: {
    call:  { paren: true, osrc: '(', csrc: ')', preval: { active: true } },
    index: { paren: true, osrc: '[', csrc: ']', preval: { required: true } },
  },
})

S(tn.parse('max(1,2)'))   // => [ '(', 'max', [ 1, 2 ] ]
S(tn.parse('f(1)'))       // => [ '(', 'f', 1 ]
S(tn.parse('a[1]'))       // => [ '[', 'a', 1 ]
```

`active` means the preceding value is used *if present*, so `(1+2)` still
groups. `required` means the operator only matches with one, which is what
keeps `[` working as a list bracket everywhere else.

Ternaries are declared with a two-element `src`:

```ts
const tn = new Tabnas().use(jsonic).use(Expr, {
  op: { ternary: { ternary: true, src: ['?', ':'], left: 1500000, right: 1400000 } },
})

S(tn.parse('a ? b : c'))   // => [ '?', 'a', 'b', 'c' ]
```

## Computing a value instead

Pass `evaluate` and the plugin reduces each node as it closes, so `parse`
returns the answer rather than the tree:

```ts
const math = (rule, ctx, op, terms) => {
  if (op.paren) return terms[0]
  if (op.prefix) return '-' === op.src ? -terms[0] : +terms[0]
  switch (op.src) {
    case '+': return terms[0] + terms[1]
    case '-': return terms[0] - terms[1]
    case '*': return terms[0] * terms[1]
    case '/': return terms[0] / terms[1]
    default:  return NaN
  }
}

const tn = new Tabnas().use(jsonic).use(Expr, { evaluate: math })

tn.parse('1+2*3')            // => 7
tn.parse('-(1+2)*3')         // => -9
tn.parse('a: 2*(3+4), b: 1') // => { a: 14, b: 1 }
```

Check `op.prefix` before switching on `op.src`: `-` is both a prefix and an
infix operator and they arrive at the same callback with a different number of
terms.

**Evaluating during the parse is a choice, not the default.** It is the right
one for a configuration language, where the result is a value. It is the wrong
one for anything that wants to inspect, rewrite or re-emit the source — keep
the tree and walk it afterwards.

## What this costs

The expression grammar is not a small addition. It brings its own rules,
counters and edge cases around implicit lists and maps, and it changes what
some inputs mean. On plain jsonic, `a: 1-2` parses to the string `'1-2'`;
install `Expr` and it is a subtraction. That is usually the point, but it is a
behaviour change to existing input, so install it on a derived instance
(`base.make().use(Expr)`) if other code depends on the original.

## See also

- [Handle recursion and repetition](/how-to/recursion-and-repetition) — what to
  do when the nesting isn't operator precedence.
- [Write a parameterised parser](/how-to/parameterised-parsers) — the option
  pattern `Expr` follows.
- [@tabnas/expr](https://github.com/tabnas/expr) — the full operator table and
  the Pratt implementation.

---

# Write a parameterised parser

Source: https://tabnas.dev/how-to/parameterised-parsers
One plugin, many dialects — take options and let them decide the tokens, the lexer and the rules.

Formats come in families. CSV is also TSV and also semicolon-separated. A
directive is `@` for one language and `$` for another. Writing a grammar per
member of the family is the wrong shape; a grammar that takes options is the
right one, and every published plugin is built that way.

## The shape

A plugin is a function of an instance and its options, with a `defaults`
property. The engine merges what the caller passed over the defaults, so a
plugin only ever reads one settled object:

```ts

// Recognise `250ms`, `10mb` — a number with one of a configurable set
// of unit suffixes — as a single value.
const Units = (tn, opts) => {
  tn.options({
    match: {
      value: {
        unit: {
          match: new RegExp(`^(\\d+)(${opts.suffix.join('|')})`),
          val: (res) => ({ n: +res[1], unit: res[2] }),
        },
      },
    },
  })
}

Units.defaults = { suffix: ['ms', 's', 'm', 'h'] }

new Tabnas({ plugins: [json] }).use(Units).parse('{"t": 250ms}')
// => { t: { n: 250, unit: 'ms' } }

new Tabnas({ plugins: [json] }).use(Units, { suffix: ['kb', 'mb'] }).parse('{"n": 10mb}')
// => { n: { n: 10, unit: 'mb' } }
```

The merge is deep, so a caller can override one leaf of a nested option without
restating the rest. The settled options are recorded on the instance, keyed by
the plugin's lowercased function name:

```ts
tn.internal().merged.plugin
// => { json: {}, units: { suffix: [ 'ms', 's', 'm', 'h' ] } }
```

That is worth knowing when a grammar misbehaves: it tells you what the plugin
actually ran with, rather than what you meant to pass.

## Derive, don't mutate

`use()` changes the instance it is called on. If anything else holds that
instance, parameterise a copy:

```ts
const base = new Tabnas({ plugins: [json] })
const derived = base.make().use(Units)

derived.parse('{"t":1s}')   // => { t: { n: 1, unit: 's' } }
base.parse('{"t":1}')       // => { t: 1 } — unchanged
```

## What options can reach

An option is only useful if it can change something. In practice there are five
levers, and `@tabnas/csv` — a grammar whose whole job is to be configurable —
pulls all of them.

### Redefine a token

The field separator is not a special case in the CSV grammar. The grammar is
written against `#CA`, and the option rebinds what `#CA` matches:

```ts
tn.options({ fixed: { token: { '#CA': options.field.separation } } })
```

Which is why the same grammar reads TSV:

```ts

new Tabnas().use(jsonic).use(Csv).parse('a,b\n1,2')
// => [ { a: '1', b: '2' } ]

new Tabnas().use(jsonic).use(Csv, { field: { separation: '\t' } }).parse('a\tb\n1\t2')
// => [ { a: '1', b: '2' } ]

new Tabnas().use(jsonic).use(Csv, { field: { separation: ';' } }).parse('a;b\n1;2')
// => [ { a: '1', b: '2' } ]
```

### Turn a lexer off

Whole categories of token are switches. CSV's `number`, `value` and `comment`
options are passed straight through:

```ts
new Tabnas().use(jsonic).use(Csv, { number: true }).parse('a,b\n1,2')
// => [ { a: 1, b: 2 } ]     — numbers, not strings

new Tabnas().use(jsonic).use(Csv, { comment: true }).parse('#note\na,b\n1,2')
// => [ { a: '1', b: '2' } ]
```

### Change what is ignored

Space and newline are in the `IGNORE` token set by default. CSV takes newline
out of it always, and space too in strict mode, because in that dialect they
are content. See
[parsing a line-oriented format](/how-to/line-oriented-formats).

### Include or exclude rules

Alternates carry group tags, and `rule.exclude` drops every alternate in a
group at derive time. Strict CSV switches off the embedded-JSON and implicit
structure rules with one option:

```ts
tn.options({ rule: { exclude: 'jsonic,imp' } })
```

### Wrap the parser

When the option changes the shape of the *result* rather than the grammar, wrap
`parser.start`. CSV's `stream` option does exactly this — records are handed to
a callback and the return value is empty:

```ts
const rows = []
const tn = new Tabnas().use(jsonic).use(Csv, { stream: (what, rec) => rows.push([what, rec]) })

tn.parse('a,b\n1,2\n3,4')
// => []

rows
// => [ [ 'start', null ],
//      [ 'record', { a: '1', b: '2' } ],
//      [ 'record', { a: '3', b: '4' } ],
//      [ 'end', null ] ]
```

## When the options *are* the grammar

`@tabnas/directive` is the extreme case: the plugin has no fixed syntax at all.
Every part of it — the token that opens it, the optional closing token, what it
does, and where it is allowed — arrives as an option, so the same plugin
installed twice gives two unrelated statements:

```ts

const env = { HOME: '/home/dev' }

const tn = new Tabnas().use(jsonic)
  .use(Directive, { name: 'env',   open: '$', action: (r) => { r.node = env[String(r.child.node)] } })
  .use(Directive, { name: 'upper', open: '^', action: (r) => { r.node = String(r.child.node).toUpperCase() } })

tn.parse('a: $HOME, b: ^hello')
// => { a: '/home/dev', b: 'HELLO' }
```

The `name` option is not decoration — it names the rule the plugin installs and
the counter it uses, which is what keeps two instances from colliding.

`@tabnas/expr` sits in between: the option is a *table*, merged over a default
table. Naming an operator overrides it, naming a new one adds it, and setting
one to `null` removes it. See
[parsing expressions with precedence](/how-to/expressions-with-precedence).

## Fail fast on bad options

A grammar that is misconfigured fails late and confusingly — usually as
"unexpected character" somewhere unrelated. Check what you depend on while you
still have a good message to give. `tn.rule()` with no arguments returns the
rule map, which is all a precondition needs:

```ts
const Suffix = (tn, opts) => {
  const rules = tn.rule()
  if (null == rules || null == rules.val) {
    throw new Error(
      "Suffix: the 'val' rule is missing; register a grammar that " +
      'defines it before this plugin',
    )
  }
  // …
}
```

`@tabnas/hoover` does exactly this, and the message it throws — *"the 'val'
rule is missing; register a grammar that defines it before the hoover
plugin"* — tells the caller the fix rather than the symptom. That is the
standard to aim for: the engine ships no grammar of its own, so "register a
grammar first" is the single most common mistake a plugin can catch.

## What this costs

Options are a public interface, and a deep-merged one is easy to grow and hard
to shrink. Two things keep it manageable: put every option in `defaults` so the
full surface is readable in one place, and prefer options that select between
behaviours over options that take a function — a callback is impossible to
serialise, diff, or accept from somewhere you don't trust, and the engine's
[data-first design](/docs/how-it-works) is the thing you would be giving up.

## See also

- [Include one source from another](/how-to/include-other-sources) —
  `Directive` in use.
- [Lex a token the engine doesn't know](/how-to/custom-tokens) — the
  `match.value` matcher used above.
- [Extending a grammar](/docs/extending) — `make()`, `use()` and pruning rules.

---

# Handle recursion and repetition

Source: https://tabnas.dev/how-to/recursion-and-repetition
Repeat without nesting, nest without recursing forever, and get left recursion past a push-down engine.

The engine is a push-down machine with no backtracking. It never re-enters a
rule at the same input position, which is what makes a parse linear and
predictable — and also what makes "a list of things" and "a thing inside a
thing" two different constructions rather than one.

Getting them the right way round is most of what a rule table gets wrong.

## Repetition is `r`

`r` runs a rule again **at the same stack depth**. Nothing nests, so every
repetition has the same parent, and an accumulator can live in one place:

```ts
tn.grammar({
  options: {
    fixed: { token: { '#PL': '+' } },
    rule: { start: 'val' },
  },
  rule: {
    val: {
      open:  [ { p: 'add', a: (r) => { r.node = 0 } } ],
      close: [ {} ],
    },
    add: {
      open:  [ { s: '#NR', a: (r) => { r.parent.node += r.o[0].val } } ],
      close: [ { s: '#PL', r: 'add' }, {} ],
    },
  },
})

tn.parse('1+2+3')   // => 6
```

`r.parent` is `val` for the first `add` and for every one after it, and
`r.d` — the stack depth — stays at 1. That is the property to reach for: if
what you are parsing is a *sequence*, `r` keeps it flat and the result lives
somewhere you can get at.

In ABNF this shape is written as a tail self-reference, and the compiler emits
the same repeat:

```ts
tn.abnf(`
  val = add
  add = NR [ PL add ]
  PL  = "+"
`)

tn.parse('1+2+3')
// => { rule: 'val', src: '1+2+3', kids: [
//      { rule: 'add', src: '1', kids: [] },
//      { rule: 'add', src: '2', kids: [] },
//      { rule: 'add', src: '3', kids: [] } ] }
```

Three siblings, not three levels. `*( … )` and `1*( … )` also repeat, but they
desugar into a generated group rule, so the repeated content is a child of a
rule you did not write — fine for recognition, awkward for actions. The tail
self-reference is the one that stays flat.

### The trap

`r` in a **close** phase replaces the *current* rule at its depth, so the
repetition's parent is the current rule's parent — not the current rule. Put a
repeat in the wrong phase and the accumulator you were reaching for is gone:

```ts
// WRONG: `line` replaces `doc`, so line's parent is whatever contained doc.
doc:  { close: [ { r: 'line' } ] }

// RIGHT: `line` replaces itself, so its parent is still doc.
doc:  { open: [ { p: 'line' } ] }
line: { close: [ { s: '#LN', r: 'line' }, {} ] }
```

The symptom is `r.parent.node` being `undefined` on the second repetition and
only the second. If you see that, this is why.

## Nesting is `p`

`p` **pushes** a child rule, so depth grows and the child's `parent` is the
rule that pushed it. Use it when the structure really is inside something else:

```ts
const tn = new Tabnas()
tn.options({
  fixed: { token: { '#OP': '(', '#CP': ')' } },
  rule: { start: 'val' },
})

const OP = tn.fixed('(')

tn.rule('val', (rs) => rs
  .open([
    { s: '#OP', p: 'val' },
    { s: '#NR', a: (r) => { r.node = r.o[0].val } },
  ])
  .close([
    // Only a val that opened on '(' may consume ')'.
    { s: '#CP', c: (r) => OP === r.o0?.tin, a: (r) => { r.node = [r.child.node] } },
    {},
  ]))

tn.parse('1')         // => 1
tn.parse('(1)')       // => [ 1 ]
tn.parse('(((1)))')   // => [ [ [ 1 ] ] ]
```

The condition on the close alternate is not optional decoration. Without it the
innermost `val` — the one that opened on the number — happily consumes the
first `)`, the outer one never sees its closing bracket, and the result is
`undefined` with no error. **A rule that can close on a delimiter must check
that it opened on the matching one.** `r.o0` is the first token the rule
matched, so its `tin` is that check.

## Left recursion

`P = P a / b` is the natural way to write a left-associative operator and the
one thing a push-down engine cannot run directly: `P` would re-enter itself
without consuming input. Write it in a hand-built rule table and you get
infinite recursion.

`@tabnas/abnf` accepts it anyway. A rewriting pass (Paull's algorithm) turns
direct and indirect left recursion into the iterative form before the grammar
is built:

```
P = P a / b     →     P = b *(a)
```

```ts
tn.abnf(`
  expr = expr PL term / term
  term = NR
  PL   = "+"
`)

tn.parse('1+2+3').kids.map((k) => k.rule)
// => [ 'term', 'term' ]
```

Three costs, all worth knowing before you rely on it:

- **The tree is flat, not left-nested.** The leading operand folds into the
  rule itself, so `1+2+3` yields two `term` children, not a left spine.
  Associativity has to be applied in an action.
- **`@ref` actions on the rewritten branches are look-up-only.** Attach actions
  to the sub-rules instead.
- **A purely left-recursive rule is an error.** `P = P a` with no base branch
  cannot be rewritten, and the compiler says so.

If you want operator precedence rather than a single left-associative rule,
don't write it as recursion at all —
[use `@tabnas/expr`](/how-to/expressions-with-precedence), which does it with
binding powers and no rule chain.

## Stopping unbounded nesting

Recursion that terminates on well-formed input still doesn't terminate on
hostile input. A counter (`n`) and a condition (`c`) put a ceiling on it. Here
the counter is added to the two alternates that push a structure, and a guard
alternate in front refuses to open another one past the limit:

```ts

const MAX = 3
const tn = new Tabnas({ plugins: [json] })

tn.options({
  error: { too_deep: 'nested deeper than {max} levels' },
  hint:  { too_deep: 'Deeply nested input is often hostile. Raise the limit if it is not.' },
})

tn.rule('val', (rs) => rs.open(
  [ { s: [['#OB', '#OS']], b: 1,
      c: (r) => !r.lt('depth', MAX),
      e: (r) => r.o0.bad('too_deep', { max: MAX }) } ],
  { custom: (alts) => (alts[1].n = alts[2].n = { depth: 1 }, alts) },
))

tn.parse('{"a":{"b":{"c":1}}}')        // => { a: { b: { c: 1 } } }
tn.parse('[1,2,3]')                    // => [ 1, 2, 3 ]
tn.parse('{"a":{"b":{"c":{"d":1}}}}')  // throws [tabnas/too_deep]
tn.parse('[[[[1]]]]')                  // throws [tabnas/too_deep]
```

Three details do the work. `s: [['#OB', '#OS']]` is *one* position matching
either token — a nested array is alternation, a flat one is a sequence.
Counters set with `n` propagate to pushed and repeated rules, so `depth` counts
levels rather than occurrences. And `b: 1` puts the token back, so the guard
inspects without consuming.

Note the double negative in the condition: an unset counter reads as `0`, so
`r.lt('depth', MAX)` is true at depth 0, and the guard wants the opposite.
(Before 0.6 an unset counter compared as `true` against *every* limit, so the
same reasoning had to hold in both directions at once.)

The `custom` modifier reaches into the host grammar's alternates by index,
which is a real coupling — indices 1 and 2 are the `map` and `list` pushes in
`@tabnas/json` as it stands today. Print `tn.rule('val').def.open` before and
after, and pin the version.

`options.rule.maxmul` is a different backstop, and worth knowing about: the
engine caps total rule steps at a multiple of the input length (`maxmul`
defaults to `3`), which catches a grammar that loops without consuming rather
than one that nests too far.

## See also

- [Choose between alternates](/how-to/choose-between-alternates) — `c`, `b` and
  multi-token lookahead in their own right.
- [The rule table](/docs/rule-table) — `p` versus `r`, and every alternate
  field.
- [ABNF grammars](/docs/abnf-grammars) — repetition notation and the
  left-recursion pass.

---

# Choose between alternates

Source: https://tabnas.dev/how-to/choose-between-alternates
Order, lookahead, conditions, counters and group tags — how the engine picks a branch, and how to make it pick yours.

Every rule phase is a list of alternates, tried in order, first match wins.
There is no backtracking: once an alternate is taken the parse commits to it.
That makes dispatch fast and predictable, and it makes *the order of your
alternates part of your grammar*.

Nearly every "why did it parse that way" question is one of the five tools
below.

## Order — best practice: most specific first

An alternate is taken only when its **whole** token sequence matches. A
sequence that fails partway costs nothing: the engine abandons it and tries the
next one, having consumed nothing.

So sharing leading tokens is not itself a problem. Here two alternates agree on
their first two tokens and differ on the third, and the input still reaches the
second one:

```ts
tn.rule('stmt', (rs) => rs.open([
  { s: ['#TX', '#CO', '#AR'], a: arrowForm },
  { s: ['#TX', '#CO', '#TX'], a: typed },
]))

tn.parse('a :: int')   // => typed — the first alternate failed on its third token
tn.parse('a :: ->')    // => arrowForm
```

What *does* cost you is a **shorter alternate placed before a longer one it is
a prefix of**. The short one matches in full, the parse commits, and the long
one never gets a turn. So the longer, more specific pattern goes first:

```ts
tn.rule('stmt', (rs) => rs.open([
  { s: ['#TX', '#CO', '#TX', '#AR'], p: 'tail', a: typedArrow },
  { s: ['#TX', '#CO', '#TX'],                   a: typed },
  { s: ['#TX'],                                 a: bare },
]))

tn.parse('a :: int -> b')   // => { kind: 'typed-arrow', name: 'a', type: 'int' }
tn.parse('a :: int')        // => { kind: 'typed',       name: 'a', type: 'int' }
tn.parse('a')               // => { kind: 'bare',        name: 'a' }
```

Reverse those three and `a` still parses, but `a :: int` does not. `{ s:
['#TX'] }` matched, the rule closed, and nothing is left that can accept a
`::`. The error is not much help either:

```
[tabnas/unexpected]: unexpected character(s):
  --> <no-file>:1:1
  1 | a :: int
      ^ unexpected character(s):
```

Column 1, and an empty character list. The cause is three alternates in the
wrong order; nothing in the message says so. When a grammar rejects input that
is obviously valid, alternate order is the first thing to check.

## Lookahead — as many tokens as you need

`s` is a *sequence*: the alternate matches only if all of its tokens match, in
order. The second example above looks four tokens ahead, and six works the same
way. There is no two-token limit — that claim appears in some older notes and
is wrong.

Lookahead is free in the sense that it does not backtrack: the tokens are
peeked, and the alternate either matches or the next one is tried.

What it does **not** do is re-lex. Lookahead peeks at tokens the lexer has
already produced, so the tokenisation is fixed before any alternate sees it —
an alternate cannot ask for the same characters to be read a different way. If
two constructs in your language need the same text lexed differently, that is a
lexer problem, not an alternate-ordering one: give them distinct tokens (see
[lexing a token the engine doesn't know](/how-to/custom-tokens)), or use a
matcher whose behaviour depends on the rule it is called from.

## Alternation inside one position

A nested array is "any of these", at a single position:

```ts
{ s: [['#OB', '#OS']] }   // one token: either { or [
{ s: ['#OB', '#OS'] }     // two tokens: { followed by [
```

That is the most common typo in a hand-written table, and it fails as
"unexpected character" on input that looks obviously valid.

## Conditions — `c`

When the tokens cannot tell two cases apart, the state can. `c` is a predicate
on the rule instance; the alternate only applies if it returns true:

```ts
// Only a val that opened on '(' may consume ')'.
{ s: '#CP', c: (r) => OP === r.o0?.tin, a: (r) => { r.node = [r.child.node] } }
```

`r.o0` is the first token matched in the open phase, `r.parent` the enclosing
rule, `r.child` the one that just closed. Anything reachable from the rule
instance is fair game.

## Counters — `n`

`n` sets or increments a named counter, and counters **propagate to pushed and
repeated rules** — so a counter set at the top is visible all the way down. The
comparison helpers read them:

| Helper | True when |
|---|---|
| `r.eq('k', n)` | counter equals `n` |
| `r.lt` `r.lte` | counter is below / at most `n` |
| `r.gt` `r.gte` | counter is above / at least `n` |

```ts
{ s: '#OB', p: 'map', n: { depth: 1 } }                     // count a level
{ s: '#OB', b: 1, c: (r) => !r.lt('depth', 3), e: tooDeep } // refuse past three
```

**An unset counter reads as `0`.** `r.lt('depth', 3)` is true before anything
is counted, `r.gt('depth', 3)` is false, and exactly one of `<`, `=`, `>`
holds — so a guard means what it says wherever you put it. Use
`r.exist('depth')` when you need to tell "never counted" from "counted zero";
the comparisons cannot.

> Before 0.6 an unset counter compared as true against *every* limit, so
> `r.lt('depth',3)` and `r.gt('depth',3)` were both true and guards written
> the obvious way fired on the very first token.

Setting a counter to `0` resets it rather than incrementing — `n: { pk: 0 }`
in the JSON grammar is a reset, not a no-op.

## Push-back — `b`

`b: n` returns `n` matched tokens to the stream. It is how an alternate can
*inspect* without *consuming*:

```ts
{ s: '#OB', p: 'map', b: 1 }   // decide to parse a map, let map read the '{'
```

The JSON grammar uses it for exactly that: `val` recognises `{`, pushes `map`,
and hands the brace back so `map` can match its own opening token.

## Group tags — `g`

Every alternate can carry group tags, and an instance can include or exclude
whole groups when it is derived. This is how one grammar ships several
dialects:

```ts
const mini = (tn) => {
  tn.options({ rule: { start: 'val' } })
  tn.rule('val', (rs) => rs.open([
    { s: '#NR', a: (r) => { r.node = r.o[0].val }, g: 'num' },
    { s: '#TX', a: (r) => { r.node = r.o[0].src }, g: 'text' },
  ]))
}

const base = new Tabnas({ plugins: [mini] })
base.parse('x')     // => 'x'

const strict = base.make({ rule: { exclude: 'text' } })
strict.parse('1')   // => 1
strict.parse('x')   // throws [tabnas/unexpected]
base.parse('x')     // => 'x' — the original is untouched
```

`rule.include` is the inverse: with any include set, only tagged alternates
that match survive. `@tabnas/csv` uses `exclude: 'jsonic,imp'` to turn its
strict mode on.

**Filtering happens when an instance is derived, and derivation re-runs
plugins.** Rules registered with a bare `tn.rule(…)` outside a plugin are *not*
carried into `make()` — the derived instance simply won't have them. Put the
grammar in a plugin function, as above, and this works; define it inline and it
silently doesn't.

## The empty alternate

`{}` matches anything and consumes nothing, which is how a phase ends. Every
phase needs one, or a way to reach a token that satisfies it — if no alternate
matches, that is a parse error.

It is also a trap in a close phase: an empty alternate will happily end a rule
that should have insisted on a closing token, producing `undefined` rather than
an error. If a rule has a required terminator, make the empty alternate
conditional or replace it with an
[error alternate](/how-to/parse-errors).

## Seeing which one fired

Guessing is optional. `@tabnas/debug`'s trace prints the alternate index chosen
at every step:

```
parse  "2"   ["+"]~[#PL]   2  . . alt=0  []   g:abnf   r:add
```

`alt=0` is the index into the phase's alternate list, `g:` its group tags, and
`r:`/`p:` what it did next. See [debugging a grammar](/how-to/debug-a-grammar).

## See also

- [The rule table](/docs/rule-table) — every alternate field in one table.
- [Handle recursion and repetition](/how-to/recursion-and-repetition) — `p`,
  `r`, and counters as depth guards.
- [Give good parse errors](/how-to/parse-errors) — `e`, the alternate that
  exists to fail well.

---

# Parse a line-oriented format

Source: https://tabnas.dev/how-to/line-oriented-formats
Make newlines significant — records, sections and one-statement-per-line syntax.

Most grammars want whitespace gone. INI files, CSV, log lines and
one-statement-per-line configuration all want the opposite: a newline is the
thing that ends a record, and throwing it away destroys the format.

The engine ignores space, newline and comment tokens by default because they
are in the `IGNORE` token set. Taking newline back out of that set is the whole
of the trick.

## Stop ignoring the newline

`IGNORE` is positional: `#SP`, `#LN`, `#CM`. Pass `null` to drop an entry and
`undefined` to leave it alone.

```ts
const tn = new Tabnas()
tn.options({
  //           #SP        #LN   #CM
  tokenSet: { IGNORE: [undefined, null, undefined] },
})
```

From that point `#LN` arrives as an ordinary token that your rules must handle
— including in places you did not think about, which is why line-oriented
grammars tend to have an explicit blank-line alternate.

## Know what a newline token is

**Runs of newlines lex as one token.** That is usually what you want, and
occasionally not:

```ts
'1\n\n2'   // => #NR"1"  #LN"\n\n"  #NR"2"
```

`line.single` makes each newline its own token; `line.chars` and
`line.rowChars` change which characters count as one:

```ts
tn.options({ line: { single: true } })
'1\n\n2'   // => #NR"1"  #LN"\n"  #LN"\n"  #NR"2"

tn.options({ line: { chars: ';', rowChars: ';' } })
'1;2'      // => #NR"1"  #LN";"  #NR"2"
```

The second is how a record separator that isn't a newline is handled: it is
still a *line* to the lexer.

## A worked example: INI

Sections, `key = value`, blank lines. About twenty lines of grammar:

```ts

const tn = new Tabnas()
tn.options({
  fixed: { token: { '#EQ': '=' } },
  tokenSet: { IGNORE: [undefined, null, undefined] },   // #LN is content
  rule: { start: 'doc' },
})

// The section a key belongs to. Parse-scoped, so instances stay reusable.
const store = (r, ctx) => { if (null != r.u.key) ctx.u.sect[r.u.key] = r.child.node }

tn.rule('doc', (rs) => rs
  .bo((r, ctx) => { r.node = {}; ctx.u.sect = r.node })
  .open([{ s: '#ZZ' }, { p: 'line' }])
  .close([{}]))

tn.rule('line', (rs) => rs
  .open([
    { s: '#ZZ' },                                        // end of input
    { s: '#LN', r: 'line' },                             // blank line
    { s: ['#OS', '#TX', '#CS'],                          // [section]
      a: (r, ctx) => { ctx.u.sect = r.parent.node[r.o[1].src] = {} } },
    { s: ['#TX', '#EQ'], p: 'val',                       // key = value
      a: (r) => { r.u.key = r.o[0].src } },
  ])
  .close([
    { s: '#LN', a: store, r: 'line' },                   // next line
    { a: store },                                        // last line
  ]))

tn.rule('val', (rs) => rs.open([
  { s: '#VAL', a: (r, ctx) => { r.node = r.o0.resolveVal(r, ctx) } },
]))

tn.parse('a = 1\nb = two\n\n[db]\nhost = localhost\nport = 5432\n')
// => { a: 1, b: 'two', db: { host: 'localhost', port: 5432 } }

tn.parse('a = 1')          // => { a: 1 }
tn.parse('\n\na = 1\n\n')  // => { a: 1 }
```

Four things in there are the general pattern, not INI trivia.

**The repeat lives in `line`, not `doc`.** `{ r: 'line' }` in `line`'s close
replaces `line` at its own depth, so every line's parent is still `doc`. Put
the same repeat in `doc`'s close and it replaces *`doc`*, `r.parent.node`
becomes `undefined` on the second line, and the failure looks like a data bug
rather than a grammar bug.

**`#ZZ` gets its own alternate.** A line-oriented grammar has to say what
end-of-input means, in both phases — otherwise a file with no trailing newline
parses differently from one with.

**Blank lines are an alternate, not an accident.** `{ s: '#LN', r: 'line' }`
consumes the run and goes round again.

**Current-section state lives on `ctx.u`, not on the instance.** `ctx` is
created per parse, so the instance stays reusable and concurrent parses cannot
collide. `r.u` is per-rule user data, which is why the key can sit there
between the open and close phases of the same line.

`#VAL` is the built-in token set covering number, string, text and value
literals, and `resolveVal` turns the matched token into a real value — so
`5432` is a number and `true` is a boolean without any work.

## The reference solution: CSV

`@tabnas/csv` is this idea taken all the way, and it is the thing to read (or
just use) before writing your own record parser. It removes `#LN` from `IGNORE`
always and `#SP` too in strict mode, because in CSV a leading space is part of
the field.

```ts

const tn = new Tabnas().use(jsonic).use(Csv)

tn.parse('name,age\nAlice,30\nBob,25')
// => [ { name: 'Alice', age: '30' }, { name: 'Bob', age: '25' } ]
```

Blank lines are skipped by default and preserved on request, and the record
separator does not have to be a newline:

```ts
new Tabnas().use(jsonic).use(Csv).parse('a,b\n1,2\n\n3,4')
// => [ { a: '1', b: '2' }, { a: '3', b: '4' } ]

new Tabnas().use(jsonic).use(Csv, { record: { empty: true } }).parse('a,b\n1,2\n\n3,4')
// => [ { a: '1', b: '2' }, { a: '', b: '' }, { a: '3', b: '4' } ]

new Tabnas().use(jsonic).use(Csv, { record: { separators: ';' } }).parse('a,b;1,2;3,4')
// => [ { a: '1', b: '2' }, { a: '3', b: '4' } ]
```

For very large inputs, `stream` hands each record to a callback instead of
building an array — see
[writing a parameterised parser](/how-to/parameterised-parsers).

## What this costs

Once `#LN` is significant, *every* rule in the grammar has to have an opinion
about it. That is the real expense of a line-oriented format, and it is why
mixing one with a free-form nested syntax is harder than either alone. If only
part of your format is line-oriented, consider parsing the line structure first
and the contents second.

## See also

- [Handle comments and whitespace](/how-to/comments-and-whitespace) — the other
  two members of `IGNORE`.
- [Choose between alternates](/how-to/choose-between-alternates) — `r` versus
  `p`, and the empty alternate.
- [@tabnas/csv](https://github.com/tabnas/csv) — the grammar, and every option.

---

# Lex a token the engine doesn't know

Source: https://tabnas.dev/how-to/custom-tokens
Fixed literals, regex tokens, value literals, and a hand-written matcher for the cases none of those reach.

You write the lexer. The engine ships a small set of built-in tokens — number,
string, text, the JSON punctuation, space, newline, comment, end-of-source —
and everything else in your language is a token you declare.

There are four ways to declare one, in increasing order of effort. Use the
first that works.

## 1 · A fixed literal

The common case: a symbol or keyword with exactly one spelling.

```ts
tn.options({ fixed: { token: { '#EQ': '=', '#AR': '->' } } })
```

The name is yours; the value is matched literally. Longer literals win over
shorter ones, so `->` and `-` can coexist.

**Pick a name that isn't taken.** `#CM` is *comment*, not comma — the comma is
`#CA` — and silently redefining a built-in token is an afternoon you won't get
back. `tn.fixed('=')` returns the tin for a literal, or `undefined` if nothing
claims it; `tn.token('#EQ')` does the same by name and creates the token if
needed.

Redefining one on purpose is a legitimate technique:
`fixed: { token: { '#CA': ';' } }` is how `@tabnas/csv` implements
`field.separation` without touching its grammar.

## 2 · A regex token

When the spelling is a pattern rather than a literal:

```ts
const tn = new Tabnas()
tn.options({
  match: { token: { '#DUR': /^\d+(ms|s|m|h)/ } },
  rule: { start: 'val' },
})

tn.rule('val', (rs) => rs.open([
  { s: '#DUR', a: (r) => { r.node = r.o[0].src } },
]))

tn.parse('250ms')   // => '250ms'
tn.parse('3h')      // => '3h'
```

**Anchor the pattern.** A regex without `^` will match further down the input
and the lexer will not have consumed the characters in between; the symptom is
"unexpected character" pointing at the start of a value that clearly matches.

A `match.token` entry gives you a *new* token, which means every rule that
should accept it needs an alternate for it. That is the right shape when the
token is syntax. When it is a **value**, the next option is much less work.

## 3 · A value literal

`match.value` produces a `#VL` token carrying a computed value — and `#VL` is
already in the `VAL` token set every grammar accepts, so it works everywhere a
value works without touching a single rule:

```ts

const tn = new Tabnas({ plugins: [json] })
tn.options({
  match: {
    value: {
      hex: { match: /^0x[0-9a-f]+/, val: (res) => parseInt(res[0], 16) },
    },
  },
})

tn.parse('{"n": 0xff}')   // => { n: 255 }
```

`val` receives the regex match array, so captures are available. For a fixed
set of words rather than a pattern, `value.def` is simpler still:

```ts
tn.options({ value: { def: { yes: { val: true }, no: { val: false }, nil: { val: null } } } })

tn.parse('{"a":yes,"b":no,"c":nil}')   // => { a: true, b: false, c: null }
```

Setting an entry to `null` removes it, which is how a dialect drops `true` or
`null` from the language.

## 4 · A matcher function

Some tokens are not regular: a raw block that runs to a terminator, a
heredoc, an indentation counter. Write a matcher. It is handed the lexer, and
its job is to return a token and advance the point — or return `undefined` and
leave the point alone.

```ts
const tn = new Tabnas({ plugins: [json] })

tn.options({
  lex: {
    match: {
      raw: {
        order: 2e6,
        make: () => function rawMatcher(lex) {
          const pnt = lex.pnt
          const src = lex.src.substring(pnt.sI)
          if (!src.startsWith('<<')) return undefined
          const end = src.indexOf('>>', 2)
          if (-1 === end) return undefined

          const tkn = lex.token('#RAW', src.substring(2, end), src.substring(0, end + 2), pnt)
          pnt.sI += end + 2
          pnt.cI += end + 2
          return tkn
        },
      },
    },
  },
})

tn.token('#RAW')
tn.rule('val', (rs) => rs.open([{ s: '#RAW', a: (r) => { r.node = r.o[0].val } }]))

tn.parse('{"a": <<x: 1, y: 2>>}')   // => { a: 'x: 1, y: 2' }
tn.parse('[<<a>>, 1, "b"]')         // => [ 'a', 1, 'b' ]
```

`order` decides where the matcher sits in the chain — lower runs earlier. It
matters whenever your syntax shares a prefix with a built-in one: a matcher for
`//path` has to run before the comment matcher, not after.

Two responsibilities are yours and the engine will not check them. **Advance
`pnt` by exactly the characters you consumed**, including `rI`/`cI` if the
token can span a newline, or every error position after it is wrong. And
**return `undefined` rather than throwing** when the input isn't yours, so the
rest of the chain gets a turn.

## Seeing the token stream

Before debugging a rule, check that the lexer produced what you think it did.
`sub` gets a callback on every token:

```ts
tn.sub({ lex: (tkn) => console.log(tkn.name, JSON.stringify(tkn.src), tkn.val) })

tn.parse('{"n": 0xff}')
// #OB "{"      undefined
// #ST "\"n\""  n
// #CL ":"      undefined
// #SP " "      undefined
// #VL "0xff"   255
// #CB "}"      undefined
// #ZZ ""       undefined
// #ZZ ""       undefined
```

Half of "my rule never fires" turns out to be "my token never lexed". (The end
token being reported twice is the parser peeking past the end, not a bug in
your grammar.)

## See also

- [The rule table](/docs/rule-table) — the built-in tokens, and their names.
- [Handle strings, quotes and escapes](/how-to/strings-and-quoting) — the
  string matcher's own options, which usually beat writing a matcher.
- [Write a parameterised parser](/how-to/parameterised-parsers) — making the
  token set an option.

---

# Handle comments and whitespace

Source: https://tabnas.dev/how-to/comments-and-whitespace
Turn comment styles on, define your own, and decide what the parser is allowed to throw away.

Space, newline and comment are lexed as real tokens — `#SP`, `#LN`, `#CM` —
and then discarded, because all three are in the `IGNORE` token set. Nearly
everything you want here is a change to that set, or to what counts as a
comment.

## Turning comments on

Comment lexing is a switch, and a grammar can have it off: `@tabnas/json` does,
because JSON has no comments.

```ts

const strict = new Tabnas({ plugins: [json] })
strict.parse('{"a":1} // hi')
// throws [tabnas/unexpected]: unexpected character(s): /
```

Flip it and the three built-in styles appear — `#`, `//` and `/* … */`. That
one line is the whole difference between JSON and JSONC:

```ts
const jsonc = new Tabnas({ plugins: [json] })
jsonc.options({ comment: { lex: true } })

jsonc.parse('{"a":1} // hi')      // => { a: 1 }
jsonc.parse('{"a":1 /* x */}')    // => { a: 1 }
jsonc.parse('{"a":1} # h')        // => { a: 1 }
```

## Choosing which styles

The definitions are a map keyed by name — `hash`, `slash`, `multi` — so
removing one is setting it to `null`:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ comment: { lex: true, def: { hash: null, multi: null } } })

tn.parse('{"a":1} // hi')   // => { a: 1 }
tn.parse('{"a":1} # hi')    // throws [tabnas/unexpected]
```

## Defining your own

A definition is a start marker, optionally an end marker, and a flag saying
whether it runs to end of line:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({
  comment: {
    lex: true,
    def: {
      semi: { line: true,  start: ';',    lex: true, eatline: false },
      sql:  { line: true,  start: '--',   lex: true, eatline: false },
      xml:  { line: false, start: '<!--', end: '-->', lex: true, eatline: false },
    },
  },
})

tn.parse('{"a":1} ; note')          // => { a: 1 }
tn.parse('{"a":1} -- note')         // => { a: 1 }
tn.parse('{"a": <!-- x --> 1}')     // => { a: 1 }
```

**A new definition must set `lex: true` on itself.** The built-in definitions
carry it, and the outer `comment.lex` switch does not supply it for entries you
add — leave it out and the definition is registered, ignored, and your comment
marker comes back as "unexpected character". This is the single most common way
to get this wrong.

## `eatline`

A line comment normally stops *before* the newline, so a `#LN` token follows
it. With `eatline: true` the comment token swallows the newline as well:

```ts
// eatline: false
'1 // hi\n2'   // => #NR"1"  #SP" "  #CM"// hi"     #LN"\n"  #NR"2"

// eatline: true
'1 // hi\n2'   // => #NR"1"  #SP" "  #CM"// hi\n"   #NR"2"
```

Irrelevant while newlines are ignored, and decisive once they are not: in a
[line-oriented grammar](/how-to/line-oriented-formats), a comment on its own
line otherwise emits a record separator that isn't there.

## Keeping what is normally thrown away

`IGNORE` is positional — `#SP`, `#LN`, `#CM` — and `null` drops an entry while
`undefined` leaves it:

```ts
// Newlines become significant; space and comments still ignored.
tn.options({ tokenSet: { IGNORE: [undefined, null, undefined] } })

// Comments become significant; a rule must now handle #CM.
tn.options({ tokenSet: { IGNORE: [undefined, undefined, null] } })
```

Keeping `#CM` is how a formatter or a doc-comment extractor gets at comment
text: the token is in the stream with its source, and a rule can attach it to
whatever it precedes. Be aware of the cost — *every* rule that a comment can
appear before now needs an alternate for it, which is most of them.

Making space significant is rarer, and drastic:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ tokenSet: { IGNORE: [null, undefined, undefined] } })

tn.parse('{"a":1}')    // => { a: 1 }
tn.parse('{"a": 1}')   // throws — the space is now a token nothing accepts
```

`@tabnas/csv` does exactly this in strict mode, because a leading space in a
CSV field is part of the value. It is the right call there and almost nowhere
else; `space.chars` (which characters count as space) is the gentler knob.

## Checking what the lexer did

When a comment marker "doesn't work", look at the tokens before looking at the
rules:

```ts
tn.sub({ lex: (tkn) => console.log(tkn.name, JSON.stringify(tkn.src)) })
```

A `#CM` token in the stream means the definition took and the problem is
elsewhere. No `#CM` means the definition never registered — check `lex: true`.

## See also

- [Parse a line-oriented format](/how-to/line-oriented-formats) — the other
  reason to change `IGNORE`.
- [Lex a token the engine doesn't know](/how-to/custom-tokens) — matcher order,
  and why a comment-like token has to run early.
- [The rule table](/docs/rule-table) — the built-in token names.

---

# Handle strings, quotes and escapes

Source: https://tabnas.dev/how-to/strings-and-quoting
Change what quotes a string, which escapes exist, and what happens to the ones that don't.

The string matcher is configuration, not code. Which characters open a string,
which of them may span lines, what the escape character is, which escapes are
defined, and what an undefined escape means — all of it is `options.string`,
and the difference between a strict format and a relaxed one is a handful of
those fields.

## Start from what you have

The two published grammars sit at opposite ends, which makes them a useful
reference:

| | `@tabnas/json` | `@tabnas/jsonic` |
|---|---|---|
| Quote characters | `"` | `'` and `"` and backtick |
| Multi-line | none | backtick |
| Escapes | `b f n r t " \ /` | those, plus `v`, `'` and backtick |
| Unknown escape | error | passed through |

```ts
new Tabnas({ plugins: [json] }).parse(String.raw`"a\nb"`)   // => 'a\nb'
new Tabnas({ plugins: [json] }).parse(String.raw`"a\qb"`)   // throws — \q is undefined

new Tabnas().use(jsonic).parse("'x'")       // => 'x'
new Tabnas().use(jsonic).parse('`a b`')     // => 'a b'
```

Extending whichever is closer is nearly always less work than configuring the
matcher from scratch.

## More quote characters

`string.chars` is the complete set — set it, don't add to it:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ string: { chars: `"'` } })

tn.parse("'x'")   // => 'x'
```

## Strings that span lines

A raw newline inside an ordinary string is an error, in both grammars:

```ts
new Tabnas().use(jsonic).parse('a: "line1\nline2"')
// throws [jsonic/unprintable]: unprintable character
```

`string.multiChars` lists the quote characters that are allowed to. It must
also be in `chars`:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ string: { chars: '"`', multiChars: '`' } })

tn.parse('`line1\nline2`')   // => 'line1\nline2'
```

jsonic already does this for the backtick:

```ts
new Tabnas().use(jsonic).parse('a: `line1\nline2`')
// => { a: 'line1\nline2' }
```

For block-delimited strings with markers rather than quotes — triple quotes,
heredocs — see
[@tabnas/hoover](https://github.com/tabnas/hoover), which adds a configurable
"hoovering" matcher, or write
[a matcher of your own](/how-to/custom-tokens#4--a-matcher-function).

## Escapes

`string.escape` maps the character *after* the escape character to what it
produces. Adding an entry defines an escape; setting it to `null` removes one:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ string: { escape: { z: ' ZZ' } } })
tn.parse(String.raw`"a\zb"`)   // => 'a ZZb'

const strict = new Tabnas({ plugins: [json] })
strict.options({ string: { escape: { n: null } } })
strict.parse(String.raw`"a\nb"`)   // throws — \n is no longer defined
```

The escape character itself is `string.escapeChar`:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ string: { escapeChar: '~' } })
tn.parse('"a~nb"')   // => 'a\nb'
```

`allowUnknown` decides what an undefined escape does. `false` — the JSON
setting — is an error; `true` — the jsonic setting — drops the escape character
and keeps the character after it:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.options({ string: { allowUnknown: true } })
tn.parse(String.raw`"a\qb"`)   // => 'aqb'
```

Prefer `false` for a format you control. An unknown escape is almost always a
typo, and silently eating the backslash turns it into a data bug much later.

## Doubling instead of escaping

CSV does not use a backslash: a quote inside a quoted field is written twice,
and a quoted field may contain the separator and even a newline.
`@tabnas/csv` installs its own string matcher for this, which is the
general answer when quoting is not backslash-shaped:

```ts
const tn = new Tabnas().use(jsonic).use(Csv)

tn.parse('name,note\n"Smith, J","said ""hi"""')
// => [ { name: 'Smith, J', note: 'said "hi"' } ]

tn.parse('a,b\n"x\ny",2')
// => [ { a: 'x\ny', b: '2' } ]
```

The `string.csv` option turns that matcher on and off independently of strict
mode, which is the hook to copy if your format has the same convention.

## Unterminated strings

The engine reports these as their own error code, which means you can give them
your own message:

```ts
new Tabnas({ plugins: [json] }).parse('"abc')
// throws [tabnas/unterminated_string]: unterminated string: "abc
```

```ts
tn.options({
  error: { unterminated_string: 'string is missing its closing quote' },
})
```

See [giving good parse errors](/how-to/parse-errors).

## See also

- [Lex a token the engine doesn't know](/how-to/custom-tokens) — when the
  string matcher's options run out.
- [Handle comments and whitespace](/how-to/comments-and-whitespace) — the other
  half of the lexer's configuration.
- [Extending a grammar](/docs/extending) — deriving a stricter or looser
  dialect without touching the original.

---

# Debug a grammar

Source: https://tabnas.dev/how-to/debug-a-grammar
See the rules you actually have, watch a parse step by step, and draw the result.

A grammar that doesn't work is rarely a mystery for long, because a grammar is
data: you can print it. The order below is the order to try things in — each
step is cheaper than the one after it, and most bugs are caught in the first
two.

## 1 · Print the rules you have

`rule()` with no arguments returns the rule map. It is the fastest way to check
that a plugin did what you thought:

```ts

Object.keys(new Tabnas({ plugins: [json] }).rule())
// => [ 'val', 'map', 'list', 'pair', 'elem' ]
```

One level down, `def.open` and `def.close` are the alternates in the order they
will be tried — which is the order that decides everything:

```ts
const tn = new Tabnas({ plugins: [json] })
tn.rule('val').def.open.map((a) => ({ s: a.s, p: a.p, b: a.b }))
// => [ { s: [ '#OB' ], p: 'map',  b: 1 },
//      { s: [ '#OS' ], p: 'list', b: 1 },
//      { s: [ '#VAL' ], p: null,  b: null } ]
```

Print this before and after your plugin runs. "My alternate never fires" is
usually "my alternate is third and the second one matches too".

## 2 · Print the tokens

Half of the remaining bugs are in the lexer, not the rules. `sub` gets a
callback per token:

```ts
tn.sub({ lex: (tkn) => console.log(tkn.name, JSON.stringify(tkn.src), tkn.val) })
```

and per rule step, which is a one-line parse trace when you don't want the
full one:

```ts
const steps = []
tn.sub({ rule: (r) => steps.push(`${r.name}~${r.state}@${r.d}`) })
tn.parse('1+2')

steps.join(' ')
// => '__start__~o@0 val~o@1 add~o@2 add~c@2 add~o@2 add~c@2 val~c@1 __start__~c@0'
```

`o`/`c` is the phase and `@n` the stack depth, so a rule that should be
repeating at one depth and is instead nesting shows up immediately.

## 3 · Describe the whole instance

`@tabnas/debug` adds a `debug` property with three views. It is a development
dependency — never ship it in a runtime path.

```ts

const tn = new Tabnas({ plugins: [json], tag: 'demo' })
tn.use(Debug, { print: false })

tn.debug.describe()   // printable text: tokens, token sets, rules, lexer, config
tn.debug.model()      // the same thing as a JSON-serialisable object
tn.debug.abnf()       // the live grammar rendered back as ABNF
```

`print: false` matters. The default is `true`, which prints a full description
every time `use()` is called afterwards — useful when you are bisecting which
plugin broke a grammar, and overwhelming otherwise.

`model()` is the one to reach for in a test. Its fields:

| Field | What it holds |
|---|---|
| `tag` | the instance tag |
| `tokens` | `{ tin, name, fixed? }[]` — the token table |
| `tokenSets` | named sets (`IGNORE`, `VAL`, `KEY`) → member tins |
| `rules` | each rule's open/close alternates, structurally |
| `graph` | per-rule push/replace edges |
| `lexer` | the matcher chain, in order |
| `config` | start rule, finish flag, per-lexer enable flags |
| `plugins` | what was applied, with the options it settled on |
| `abnf` | the live grammar as ABNF text |

```ts
const m = tn.debug.model()
m.config.start                 // => 'val'
m.plugins.map((p) => p.name)   // => [ 'json', 'Debug' ]
```

Worth knowing: with `@tabnas/abnf` installed, `m.config.start` is `__start__`,
not your first production. The compiler wraps the grammar in a start rule that
consumes `#ZZ`, which is what gives ABNF grammars an end-of-source check that
hand-written ones don't have.

`abnf()` is the useful trick for a grammar built by plugins: it renders whatever
is actually installed as ABNF, so you can read a composed grammar as one
document rather than as a stack of `use()` calls. It reads only the running
engine, and it is best-effort — a token-set alternate or an arbitrary match
regex has no ABNF spelling, and comes out as a comment or an empty alternative.
For a grammar that *came* from ABNF it round-trips exactly, which makes it a
good equality check in a test.

## 4 · Trace the parse

When the rules and tokens both look right, watch it run:

```ts
tn.use(Debug, { print: false, trace: true })
tn.parse('1+2')
```

The trace prints six kinds of line — `step`, `rule`, `lex`, `parse`, `node`,
`stack` — and you can switch off the ones you don't need. Note that the option
is *merged* over defaults where everything is on, so narrowing means setting
entries to `false`, not listing the ones you want:

```ts
tn.use(Debug, { print: false, trace: { lex: false, node: false, stack: false, step: false } })
```

```
rule   "1+2"   []~[]        0  __start__~1:OPEN  prev=0 parent=0 child=0
parse  "1+2"   []~[]        0  alt=0  []   g:abnf   p:val
rule   "1+2"   []~[]        1  . val~2:OPEN      prev=0 parent=1 child=0
parse  "1+2"   []~[]        1  . alt=0  []   g:abnf   p:add
rule   "1+2"   []~[]        2  . . add~3:OPEN    prev=0 parent=2 child=0
parse  "2"     ["1"]~[#NR]  2  . . alt=0  []   g:abnf
rule   "2"     []~[]        2  . . add~3:CLOSE   prev=0 parent=2 child=0
parse  "2"     ["+"]~[#PL]  2  . . alt=0  []   g:abnf   r:add
```

Read the `parse` lines. `alt=` is which alternate matched, `g:` its group tags,
and `p:`/`r:` whether it pushed or repeated. The dots are stack depth. The two
`add` rules at depth 2 with `r:add` between them are a repeat, not a nest —
exactly the distinction that is hard to see any other way.

To capture a trace instead of printing it — in a test, say — supply a console:

```ts
const lines = []
const tn = new Tabnas({
  plugins: [json],
  debug: { get_console: () => ({ log: (...a) => lines.push(a.join(' ')) }) },
})
```

## 5 · Draw it

`@tabnas/railroad` introspects a live instance and renders it. This is the view
that makes a *shape* problem obvious — an optional that should have been a
repetition, an alternative that can never be reached.

```ts

const tn = new Tabnas({ plugins: [json, railroad] })

tn.railroad.toJson()    // declarative model: { start, rules, meta }
tn.railroad.toSvg()     // vertical-flow SVG
tn.railroad.toAscii()   // vertical ASCII, for a terminal or a diff
```

```
val:
              │
   ┌──────────┼──────────┐
┌──┴──┐   ┌───┴──┐   ╭───┴───╮
│ map │   │ list │   │ "VAL" │
└──┬──┘   └───┬──┘   ╰───┬───╯
   └──────────┼──────────┘
              │
```

`toAscii({ ascii: true })` uses plain `| - +` glyphs, which survives copy-paste
into an issue. There is also a CLI, so a diagram can be a build artifact:

```bash
tabnas-railroad --grammar @tabnas/json -o diagrams
# wrote grammar.railroad.json, grammar.svg, grammar.txt to diagrams/
```

The `--text` form is a compact per-rule EBNF, and is the quickest whole-grammar
overview there is:

```
val = (map | list | "VAL")
map = "{" [pair] "}"
list = "[" [elem] "]"
pair = "KEY" ":" val+ /* "," */
elem = val+ /* "," */
```

Railroad introspects `@tabnas/parser` instances. Grammars still targeting the
older `@tabnas/jsonic` engine — `ini` and `yaml` — are not yet supported.

## Reading the error you already have

Before any of the above: a parse error already tells you which rule and phase
gave up, and on which token. It is on the last line of the message.

```
[tabnas/unexpected]: unexpected character(s): }
  --> <no-file>:1:7
  1 | {"a": }
            ^ unexpected character(s): }

  --internal: tag=-; rule=val~o; token=#CB; plugins=json--
```

`rule=val~o` is the open phase of `val`; `token=#CB` is what it was offered.
That is two of the four things you were about to go and find out.

## See also

- [Give good parse errors](/how-to/parse-errors) — making that message useful
  to someone who isn't you.
- [Test a grammar](/how-to/test-a-grammar) — turning today's bug into a test.
- [Choose between alternates](/how-to/choose-between-alternates) — what `alt=`
  in the trace is indexing.

---

# Give good parse errors

Source: https://tabnas.dev/how-to/parse-errors
Name the file, define your own error codes, and raise them from the alternate that knows what went wrong.

A parser is a user interface. Most of the time it is being used by someone who
got the syntax slightly wrong, and the error message is the entire product.

The engine gives you a good default and four ways to improve on it.

## What you get for free

Every parse error is a `TabnasError` — a `SyntaxError` subclass — rendered with
the source, a caret, and an explanation:

```
[tabnas/unexpected]: unexpected character(s): }
  --> <no-file>:1:7
  1 | {"a": }
            ^ unexpected character(s): }

  The character(s) } do not match any rule alternative active at
  this position.

  --internal: tag=-; rule=val~o; token=#CB; plugins=json--
```

The last line is for you, not your users: the rule and phase that gave up, the
token it was offered, and the plugins in play.

## 1 · Tell it the file name

`<no-file>` is not helpful and it is one argument away. Anything in the parse
metadata is available to the error formatter, and `fileName` is used directly:

```ts
tn.parse(src, { fileName: 'settings.json' })
```

```
  --> settings.json:1:7
```

## 2 · Name your language

`errmsg.name` replaces the `tabnas/` prefix, so the error looks like it came
from your tool rather than from a dependency. `errmsg.link` adds a
documentation URL:

```ts
tn.options({ errmsg: { name: 'cfg', link: 'https://example.com/errors/' } })
```

```
[cfg/unexpected]: unexpected character(s): }
  …
  https://example.com/errors/
```

## 3 · Define your own errors

An error is a code with a short message and a longer hint. Both are templates,
and **placeholders are `{braces}`** — values come from the `details` object you
pass when raising it, and from the token, rule and context:

```ts
tn.options({
  error: { missing_equals: 'expected = after key "{key}"' },
  hint:  { missing_equals: 'Settings are written `name = value`.\nThe key {key} had no `=` after it.' },
})
```

Overriding a built-in code works the same way, and is the cheapest improvement
available on a format with unusual quoting:

```ts
tn.options({ error: { unterminated_string: 'string is missing its closing quote' } })

tn.parse('"abc')
// throws [tabnas/unterminated_string]: string is missing its closing quote
```

## 4 · Raise it from where you know

`e` is an alternate field: an alternate that exists in order to fail well.
It runs when that alternate is selected, and returns the token to blame —
`token.bad(code, details)` builds it.

The pattern is an alternate *after* the good one, matching the prefix they
share:

```ts
const tn = new Tabnas()
tn.options({
  fixed: { token: { '#EQ': '=' } },
  rule: { start: 'setting' },
  error: { missing_equals: 'expected = after key "{key}"' },
  hint:  { missing_equals: 'Settings are written `name = value`.\nThe key {key} had no `=` after it.' },
})

tn.rule('setting', (rs) => rs
  .open([
    { s: ['#TX', '#EQ'], p: 'val', a: (r) => { r.node = { key: r.o[0].src } } },
    { s: ['#TX'], e: (r) => r.o0.bad('missing_equals', { key: r.o0.src }) },
  ])
  .close([{ a: (r) => { r.node.value = r.child.node } }]))

tn.rule('val', (rs) => rs.open([{ s: '#NR', a: (r) => { r.node = r.o[0].val } }]))

tn.parse('port = 8080')   // => { key: 'port', value: 8080 }
tn.parse('port 8080')
```

```
[tabnas/missing_equals]: expected = after key "port"
  --> <no-file>:1:1
  1 | port 8080
      ^^^^ expected = after key "port"

  Settings are written `name = value`.
  The key port had no `=` after it.
```

Without that second alternate the message would have been "unexpected
character(s): 8080", pointing at the number rather than the missing `=`.

An action can raise one too, by *returning* the token rather than throwing —
which is how `@tabnas/multisource` reports a missing include:

```ts
action: (rule, ctx) => {
  const src = FILES[String(rule.child.node)]
  if (null == src) return rule.parent.o0.bad('include_not_found', { path })
  rule.node = ctx.inst().parse(src)
}
```

## Point at the right token

The token you call `bad()` on decides where the caret goes, and the obvious
choice is often wrong. For an unclosed bracket, the useful position is the
*opening* one — which is `r.o0`, the token the rule opened on, not the token
that surprised it:

```ts
const OP = tn.fixed('(')
const grouped = (r) => OP === r.o0?.tin

tn.rule('val', (rs) => rs
  .open([{ s: '#OP', p: 'val' }, { s: '#NR', a: (r) => { r.node = r.o[0].val } }])
  .close([
    { s: '#CP', c: grouped, a: (r) => { r.node = [r.child.node] } },
    { c: grouped, e: (r) => r.o0.bad('unclosed') },
    {},
  ]))

tn.parse('((1)')
```

```
[tabnas/unclosed]: unclosed group
  --> <no-file>:1:1
  1 | ((1)
      ^ unclosed group

  A ( opened here was never closed with a matching ).
```

Column 1 — the bracket that was never closed — rather than the end of input.

## Handling errors in code

```ts
try {
  tn.parse(src, { fileName: 'settings.cfg' })
} catch (e) {
  e instanceof SyntaxError   // => true
  e.code                     // => 'missing_equals'
  e.fileName                 // => 'settings.cfg'
  e.lineNumber               // => 1
  e.columnNumber             // => 1
  e.message                  // the rendered block above
}
```

`TabnasError` is exported if you want an exact `instanceof`, but the `code`
field is the thing to branch on — it is stable, and it is what your `error` and
`hint` tables are keyed by.

## What this can't do

**The parse stops at the first error.** There is no recovery pass and no way to
collect several errors from one input: the engine is deterministic and does not
backtrack, so once a token cannot be matched there is no defined state to
continue from. If you need a list of problems rather than the first one — an
editor integration, say — the shape that works is parsing smaller units
separately (a line, a record, a section) and collecting their failures.

That is a real limitation, and it is the price of the parse being linear and
having exactly one interpretation.

## See also

- [Debug a grammar](/how-to/debug-a-grammar) — reading the `--internal` line,
  and what to do next.
- [Choose between alternates](/how-to/choose-between-alternates) — where the
  failing alternate goes in the list.
- [The rule table](/docs/rule-table) — the `e` field, and the rest.

---

# Test a grammar

Source: https://tabnas.dev/how-to/test-a-grammar
Assert what parses, what doesn't, and that the grammar is still the shape you think it is.

A grammar is data, and it is mutable data — every `use()` changes the instance
it is called on. That makes two kinds of test worth writing: the obvious one
about inputs and outputs, and a less obvious one about the *grammar itself*,
which catches the class of bug where a plugin quietly stopped applying.

## A fresh instance per test

Start here, because it is the mistake that produces the most confusing
failures. If a test calls `use()`, `options()` or `rule()` on a shared
instance, it has changed the grammar for everything after it — and test order
is not something you want to depend on.

```ts
const make = () => {
  const tn = new Tabnas({ plugins: [abnf] })
  tn.abnf(GRAMMAR, { actions: { /* … */ } })
  return tn
}
```

Parsing is safe to share: state lives on the parse context, not the instance,
so one instance can serve many `parse()` calls. It is *modification* that has
to be per-test. `make()` is also available on an instance, and derives a copy
without touching the original — see
[extending a grammar](/docs/extending).

## Accept and reject

Table-driven, because a grammar is a lot of small cases and each one wants its
own name in the output:

```ts

test('accepts', () => {
  const tn = make()
  for (const [src, want] of [['1', 1], ['1+2', 3], ['12+3+45', 60]]) {
    assert.equal(tn.parse(src).value, want, src)
  }
})

test('rejects', () => {
  const tn = make()
  for (const src of ['1+', '+1', 'a']) {
    assert.throws(() => tn.parse(src), (e) => 'unexpected' === e.code, src)
  }
})
```

**Assert on `e.code`, not on `e.message`.** The rendered message includes the
source line, a caret and a hint, all of which are meant to change as you
improve them. The code is the contract. This matters most for
[errors you defined yourself](/how-to/parse-errors) — a test on the code is
what stops a message rewrite from being a breaking change.

The rejection cases are the ones people skip and shouldn't. A grammar that is
too permissive still passes every accept test.

## Test the grammar, not just the parse

`@tabnas/debug` gives you the installed grammar as data, so you can assert its
shape. This catches the failure that input-output tests miss: a plugin that
silently didn't apply, and a grammar that still parses your happy path for a
different reason.

```ts

test('grammar shape is what we think', () => {
  const tn = make()
  tn.use(Debug, { print: false })

  const model = tn.debug.model()
  assert.deepEqual(model.rules.map((r) => r.name).sort(), ['__start__', 'add', 'val'])
  assert.equal(model.abnf, GRAMMAR)
})
```

That last assertion is the strongest cheap test there is for an ABNF grammar:
the engine's rendering of what it actually installed, compared against the
source you wrote. If a compiler change or a plugin alters the rule table, it
fails.

`model()` is JSON-serialisable, so a snapshot works too — `tokens`,
`tokenSets`, `rules`, `graph`, `lexer`, `config` and `plugins` in one object.
So does `@tabnas/railroad`'s `toJson()`, which is a smaller and more readable
snapshot if structure is what you care about.

## Test the samples in your README

The `tabnas-abnf` CLI parses samples against a grammar file and exits non-zero
if any of them fail, which makes a grammar's examples testable from a
`Makefile` or a CI step with no test harness at all:

```bash
tabnas-abnf -f grammar.abnf --parse '1+2' --parse '12+3+45'
# ok: "1+2" -> { "rule": "val", … }
# ok: "12+3+45" -> { "rule": "val", … }

tabnas-abnf -f grammar.abnf --parse '1+'
# exits 1
```

`--parse-file` takes the sample from a file, so the fixtures on disk and the
ones in CI are the same bytes.

## Pin the mark names

If you attach actions by `@ref`, the mark names are assigned by the compiler
and are not yours. A test that asserts the listing turns a silent regression —
actions that stop firing because a mark was renamed — into a failure:

```bash
tabnas-abnf --marks -f grammar.abnf
```

```
val  o:add  p:add
val  c:_  (empty)
add  o:NR  s:#NR
add  c:PL  s:#PL
add  c:_  (empty)
```

`markListing(spec)` gives the same listing from code. Attaching an action to a
mark that doesn't exist is already an error rather than a silent no-op, so this
is belt and braces — but the listing also documents the compiled shape, which
is worth having in the repository.

## Pin the versions

Everything in the org is pre-1.0, where a caret range refuses the next minor
and a minor can change behaviour. Pin exact versions in `package.json`
(`"0.5.0"`, not `"^0.5.0"`) and upgrade deliberately, with these tests as the
thing that tells you what moved.

## See also

- [Debug a grammar](/how-to/debug-a-grammar) — what `model()` contains, and how
  to read a trace when a test goes red.
- [Give good parse errors](/how-to/parse-errors) — why `e.code` is the stable
  surface.
- [Attaching actions](/docs/actions) — `@ref` marks and where they come from.