# Backlog

Ideas and known limits, newest thinking first. This file is development-only —
`.gitattributes` `export-ignore`s `/docs`, so it never reaches an installed package.

## A Sublime plugin backed by a real engine

The grammar is a presentation scan, and deliberately so: it never claims a construct the
engine does not see. Everything below needs the **whole document**, which a syntax
definition cannot see (it is handed one line at a time), and so belongs in a Python plugin
rather than in `Spintax.sublime-syntax`.

The enabler is [`spintax-core`](https://pypi.org/project/spintax-core/) — the family's
Python engine (`investblog/spintax-py`), MIT, zero runtime dependencies, held to the same
shared golden corpus as the TypeScript, PHP and Pascal engines. Sublime plugins are Python,
so the engine can run in-process rather than being reimplemented a fourth time.

What it would buy, roughly in order of value:

1. **Diagnostics.** `validate()` already returns findings carrying `offset` and `length`,
   so squiggles via `view.add_regions` and the error code in the status bar are close to
   free. No editor in the family has this except Spintax Studio.
2. **The two comment limits the grammar cannot fix.** The engine opens a comment only when
   a `#/` occurs later in the document. A plugin sees the whole buffer, so both residual
   errors documented in `Spintax.sublime-syntax` disappear: a `/#` at line start with no
   closer anywhere would stop greying the rest of the file, and a `/#` glued to text whose
   closer lands on a later line would be recognised. Spintax Studio solves it exactly this
   way — the window pushes the answer in (`SpxSynHighlighter.SetCloserLine`).
3. **Bracket pair + the construct's own separators** under the caret, skipping comments and
   permutation configs the way `SpxMatchBracket` / `SpxConstructOf` do. Sublime's stock
   matcher and BracketHighlighter both count parentheses and quotes, which are ordinary
   text in spintax.
4. **Commands**: render a preview, count the variants, generate N samples.
5. **The `#include` target on a following line**, which the grammar leaves plain because a
   forward-only line scan cannot know whether a quoted target ever arrives.

Open questions to settle before starting:

- **Python version.** The ST4 plugin host is Python 3.8; `spintax-core` declares 3.10+. A
  quick read suggests nothing blocks it — every module carries
  `from __future__ import annotations` and there are no `match` statements — but this has to
  be *measured*, not assumed. If it does not fit, either request a 3.8 floor upstream or
  ship the plugin as an ST4-only package.
- **Shipping the dependency**: Package Control `dependencies.json` vs. vendoring the
  package. Vendoring keeps installs self-contained; a dependency keeps updates cheap.
- **Packaging**: a second package (`Spintax Tools`?) that depends on this one, or fold the
  plugin into this package. Folding it in makes every user pay for the engine; a separate
  package keeps the syntax install as light as it is now.

## Smaller items

- **Differential test oracle.** `spintax-py` could gate the grammar in CI: feed a corpus of
  templates to both the engine and Sublime's tokenizer and compare what each calls a
  separator / config / plural head. Today the parity claims rest on hand-written assertions
  in `tests/syntax_test_spintax.spintax` plus a read of the Pascal tokenizer.
- **Deeper nesting inside a trailing separator.** The separator regex matches one level of
  balanced `{}` / `[]` (`[a<{, | and }>|b]`); the engine counts to any depth. Deeper nesting
  is left as text — under-claiming, not mis-painting. A regex cannot count; a plugin can.
- **Long single lines.** The comment opener's lookahead is lazy but still unbounded, and
  engine output has no obligation to contain a line break. The reference tokenizer measured
  this exact shape (`SpxTokens.pas`: 720 ms / 6.6 s on 40–80 KB single lines) and fixed it by
  precomputing the closer offsets — which, again, needs the whole document.