# Culebra Standard Library
This document is the **API reference** for Culebra's built-in
library: the namespace objects (`Math`, `IO`, `Sys`, `FS`, `Time`,
`Args`, `Random`, `String`) that group the runtime utilities.
Everything described here is available without any `import`
statement.
For an introductory tour and usage idioms see
[`handbook.md` §14](handbook.md#14-standard-library).
Language-level built-ins — `to_long`, `to_float`, `to_string`,
`type_of`, `range`, `iota`, `repeat` — are specified in
[§19 of the language spec](language.md). The matcher family
(`assert_true` / `assert_eq` / `assert_throws` / etc.) is documented
in [§13 below](#13-matchers). Methods on built-in types (`String`,
`Array`, `Object`) are specified in
[§18 of the language spec](language.md).
`inspect`, `print` and `println` are additionally bound as globals
aliased to `IO.inspect` / `IO.print` / `IO.println` (see
[§22 of the language spec](language.md)). They are unconditional, so an
embedded program sees them exactly as a script does.
Conventions used below:
* Types follow the annotations described in [§14 of the language
spec](language.md). `Any` denotes any value.
* Throws clauses describe runtime errors of the form
`type error at L:C.` etc. See [§15 of the language spec](language.md).
* Namespaces are **closed**: reading a member a namespace doesn't have
raises `AttributeError: namespace 'IO' has no member 'read_all'` at the
access site, rather than yielding `nil`. This turns a typo or a use of a
removed API into an immediate, catchable error instead of a confusing
failure at a later call. (Plain dicts keep the permissive rule — a missing
key reads as `nil`.) The dict builtins `keys`/`values`/`has`/`get`/`size`
still work on a namespace. `Path` is the one exception: it is a class, not
a namespace, so a missing property reads as `nil` the way it does on any
other class.
## Index
1. [`Math`](#1-math) — numeric utilities, constants, integer sequences
2. [`IO`](#2-io) — output, stdin (file I/O lives under `FS`)
3. [`FS`](#3-fs) — path manipulation, file/dir queries and mutations
4. [`File`](#4-file) — stateful handle for streaming read/write/seek
5. [`Time`](#5-time) — `Instant` / `Duration` classes, ISO 8601, calendar arithmetic, nanosecond precision
6. [`Random`](#6-random) — seedable PRNG (uniform, gauss, shuffle, choice, weighted_choice)
7. [`Sys`](#7-sys) — argv, exit, env, executable; `GC` heap introspection lives here too
8. [`Tensor`](#8-tensor) — N-dimensional numeric tensor with a BLAS-backed lazy graph
9. [`JSON`](#9-json) — stringify / parse round-trip
10. [`Args`](#10-args) — declarative CLI argument parser (positional / option / subcommand / `--help`)
11. [`Proc`](#11-proc) — run external commands synchronously, capture stdout/stderr/exit
12. [`Isolate`](#12-isolate) — run a closure on another thread (own heap), copy values across the boundary; `Channel`, `Parallel`, `Signal` (route Ctrl+C to a channel), `SharedBuffer` (zero-copy shared fixed-layout data), and `Shared` (immutable values shared by reference) live here too
13. [Matchers](#13-matchers) — `assert_true` / `assert_eq` / `assert_throws` / `assert_close` family
14. [`Regex`](#14-regex) — linear-time, grapheme-aware regular expressions
15. [`Http`](#15-http) — synchronous HTTP/HTTPS client (get/post/put/delete/head/request), server (routes, static files, WebSocket), and Server-Sent Events
16. [`Encoding`](#16-encoding) — text codecs by scheme (`Encoding.html`, `Encoding.base64`, `Encoding.hex`, `Encoding.url`, `Encoding.utf16`)
17. [`Compress`](#17-compress) — gzip / deflate (de)compression for data and files, and packing files into a ZIP archive
18. [`Hash`](#18-hash) — SHA-256/SHA-1/SHA-512/MD5 digests and HMAC (hex output)
19. [`CSV`](#19-csv) — parse / stringify RFC 4180-ish comma-separated values
20. [`Env`](#20-env) — parse / load dotenv-style `.env` files
21. [`UUID`](#21-uuid) — generate v4 (random) and v7 (time-ordered) UUIDs
22. [`Term`](#22-term) — terminal color, cursor control, size, and key/mouse input for TUIs
23. [`Log`](#23-log) — leveled, structured logging to stderr (text / JSON, child loggers)
24. [`TOML`](#24-toml) — parse / stringify TOML configuration
25. [`XML`](#25-xml): parse / search (an XPath subset) / stringify XML; a lazy event stream for large documents
26. [`SQLite`](#26-sqlite) — embedded SQL database (query / execute / prepared statements / transactions)
27. [`Canvas`](#27-canvas) — immediate-mode 2D framebuffer for games (shapes, sprites, offscreen targets, text, keys/mouse/gamepad, window controls)
28. [`Scene`](#28-scene) — retained-mode 3D renderer for procedural geometry
29. [`Net`](#29-net) — raw TCP / UDP sockets and name resolution (the layer under `Http`)
30. [`Desktop` / `Webview`](#30-desktop--webview) — native WebView desktop app: local HTTP server + window, one call
31. [`Vector2`](#31-vector2) — minimal 2D float vector for graphics/game code (also stands in for a "Point")
32. [`Vector3`](#32-vector3) — the 3D counterpart of `Vector2`
33. [`Deque`](#33-deque) — double-ended queue, O(1) amortized push/pop at either end
34. [`PriorityQueue`](#34-priorityqueue) — binary min-heap, O(log n) push/pop
35. [`PEG`](#35-peg) — PEG parser generator: write a grammar, get a syntax tree
36. [`CodeGen`](#36-codegen) — build a small language's IR by hand and run it
37. [`StateMachine`](#37-statemachine) — hierarchical state machine, with a text DSL
38. [`FST`](#38-fst) — compiled read-only dictionary: prefix, predictive and fuzzy search
39. [`Search`](#39-search) — full-text index over your own documents, ranked
40. [`Audio`](#40-audio) — sound for any program, with or without a window: WASM-4 tones, samples, streamed music, synthesised PCM, the microphone
41. [`Dir`](#41-dir) — a read-only set of files on disk, in memory, baked into the binary or in a ZIP archive, read one way (`Dir.disk` / `Dir.memory` / `Dir.embedded` / `Dir.zip`, and the `Dir` trait)
42. [Design notes](#42-design-notes)
43. [Not included (yet)](#43-not-included-yet)
**Where to find what**
| Need… | Look at |
|---|---|
| Constants (π, e, inf, nan) | [§1 Math constants](#math-pi) |
| Scalar arithmetic (abs, min, max, log, exp, sqrt, floor, ceil, round) | [§1 Math](#1-math) |
| Trigonometry (sin, cos, tan, asin, acos, atan, atan2; radians) | [§1 Math](#1-math) |
| Print to stdout | `IO.inspect` (newline + quoting) / `IO.println` (newline, raw) / `IO.print` (raw, no newline) |
| Read back what a call printed | [§2 IO](#2-io) — `IO.capture(fn)` |
| Read a whole file | `FS.read` (throws on failure) |
| Stream a file (lines / chunks / seek) | [§4 File](#4-file) — `File.open` / `File.with` |
| Path manipulation (join, basename, dirname, stem, extension) | [§3 FS](#3-fs); fluent `Path` wrapper: [§3 `Path`](#path--the-fluent-wrapper) |
| Stat / walk / glob / copy / rename / symlink / chmod / chown | [§3 FS](#3-fs) |
| Watch a directory for changes | [§3 FS](#3-fs) — `FS.watch` |
| Directory listing / create / remove | `FS.list_dir`, `FS.mkdir`, `FS.remove` |
| Read a ZIP archive, the assets baked into the binary, or files held in memory, one way | [§41 Dir](#41-dir) — `Dir.zip(path)` / `Dir.embedded("assets")` / `Dir.memory({...})` |
| `Instant` / `Duration`, ISO 8601, calendar arithmetic | [§5 Time](#5-time) |
| Wrap an index that can go negative into `0..n` | [§1 Math](#1-math) — `Math.wrap(i, n)` (`%` truncates, so it stays negative) |
| Random numbers | `Random.int`, `.uniform`, `.gauss`, `.shuffle`, `.choice`, `.weighted_choice` |
| CLI argument parsing | [§10 Args](#10-args) |
| Process info | `Sys.argv`, `Sys.exit`, `Sys.env`, `Sys.set_env`, `Sys.getcwd`, `Sys.chdir`, `Sys.executable`, `Sys.script` |
| Save data that outlives the process | [§7 Sys](#7-sys) — `Sys.data_dir("myapp")` (the platform's per-user data directory) |
| Run an external command | [§11 Proc](#11-proc) — `Proc.run(["git", "status"])` |
| Call an HTTP/HTTPS API | [§15 Http](#15-http) — `Http.get("https://api.example/x")` |
| Serve HTTP — routes, static files, WebSocket | [§15 `Http.server()`](#httpserver---object) — `Http.server().get("/", h).listen(8080)` |
| Speak a raw TCP / UDP protocol, resolve a hostname | [§29 Net](#29-net) — `Net.connect(host, port)` / `Net.listen(port)` / `Net.udp()` / `Net.resolve(host)` |
| Escape / unescape HTML entities | [§16 Encoding](#16-encoding) — `Encoding.html.unescape("a & b")` |
| Encode / decode base64, hex, url | [§16 Encoding](#16-encoding) — `Encoding.base64.encode(s)` |
| gzip / gunzip data or files | [§17 Compress](#17-compress) — `Compress.gzip(s)` / `Compress.gunzip(z)` |
| Pack files into a ZIP archive | [§17 Compress](#17-compress) — `Compress.zip(files)` |
| Compress without the gzip envelope | [§17 Compress](#17-compress) — `Compress.deflate(s, level: 9)` (decode with `Compress.gunzip`) |
| Hash / checksum / HMAC | [§18 Hash](#18-hash) — `Hash.sha256(s)` / `Hash.hmac_sha256(key, s)` |
| Parse / write CSV | [§19 CSV](#19-csv) — `CSV.parse(text)` / `CSV.stringify(rows)` |
| Parse / write TOML | [§24 TOML](#24-toml) — `TOML.parse(text)` / `TOML.stringify(obj)` |
| Parse / search / write XML | [§25 XML](#25-xml): `XML.parse(text)` / `root.find_all(".//item")` / `el.stringify()` |
| Query an embedded SQL database | [§26 SQLite](#26-sqlite) — `SQLite.open(path)` → `db.query(sql, params)` / `db.execute(...)` |
| Load a `.env` config file | [§20 Env](#20-env) — `Env.load(".env")` / `Env.parse(text)` |
| Generate a UUID | [§21 UUID](#21-uuid) — `UUID.v4()` / `UUID.v7()` |
| Leveled / structured logging | [§23 Log](#23-log) — `Log.info("msg", {k: v})` / `Log.with({req: id})` |
| Run work on another thread (CPU parallelism) | [§12 Isolate](#12-isolate) — `Isolate.spawn(\|\| fib(40))` |
| Share fixed-layout data across threads/processes (zero copy) | [§12 SharedBuffer](#sharedbuffer--zero-copy-shared-fixed-layout-data) — `SharedBuffer.new(n, FloatPair)` / `.file` / `.shared` |
| Share variable-length read-only data across threads (no copy) | [§12 Shared](#shared--immutable-values-shared-by-reference) — `Shared.new(value)` |
| Handle Ctrl+C / SIGINT gracefully | [§12 Signal](#signal--signalnotify--signalreset) — `Signal.notify(tx)` / `Signal.reset()` |
| Desktop GUI (native WebView + local server) | [§30 Desktop](#30-desktop--webview) — `Desktop.run({title, assets, routes})` |
| Play a tone, a sound effect or music; synthesise audio | [§40 Audio](#40-audio) — `Audio.tone(440, 10)` / `Audio.Sound.new(bytes).play()` / `Audio.Music.new(bytes)` / `Audio.PCM.new(44100, 1, 1024)` / `Audio.Capture.new(44100)` |
| Heap introspection / leak checks | [§7 GC](#gc--heap-introspection) — `GC.stat()` → `{live_objects, rc_objects, heap_bytes}` |
| 2D/3D vector math (dot, length, normalize, distance) | [§31 `Vector2`](#31-vector2) / [§32 `Vector3`](#32-vector3) |
| FIFO queue, sliding window, front+back stack | [§33 `Deque`](#33-deque) — `Deque.new()` — `push_back`/`pop_front` |
| Priority scheduling, event simulation, shortest-path search | [§34 `PriorityQueue`](#34-priorityqueue) — `PriorityQueue.new()` — `push`/`pop` |
| Parse a language / config format of your own | [§35 PEG](#35-peg) — write a PEG grammar, `PEG.parser(grammar)`, `enable_ast()` and `parse(text)` → a tree `match` takes apart |
| Model a workflow, protocol or UI mode as states and events | [§37 `StateMachine`](#37-statemachine) — `StateMachine.parse(text)`, or a description `Object` |
| Autocomplete / prefix lookup over a large fixed word list | [§38 FST](#38-fst) — `FST.Set.new(FST.compile_set(words))` → `.predictive_search("hel")` |
| Spelling correction, fuzzy lookup | [§38 FST](#38-fst) — `.edit_distance_search(word, 1)` / `.suggest(word)` |
| Search many documents by keyword, ranked | [§39 Search](#39-search) — `Search.Index.new()` → `.add(key, text)` → `.search("quick -dog")` |
| String / Array / Object methods | [language spec §18](language.md) |
| Integer sequences (`range`, `iota`) | [language spec §19](language.md) |
| Fill an `Array` with `n` copies of a value | [language spec §19](language.md) — `repeat(n, value)` |
| Conversion (`to_long`, `to_float`, `to_string`, `type_of`) | [language spec §19](language.md) |
---
## 1. `Math`
Numeric utilities. Integer-only routines (`pow`, `sign`, `wrap`)
preserve `Long` input; `clamp` and the Float-domain routines (`log`,
`exp`, `sqrt`, …) accept either `Long` or `Float` and return the shape
documented below. See [§4](language.md#4-types) and
[§7](language.md#7-expressions) of the language spec for how `Long`
and `Float` interact.
Sub-groups in this section: **constants** (`Math.pi`, `Math.e`,
`Math.inf`, `Math.nan`) — **scalar ops** (`abs`, `min`, `max`,
`log`, `exp`, `sqrt`, `floor`, `ceil`, `round`, `pow`, `sign`,
`clamp`, `wrap`) — **trigonometry** (`sin`, `cos`, `tan`, `asin`, `acos`,
`atan`, `atan2`, in radians). Integer-sequence factories `range` / `iota`
are language-core globals — see [§19](language.md#19-core-built-in-functions).
### Constants
`Math.pi`, `Math.e`, `Math.inf`, `Math.nan` are `Float` properties.
Both backends evaluate these as compile-time constants under `--jit`.
`
Direct children of `path` as bare filenames (no `.` / `..`, no
prefix). Order is filesystem-defined — sort explicitly if needed.
Throws `IOError` if `path` is not a directory.
```culebra
# doctest: skip
let names = FS.list_dir('/tmp/build')
assert_true(names.contains('out.o'))
```
#### `FS.mkdir(path: String) -> Nil`
Create a directory at `path`, including any missing parents
(`mkdir -p` semantics). No-op if the directory already exists.
Throws `IOError` if the path exists as a file or creation fails.
#### `FS.temp_dir() -> String`
The directory this machine keeps scratch files in: `$TMPDIR` and its
fallbacks on POSIX, `GetTempPath` on Windows. No trailing separator, so
`FS.join` off it never doubles one. Throws `IOError` if the platform
names a directory that isn't one.
```culebra
assert_true(FS.is_dir(FS.temp_dir()))
```
#### `FS.mkdtemp(prefix: String) -> String`
Create a new directory under `FS.temp_dir()` whose name begins with
`prefix`, and return its path. The directory is *created*, not just
named, so two programs running at once cannot be handed the same one —
including two seeded identically with `Random.seed`, since the suffix
comes from the OS rather than the shared engine. Removing it is the
caller's job.
`prefix` is a name, not a path: a parent that doesn't already exist is
an `IOError` rather than a silent `mkdir -p`.
```culebra
let dir = FS.mkdtemp('build-')
assert_true(FS.is_dir(dir))
assert_eq(FS.dirname(dir), FS.temp_dir())
FS.remove(dir, recursive: true)
```
#### `FS.remove(path: String, recursive: Bool = false) -> Nil`
Remove a file or directory. By default removes a file or *empty*
directory and throws `IOError` if the directory is non-empty. With
`recursive: true` it removes a directory tree (`rm -rf`). Throws
`IOError` if the path doesn't exist or isn't removable.
```culebra
# doctest: skip
FS.remove('/tmp/build/out.o')
FS.remove('/tmp/build', recursive: true)
```
#### `FS.rename(src: String, dst: String) -> Nil`
Rename / move `src` to `dst` (atomic within one filesystem). Throws
`IOError` on failure.
#### `FS.copy(src: String, dst: String, recursive: Bool = false) -> Nil`
Copy a file, overwriting `dst` if it exists. With `recursive: true`
copies a directory tree. Throws `IOError` on failure.
#### `FS.chmod(path: String, mode: Long) -> Nil`
Set the permission bits of `path` to `mode` — an integer, usually written
as an octal literal (`0o755`, `0o644`). `mode` is masked to the low 12 bits
(`rwx` for owner/group/other plus setuid/setgid/sticky) and replaces the
existing bits. Read the current bits back with `FS.stat(path).mode`. Throws
`IOError` if the path is missing or the permissions can't be changed.
Only the owner-write bit crosses platforms. Windows stores a single
read-only attribute rather than permission bits, so every file there reads
back as `0o666` or `0o444` and a mode does not round-trip: `FS.chmod(p,
0o755)` followed by `FS.stat(p).mode` answers `0o666`. Clearing owner-write
(`0o444`) and setting it again is the part that behaves the same
everywhere.
```culebra
# doctest: skip
FS.chmod('deploy.sh', 0o755) # make executable
inspect(FS.stat('deploy.sh').mode) # 493 (0o755)
```
#### `FS.chown(path: String, owner = nil, group = nil) -> Nil`
Change the owner and/or group of `path`. `owner` and `group` each accept a
name (`String`), a numeric id (`Long`), or `nil` to leave that one unchanged.
Read the current ids back with `FS.stat(path).uid` / `.gid`. Changing the
owner usually requires root; changing the group works for a group you belong
to. Throws `IOError` on a missing path, an unknown user/group name, or a
permission failure; a non-String/Long/Nil argument is a `TypeError`.
```culebra
# doctest: skip
FS.chown('app.log', group: 'staff') # set group by name, keep owner
FS.chown('data', 'deploy', 'deploy') # set both by name (root)
```
### Stat / metadata
#### `FS.stat(path: String) -> Object`
Return `{size, is_dir, is_file, is_symlink, mtime, mode, uid, gid}` for
`path`. `size` is bytes (0 for non-regular files); `mtime` is seconds since
the Unix epoch; `mode` is the permission bits as an integer (compare with
octal, e.g. `st.mode == 0o644`); `uid` / `gid` are the owner and group ids;
`is_symlink` reflects the link itself while the other fields follow it. Throws
`IOError` if the path doesn't exist.
Windows has no POSIX owner model, so `uid` and `gid` are **absent** there
rather than carrying a stand-in value — `st.uid` reads as `nil`, and
`st.has('uid')` is what asks whether this platform has them. `mode` exists
everywhere but means less on Windows, which keeps one read-only attribute
instead of permission bits: see `FS.chmod` below.
```culebra
# doctest: skip
let st = FS.stat('config.toml')
inspect(st.size)
```
### Recursive traversal
#### `FS.walk(path: String) -> Array`
Every path under `path`, recursive, depth-first. Each entry is a full
path. A directory the process may not read is listed but not entered.
Throws `IOError` if `path` isn't a directory or can't be read, or if
reading one under it fails for any other reason.
#### `FS.glob(pattern: String) -> Array`
Paths matching a glob `pattern`, sorted. Supports `*`, `?`, `[...]`
per segment and `**` for recursive descent; a character is a code
point, so `?` matches `é` whole. This is shell-style glob,
distinct from `Regex`.
```culebra
# doctest: skip — reads the working directory
let sources = FS.glob('src/**/*.cul')
```
### Watching for changes
#### `FS.watch(path: String, recursive: Bool = true, match: Array? = nil) -> WatchHandle`
Watch a directory for changes. Iterating the handle blocks until the
next change and yields `{path, kind}`, where `path` is absolute and
canonical (symlinks resolved, so a `/tmp` root reports under
`/private/tmp` on macOS) and `kind` is `'created'`, `'modified'` or
`'deleted'`. `match` keeps only the paths ending in one of the given
extensions, with or without the leading dot (`'cul'` and `'.cul'` are
the same filter); the check happens before an event is queued, so a
rejected change never reaches the queue. Throws `IOError` if `path`
isn't a directory, `TypeError` if `match` isn't an array of strings.
`recursive: false` narrows the watch to the directory's own entries: a
change *below* a subdirectory is not reported, though the subdirectory
entry itself may be — Windows reports its timestamp moving when a file
inside it is written, where inotify and FSEvents do not.
Backed by FSEvents on macOS, inotify on Linux and `ReadDirectoryChangesW`
on Windows.
```culebra
# doctest: skip — watches until interrupted
for e in FS.watch('src', match: ['.cul']) {
println("{e.kind} {e.path}")
}
```
Events queue from the moment the watch opens, so a change made before
the first pull is still delivered. The handle is iterable directly, and
`break` leaves it usable — a named handle can be iterated again:
```culebra
# doctest: skip — watches until interrupted
let w = FS.watch('src')
for e in w { break } # handle stays open
for e in w { break } # picks up where the first loop left off
w.close()
```
`close()` stops the watch; a handle that goes out of scope closes
itself. An anonymous `for e in FS.watch(p)` is therefore stopped when
the loop ends. A watch belongs to the thread that opened it and cannot
be sent to an isolate.
Three things are deliberately not promised. **Kinds are best effort at
the OS's granularity**: FSEvents coalesces several changes to one path
inside a short window into a single event, while inotify reports each
separately, so the same edit can surface as one event on one platform
and two on another. **A rename is not paired** — it arrives as a
`deleted` on the old path and a `created` on the new one. **Nothing is
deduplicated or debounced**, and the queue is unbounded: a consumer
that pulls slower than the tree changes grows memory rather than
dropping events.
An iteration that is simply waiting can only be ended by `break`, by
Ctrl+C, or by cancelling the isolate it runs in — there is no timeout
or non-blocking pull.
### Path resolution
#### `FS.abspath(path: String) -> String`
Absolute, normalized form of `path` (relative to the current
directory). Does not resolve symlinks.
#### `FS.realpath(path: String) -> String`
Canonical path with symlinks resolved (`weakly_canonical`; missing
trailing components are kept).
#### `FS.normpath(path: String) -> String`
Lexically normalize (collapse `.` / `..` / duplicate separators)
without touching the filesystem.
#### `FS.is_abs(path: String) -> Bool`
Whether `path` is absolute.
### Symlinks
#### `FS.symlink(target: String, link: String) -> Nil`
Create a symbolic link at `link` pointing to `target`.
#### `FS.readlink(path: String) -> String`
Read a symlink's target. Throws `IOError` if `path` isn't a symlink.
#### `FS.is_symlink(path: String) -> Bool`
Whether `path` is a symbolic link (the link itself, not its target).
### `Path` — the fluent wrapper
`Path` is a thin, **immutable** wrapper over the `FS.*` helpers. `FS` is
the primitive layer (raw `String` paths in, `String`/values out); `Path`
is the sugar layer that carries the path around so you stop threading path
strings by hand. Every operation returns a fresh `Path` (or a `String` for
components) — a `Path` is never mutated in place.
Reach for `FS.*` in short scripts; reach for `Path` when a path is built
up in stages or passed through several operations, where the `/` operator
and property chains read more clearly:
```culebra
# doctest: skip
let root = Path.new(FS.dirname(Sys.script)).resolve()
for src in root.glob("*/content.src.js") {
let dst = src.parent / "content.js" # vs FS.join(FS.dirname(src), …)
dst.write(transform(src.read()))
IO.print("{src.parent.name}/content.js\n") # vs FS.basename(FS.dirname(src))
}
```
Construct with `Path.new(s)`, where `s` is a `String` or another `Path`.
Anywhere a method takes a path (`join`, `/`, `rename`, `==`) it likewise
accepts a `String` or a `Path`. String interpolation (`"{p}"`) and
`to_string(p)` yield the raw path string.
| Member | Returns | Delegates to |
|---|---|---|
| `p / other` / `p.join(other)` | `Path` | `FS.join` |
| `p.name` | `String` (final component) | `FS.basename` |
| `p.stem` | `String` (name without suffix) | `FS.stem` |
| `p.suffix` | `String` (extension incl. dot) | `FS.extension` |
| `p.parent` | `Path` | `FS.dirname` |
| `p.resolve()` | `Path` (absolute) | `FS.abspath` |
| `p.exists()` / `p.is_file()` / `p.is_dir()` | `Bool` | `FS.*` |
| `p.read()` / `p.write(content)` | `String` / `Nil` | `FS.read` / `FS.write` |
| `p.mkdir()` | `Nil` (creates parents) | `FS.mkdir` |
| `p.remove(recursive=false)` | `Nil` | `FS.remove` |
| `p.rename(dst)` | `Path` (the destination) | `FS.rename` |
| `p.list()` | `Array` (dir entries) | `FS.list_dir` |
| `p.glob(pattern)` | `Array` | `FS.glob` |
| `p.walk()` | `Array` (recursive) | `FS.walk` |
| `p.str()` | `String` (escape hatch) | — |
`name`, `stem`, `suffix`, and `parent` are **getters** (pure string
derivations), read without parentheses — `p.parent.name`, not
`p.parent().name()` (the call spelling still works). The filesystem
operations below stay methods because they do I/O and can throw.
Two `Path`s compare by their inner path string: `==` (also against a
`String`) and `<` / `<=` / `>` / `>=`, so a `Path` array sorts
(`paths.sorted()`) and works with `min` / `max` / `Set`. Ordering against a
non-path is a `TypeError` (no meaningful answer), whereas `==` against a
non-path is simply `false`.
The `FS.*` helpers and `File.open` / `File.with` accept a `Path` anywhere
they take a path — their path parameters are `String | Path` — so a `Path`
flows straight through without `.str()`:
```culebra
# doctest: skip
let cfg = Path.new("/etc") / "app.conf"
let text = FS.read(cfg) # FS.read(String | Path)
for line in File.open(cfg).lines() {
} # File.open(String | Path)
```
Only the *path-taking* stdlib functions opt in this way. `Path` and
`String` remain **distinct types** everywhere else: a plain `fn(x: String)`
does *not* accept a `Path`, so the type boundary stays meaningful — use
`p.str()` (or `"{p}"`) to hand a raw string to a `String`-only API.
---
## 4. `File`
A `File` is a **stateful handle** for streaming I/O — the complement
to `FS`'s one-shot whole-file operations (`FS.read` / `FS.write`).
Open one with `File.open` or the scoped `File.with`. All I/O is binary
(no text-mode newline translation); `String` is a byte string, so any
content round-trips.
The handle (`f` below) implements four method groups — **Reader** (`read` /
`lines` / `chunks`), **Writer** (`write` / `flush`), **Seekable**
(`seek` / `tell`), **Closeable** (`close`). Which are valid depends on
the open mode.
### Opening
#### `File.open(path: String, mode: String = "r") -> File`
Open `path`. `mode` is `"r"` (read), `"w"` (truncate + write), or
`"a"` (append). Throws `ValueError` for any other mode, `IOError` if
the file can't be opened.
#### `File.with(path: String, mode: String = "r", fn: Function) -> Any`
Open `path`, call `fn(handle)`, and close the handle on every exit
path (normal, `return`, or exception). Returns `fn`'s value. This is
the native equivalent of `open` + `defer { close }`, and the clearest
choice when the handle's lifetime fits one block.
```culebra
# doctest: skip
let head = File.with('big.log', 'r', fn (f) {
f.read(256)
})
```
### Resource safety — three ways to close
| Pattern | Use when |
|---|---|
| `File.with(p, m, fn (f) { … })` | the handle's lifetime fits one block |
| `let f = File.open(p, m); defer { f.close() }` | the handle outlives a block / mixes with other logic |
| `for line in File.open(p).lines() { … }` | streaming; the iterator closes on loop exit (incl. `break`) |
A handle never explicitly closed is closed by a GC backstop, but don't
rely on it — prefer one of the three patterns above.
### Reader methods
#### `f.read() -> String` / `f.read(n: Long) -> String`
Streaming read from the current position: `read()` returns the rest of
the file, `read(n)` at most `n` bytes (fewer at EOF). For a one-shot
whole-file read without a handle, use `FS.read(path)`.
#### `f.lines() -> Iterator`
Iterate lines, each with its trailing newline stripped (`\n`, `\r\n`,
and `\r` are all recognized). The iterator owns the handle and closes
it when the loop ends or breaks.
```culebra
# doctest: skip
for line in File.open('access.log').lines() {
if line.contains('ERROR') {
inspect(line)
}
}
```
#### `f.chunks(n: Long) -> Iterator`
Iterate fixed-size byte chunks of at most `n` bytes (the last may be
shorter). Same close-on-exit contract as `lines()`.
### Writer methods
#### `f.write(data: String) -> Nil`
Write `data` at the current position (raw bytes, no newline
translation). Throws `IOError` on a read-only handle.
#### `f.flush() -> Nil`
Flush buffered writes to the OS.
### Seekable methods
#### `f.seek(offset: Long, whence: String = "set") -> Nil`
Move the cursor. `whence` is `"set"` (from start), `"cur"` (relative),
or `"end"` (from end; use a negative `offset`).
#### `f.tell() -> Long`
Current byte offset.
### Closeable
#### `f.close() -> Nil`
Close the handle, flushing writes. Idempotent — closing twice is a
no-op. Operating on a closed handle throws `IOError`.
---
## 5. `Time`
Wall-clock + monotonic time, ISO 8601 round-trip, calendar
arithmetic. The module exposes two classes — `Instant` (a point in
time) and `Duration` (an interval) — backed internally by `i64`
nanoseconds since the Unix epoch (range ±292 years, full nanosecond
precision). A result past that range raises `ValueError` instead of
wrapping around.
Timezone handling is **UTC + local only** (named zones like
`Asia/Tokyo` are deferred). Methods accept a kw-only `utc:` flag;
`iso` defaults to UTC (`utc: true`) because Z-suffixed ISO 8601 is
the common interop wire form, others default to local.
### Acquisition
#### `Time.now() -> Instant`
Current wall-clock time. Subject to NTP / manual clock adjustments —
use `Time.monotonic` for measuring elapsed time.
#### `Time.monotonic() -> Float`
Seconds (with sub-second precision) elapsed since the first call /
process start. Strictly non-decreasing; immune to wall-clock
changes. The primary tool for benchmarks and timeouts.
```culebra
# doctest: skip
let t0 = Time.monotonic()
do_work()
inspect("elapsed: {Time.monotonic() - t0} s")
```
#### `Time.sleep(secs: Float) -> Nil`
Block the current thread for at least `secs` seconds. Negative or
zero is a no-op.
### `Instant` constructors
#### `Time.from_iso(s: String) -> Instant`
Parse an ISO 8601 timestamp. Accepted variants:
- `2026-05-20T15:30:00Z`
- `2026-05-20T15:30:00.123Z`
- `2026-05-20T15:30:00.000123456Z` (full nanosecond precision)
- `2026-05-20T15:30:00+09:00`
- `2026-05-20T15:30:00-0900`
- `2026-05-20` (date only — UTC midnight)
- `2026-05-20T15:30` (seconds omitted)
Throws `ValueError` on a malformed input, including a field outside its
calendar range (`2026-13-01`, `2025-02-29`, `T25:00`) — it is not carried
into the next month or day.
#### `Time.from_unix(secs: Long|Float) -> Instant`
From Unix epoch seconds. A Long is exact; a Float gives sub-second
precision, rounded to the nearest nanosecond.
#### `Time.from_parts(p: Object, utc: Bool = false) -> Instant`
Compose from a parts dict — the inverse of `Instant.parts`.
Recognised keys: `year`, `month`, `day`, `hour`, `minute`, `second`,
`nanosecond` (defaults: `year=1970`, `month=1`, `day=1`, others 0). Extra keys
are ignored. Each recognised key must be a Long (`TypeError`
otherwise) within its calendar range — `month` 1–12, `day` up to that
month's length, `hour` 0–23, `minute` and `second` 0–59, `nanosecond`
below 10⁹ — or it raises `ValueError`.
#### `Time.parse(s: String, fmt: String) -> Instant`
Strict strftime parse for non-ISO inputs. The format follows POSIX
`strptime`. Throws `ValueError` if `s` doesn't match `fmt`, if
characters are left over after `fmt` is used up, or if a field is
outside its calendar range (the ranges of `Time.from_parts` —
`2026-02-30` is refused, not carried into March). A field `fmt` does
not mention defaults as in `Time.from_parts` (`year=1970`, `month=1`,
`day=1`, others 0). Result is interpreted as local time.
```culebra
Time.parse("2026/05/20 15:30:00", "%Y/%m/%d %H:%M:%S")
```
### `Instant` methods
#### `t.iso(utc: true) -> String`
Format as ISO 8601 with full nanosecond precision (fractional
component omitted when zero). UTC by default (`...Z`); pass
`utc: false` for local time with `±HH:MM` offset.
#### `t.format(fmt: String, utc: false) -> String`
Format with a strftime format string. Local time by default.
```culebra
# doctest: skip
t.format("%Y-%m-%d %H:%M:%S") # local
t.format("%Y%m%d", utc: true) # 20260520
```
#### `t.parts(utc: false) -> Object`
Decompose into `{year, month, day, hour, minute, second,
nanosecond, weekday, dayofyear}`. `weekday` follows ISO 8601
(`0=Mon`, `6=Sun`); `dayofyear` is 1-based (`1..366`).
```culebra
let p = Time.now().parts()
if p.hour >= 9 && p.hour < 17 {
inspect("business hours")
}
```
#### `t.weekday(utc: false) -> Long`
Just the weekday component (0=Mon..6=Sun) — avoids the `parts()`
allocation when only the weekday is needed.
#### `t.add(years=0, months=0, days=0, hours=0, minutes=0, seconds=0, utc: false) -> Instant`
Calendar arithmetic. `years` / `months` use **clamp-to-end-of-month**
semantics: `2026-01-31 + 1 month → 2026-02-28`,
`2024-01-31 + 1 month → 2024-02-29` (leap year). Sub-day fields
compose as straightforward addition.
```culebra
let next_month = Time.now().add(months: 1)
let next_quarter = Time.now().add(months: 3)
let next_year = Time.now().add(years: 1)
```
#### `t.start_of(unit: String, utc: false) -> Instant`
Truncate to the start of a calendar unit. `unit` ∈ `"year"` /
`"month"` / `"day"` / `"hour"` / `"minute"`. Throws `ValueError`
on any other unit.
```culebra
# doctest: skip
let day_bucket = t.start_of("day")
let hour_bucket = t.start_of("hour")
```
#### `t.unix() -> Float`, `t.unix_nanos() -> Long`
Unix epoch as Float seconds (lossy past ~400ns near current epoch)
or Long nanoseconds (lossless).
### `Duration` constructors
```culebra
# doctest: skip
Time.seconds(n) # n seconds
Time.milliseconds(n)
Time.minutes(n)
Time.hours(n)
Time.days(n)
```
`n` may be Long or Float — a Long scales exactly, and fractional units
round to the nearest nanosecond.
### `Duration` methods
#### `d.seconds() / .milliseconds() / .minutes() / .hours() / .days() -> Float`
Express the duration in the requested unit (always Float so
fractional units round-trip).
#### `d.abs() -> Duration`
Absolute value of a (possibly negative) duration.
### Operator overloads
```culebra
# doctest: skip
let t = Time.now()
let one_hour = Time.hours(1)
t + one_hour # Instant + Duration → Instant
t - one_hour # Instant - Duration → Instant
t1 - t2 # Instant - Instant → Duration
Time.minutes(1) + Time.seconds(30) # → Duration (90s)
one_hour * 2 # → Duration (Long: exact)
one_hour / 2 # → Duration (Long: truncates)
-one_hour # → Duration
a < b, a <= b, a == b # natural ordering on both types
```
Both classes wrap a nanosecond count, but they are not interchangeable:
a combination with no meaning — `t1 + t2`, `t < one_hour`, or a bare
number on either side — raises a `TypeError` naming what it got instead
of quietly doing the arithmetic. Equality is the exception: `==` against
any other type is `false`, never an error.
---
## 6. `Random`
Random-number generation. The process has a single shared
Mersenne-Twister-64 engine, shared between the VM and JIT
backends; `Random.seed(n)` resets it and makes every subsequent draw
reproducible within one program execution. Without a `seed` call the
engine is initialised from `std::random_device`.
### `Random.seed(n: Long) -> Nil`
Reseed the shared PRNG. Same `n` → same sequence.
```culebra
Random.seed(42)
```
### `Random.int(lo: Long, hi: Long) -> Long`
Uniform integer in the half-open range `[lo, hi)`. Requires `hi > lo`,
otherwise `type error`.
```culebra
Random.seed(0)
inspect(Random.int(0, 10)) # 0..9
```
### `Random.uniform(lo: Float, hi: Float) -> Float`
Uniform real in the half-open range `[lo, hi)`. `Long` arguments are
accepted and promoted to `Float`.
### `Random.gauss(mu: Float, sigma: Float) -> Float`
A sample from a Gaussian distribution with the given mean and
standard deviation. `Long` arguments are promoted to `Float`.
```culebra
Random.gauss(0.0, 1.0) # standard normal
```
### `Random.shuffle(a: Array) -> Nil`
Fisher–Yates in-place permutation. Returns `nil`; the argument is
mutated.
### `Random.weighted_choice(pop: Array, weights: Array) -> Any`
Draw a single element from `pop` with probability proportional to the
matching `weights` entry. Weights must all be numeric and of the same
length as `pop`; empty or mismatched inputs raise `type error`.
Weights of `0` are never selected.
```culebra
Random.weighted_choice(['hit', 'miss'], [1, 9]) # ~10% 'hit'
```
### `Random.choice(pop: Array) -> Any`
Draw a single element from `pop` with uniform probability. An empty
`pop` raises `type error`.
```culebra
Random.seed(0)
Random.choice(['rock', 'paper', 'scissors'])
```
---
## 7. `Sys`
Process-level information.
### `Sys.argv -> Array`
Array of `String` arguments passed to the script on the command
line. The first non-flag argument is the script path; **everything
after it** is captured as `argv`.
culebra's own flags (`--jit`, `--debug`, …) must precede the script
path. Empty when the script is invoked with no trailing arguments or
when running in the REPL.
```culebra
# doctest: skip — prints values particular to one machine
# $ culebra run.cul hello world
inspect(Sys.argv) # ['hello', 'world']
# $ culebra --jit run.cul hello → ['hello'] (--jit is culebra's)
```
A standalone `--` is an optional escape hatch: it stops flag parsing,
so the next argument becomes the script even if it begins with a dash
(e.g. `culebra -- -weird.cul`). It is otherwise unnecessary.
A binary built by `culebra build` reports the arguments it was run
with, the same way and with the same skip of the program name. The
value is process-wide, so an [isolate](#12-isolate) or an HTTP handler
running on a worker thread reads the same array as the main thread.
### `Sys.exit(code: Long) -> Nil`
Terminate the process immediately with the given exit code. Does
not return; pending `defer` statements are *not* run.
```culebra
# doctest: skip
if error_occurred {
Sys.exit(1)
}
```
### `Sys.env(name: String, fallback = '') -> Any`
Return the value of the environment variable `name`. If it is not set,
return `fallback` unchanged — the default of `''` keeps the one-argument
form returning a `String`.
`fallback` is returned as-is whatever its type, so passing `nil` is what
separates an unset variable from one set to the empty string: the
one-argument form folds both to `''`.
```culebra
# doctest: skip — prints values particular to one machine
inspect(Sys.env('HOME')) # '/Users/alice'
inspect(Sys.env('NOT_A_VAR')) # ''
inspect(Sys.env('PORT', '8080')) # '8080' while PORT is unset
inspect(Sys.env('NOT_A_VAR', nil)) # nil — unset
inspect(Sys.env('SET_BUT_EMPTY', nil)) # '' — set, and empty
```
### `Sys.set_env(name: String, value: String) -> Nil`
Set the environment variable `name` to `value`, overwriting any existing
value. The change is visible to this process (via `Sys.env`) and to child
processes spawned afterwards (e.g. `Proc.run`). Raises `IOError` on failure
(for example, an invalid variable name).
```culebra
Sys.set_env('CULEBRA_MODE', 'fast')
inspect(Sys.env('CULEBRA_MODE')) # => 'fast'
```
### `Sys.getcwd() -> String`
Return the absolute path of the current working directory. Symlinks in the
path are resolved. Raises `IOError` if the directory cannot be determined
(for example, it was removed out from under the process).
```culebra
# doctest: skip — prints values particular to one machine
inspect(Sys.getcwd()) # '/Users/alice/project'
```
### `Sys.chdir(path: String) -> Nil`
Change the process working directory to `path`. Raises `IOError` if the path
does not exist or is not a directory.
```culebra
# doctest: skip — prints values particular to one machine
Sys.chdir('/tmp')
inspect(Sys.getcwd()) # '/tmp' (or its resolved path)
```
### `Sys.data_dir(app: String) -> String`
Return the per-user directory where `app` keeps data that outlives the process
— a save file, a high score, a settings blob. The convention is the platform's:
| platform | directory |
|---|---|
| macOS | `~/Library/Application Support/` |
| Windows | `%APPDATA%\` (falling back to `%USERPROFILE%\AppData\Roaming\`) |
| other | `$XDG_DATA_HOME/`, or `~/.local/share/` when it is unset |
The directory is **not created** — `FS.mkdir` makes the parents when the program
first writes. `app` must be a single path segment: an empty name, one holding a
`/`, `\` or `:`, and `.` / `..` all raise `ValueError`, so the result cannot
point outside the base directory (a Windows drive letter would otherwise
replace the base rather than extend it). Raises `IOError` when the platform's base directory
cannot be determined (no `HOME`, or no `APPDATA` and no `USERPROFILE`).
```culebra
# doctest: skip
let dir = Sys.data_dir('samegame')
FS.mkdir(dir)
FS.write(FS.join(dir, 'scores.json'), JSON.stringify(scores))
inspect(dir) # '/Users/alice/Library/Application Support/samegame'
```
### `Sys.executable -> String`
Absolute path to the running culebra binary. Use it to launch a worker copy of
culebra — e.g. `Proc.run([Sys.executable, "worker.cul"], ...)` — instead
of relying on `culebra` being on `PATH`. (In an AOT-built program it is the path
to that standalone binary.)
```culebra
# doctest: skip — prints values particular to one machine
inspect(Sys.executable) # '/usr/local/bin/culebra'
```
### `Sys.script -> String?`
Absolute path of the running script — the `__file__` analogue. Use it to resolve
files next to the script instead of relying on the current working directory:
`FS.join(FS.dirname(Sys.script), "data.txt")`. It is `nil` when there is no
source file at runtime — the REPL, a piped `stdin`, or an AOT-built binary (which
carries no `.cul`; use `Sys.executable` there).
```culebra
# doctest: skip — prints values particular to one machine
inspect(Sys.script) # '/Users/alice/project/build.cul'
```
### `Sys.time() -> Float`
Seconds elapsed on a **monotonic** clock, anchored at the first call in
the process. It never jumps backwards with a wall-clock adjustment, which
makes it the right tool for timing a block of code; use `Time.now()` (§5)
when you need an actual date. `Time.monotonic()` (§5) reads the same
monotonic clock; each function anchors its own origin at its first
call, so subtract two readings of the *same* function rather than
mixing the two.
```culebra
let t0 = Sys.time()
let sum = range(1000).reduce(0, |a, x| a + x)
inspect(Sys.time() - t0 >= 0.0) # => true
```
### `GC` — heap introspection
`GC.stat()` runs a full collection and returns an `Object` describing the
live heap right after it:
| key | type | meaning |
|---|---|---|
| `live_objects` | `Long` | number of reachable heap objects |
| `rc_objects` | `Long` | reachable *refcounted* objects (excludes traced-only Strings/StringViews) |
| `heap_bytes` | `Long` | bytes those objects occupy |
Because it collects first, the numbers report *reachable* state, not cycle
residue still awaiting sweep. The call itself allocates the result `Object`,
so back-to-back readings differ by a small constant — measure a delta around
the code under test rather than an absolute count.
The collection finds its roots from the reference counts (CPython's
`gc_refs`), not from a scan of the stack, so which of the program's
objects survive it — and which orphaned `drop`s it fires — is decided by
the reference counts alone: the same on every backend and every run.
`rc_objects` is therefore deterministic for a given program, and the
executor and `--jit` report the same figure. `live_objects` also counts
Strings, which are traced from a conservative stack scan and can straggle
by a few; the background collections between two calls are conservative
too, and cheaper.
```culebra
# doctest: skip
let base = GC.stat().live_objects
build_some_structure()
inspect(GC.stat().live_objects - base) # objects retained by the structure
```
This is the foundation for leak-regression tests: assert that a delta
stays bounded across many iterations. Memory is otherwise managed
automatically — see the language spec for the memory model and
deterministic `drop`.
`GC.refcount(value)` reports how many references a heap value is holding
open, or `nil` for one that carries no count — `Long`, `Float`, `Bool` and
`nil` are immediates, and a `String` is traced rather than counted. It
answers the question `GC.stat()` cannot: a value that is never collected —
anything reachable from a namespace, say — contributes no object growth
however many references pile up on it, so a reference that leaks against
such a value is invisible to a count of objects.
```culebra
let obj = {a: 1}
let base = GC.refcount(obj)
for _ in range(100) {
let alias = obj
}
inspect(GC.refcount(obj) - base) # => 0
```
Read it as a delta, never as an absolute: the call is holding a reference
of its own while it answers, and a value the runtime roots reads far above
what the program can see.
---
## 8. `Tensor`
N-dimensional numeric tensor. Builds a lazy computation graph and
launches BLAS / vDSP kernels through `Tensor.eval(...)` to materialize
values. Supported dtypes are `Float32` (default) and `Float64`.
Shapes can be supplied as variadic args or as an `[m, n]` Array.
`transpose`, `slice`, and `reshape` produce zero-copy views.
```culebra
let A = Tensor.from([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) # [2, 3]
let B = Tensor.from([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) # [3, 2]
let C = A.dot(B) + 1.0 # lazy: builds the graph only
Tensor.eval(C) # BLAS GEMM runs here
inspect(C.shape()) # => [2, 2]
inspect(C.to_array()) # => [[5.0, 6.0], [11.0, 12.0]]
```
### Construction (namespace functions)
#### `Tensor.zeros(...) -> Tensor` / `Tensor.ones(...)` / `Tensor.randn(...)`
The shape is variadic (`Tensor.zeros(3, 4)`) or an Array
(`Tensor.zeros([3, 4])`). The dtype is a string tag placed as the
**first argument**, Julia-style. `"f32"` is the only dtype (float64
has no fast path on GPU backends):
```culebra
let a = Tensor.zeros(3, 4) # F32 default
let a32 = Tensor.zeros("f32", 3, 4) # explicit
let dims = [3, 4]
let b = Tensor.zeros(dims) # computed shape
let r = Tensor.randn(2, 3) # standard normal
```
#### `Tensor.from(arr: Array) -> Tensor`
Converts a nested Culebra Array into a Tensor. Accepts 1D
(`[1.0, 2.0]`) or 2D (`[[1.0, 2.0], [3.0, 4.0]]`); stored as F32:
```culebra
let v = Tensor.from([1.0, 2.0, 3.0, 4.0]) # [4]
let m = Tensor.from([[1.0, 2.0], [3.0, 4.0]]) # [2, 2]
```
#### `Tensor.concat(parts: Array, axis: Long = 0) -> Tensor`
Stacks tensors along `axis` (rows, by default) into one Tensor. Every
part must share the same dtype, rank, and every dim but `axis`; the
result's `axis` length is the sum of the parts'. Differentiable — the
gradient's own `axis` window routes back to each part.
```culebra
let a = Tensor.from([[1.0, 2.0], [3.0, 4.0]]) # [2, 2]
let b = Tensor.from([[5.0, 6.0]]) # [1, 2]
let c = Tensor.concat([a, b]) # [3, 2]
let k1 = Tensor.randn(2, 4, 8) # [B, T, D]
let k2 = Tensor.randn(2, 1, 8) # one new position
let kv = Tensor.concat([k1, k2], 1) # [2, 5, 8] — a KV-cache append
```
#### `Tensor.where(cond: Tensor, a, b) -> Tensor`
Elementwise select, broadcasting all three operands against each other the
same way a binop does: `cond != 0 ? a : b`. `a`/`b` may be scalars, lifted
against `cond`'s dtype. The building block for masking (attention masks,
padding masks). Differentiable — `da = g*cond`, `db = g*(1-cond)`; `cond`
itself gets no gradient, matching numpy/PyTorch's own `where()`.
```culebra
let mask = Tensor.from([[1.0, 0.0, 1.0]]) # [1, 3], broadcasts over rows
let scores = Tensor.randn(4, 3)
let masked = Tensor.where(mask, scores, -1.0e9) # padding-mask pattern
```
#### `Tensor.adam_step(p, m, v, g, lr, beta1, beta2, eps, step) -> Nil`
Adam's update for one parameter, in place: the moments `m` and `v` advance on
the gradient `g`, then `p` moves by the bias-corrected ratio. All four share
one shape and dtype; `p`, `m` and `v` are written through, so none of them may
be a view and all three must already be evaluated. `step` is 1-based — the
exponent the bias correction uses.
Written out, a step is `m = m*beta1 + g*(1-beta1)`, `v = v*beta2 +
g*g*(1-beta2)` and `p -= m/(1-beta1^step) * lr / (sqrt(v/(1-beta2^step)) +
eps)`: about fifteen operations per parameter, each allocating a buffer the
size of the parameter. This is one pass over the four, and one kernel launch
on CUDA.
```culebra
# doctest: skip
for i in range(params.size()) {
let p = params[i]
Tensor.adam_step(p, moments[i], velocities[i], p.grad(),
lr: 3.0e-4, beta1: 0.9, beta2: 0.95, eps: 1.0e-8,
step: t)
}
```
#### `Tensor.index_add(indices: Tensor, values: Tensor, target_shape: Array) -> Tensor`
`.index_select()`'s exact dual: scatter-adds `values` (shaped like some
`t.index_select(indices)`) into a fresh zero buffer of `target_shape`.
Repeated indices accumulate — this is the embedding-table gradient.
Differentiable w.r.t. `values` (`.index_select(indices)` on the incoming
gradient); `indices` gets no gradient.
```culebra
let idx = Tensor.from([1.0, 1.0, 0.0]) # row 1 hit twice: accumulates
let values = Tensor.from([[10.0, 20.0], [30.0, 40.0], [1.0, 2.0]])
let grad = Tensor.index_add(idx, values, [4, 2]) # [[1,2],[40,60],[0,0],[0,0]]
```
#### `Tensor.scatter_to_axis(indices: Tensor, values: Tensor, size: Long) -> Tensor`
One-hot scatter into a new trailing axis of length `size`: `out[..., k] =
values[...]` where `indices[...] == k`, else 0. `indices`/`values` must
share the same shape. The native primitive a pooling layer's own
hand-derived backward composes with `.fold()` to reconstruct a padded
input gradient — `.max(axis)`/`.argmax(axis)` pick one element out of a
window; this scatters a gradient back into that same window shape.
Differentiable w.r.t. `values`: each one reaches exactly one slot, so the
gradient picks that slot back out. `indices` gets no gradient.
```culebra
let idx = Tensor.from([2.0, 0.0, 1.0])
let values = Tensor.from([5.0, 7.0, 9.0])
let out = Tensor.scatter_to_axis(idx, values, 4)
# [[0,0,5,0], [7,0,0,0], [0,9,0,0]]
```
#### `Tensor.from_csv(path: String) -> Tensor`
Reads a CSV file directly into a contiguous Tensor. Always returns
**rank-2** — a single-column CSV becomes `[N, 1]` (the bias-vector
form). Skips the nested Array intermediary, which is 3-5× faster than
the `Tensor.from(load_2d(path))` pattern (measured on MNIST):
```culebra
# doctest: skip
let W1 = Tensor.from_csv("W1.csv") # [30, 784]
let b1 = Tensor.from_csv("b1.csv") # [30, 1]
let X = Tensor.from_csv("X.csv") # [N, 784]
```
#### `Tensor.eval(t1, t2, ...) -> Nil`
Takes a variable number of Tensor arguments and evaluates the
dependency graph in topological order. Shared subexpressions are
computed once. Call this **at least once per mini-batch boundary** in
a training loop — otherwise the graph accumulates and memory grows
unbounded.
```culebra
# doctest: skip
W2 -= d2.dot(a1.transpose()) * lr
b2 -= d2.sum(1).reshape([N_OUT, 1]) * lr
W1 -= d1.dot(xb.transpose()) * lr
b1 -= d1.sum(1).reshape([N_HID, 1]) * lr
Tensor.eval(W1, b1, W2, b2) # evaluate all four in one pass
```
### Activation functions
Instance methods on a Tensor. A user class may still define its own
`relu` / `sigmoid` / `softmax` (e.g. microGPT's `Value.relu()`) — method
lookup gives the class method priority over the builtin, so there is no
clash.
```culebra
# doctest: skip
let h = z.sigmoid() # 1/(1+exp(-z)) elementwise
let r = x.relu() # max(0, x)
let p = logits.softmax() # over the last axis, online-stable
let l = p.log() # natural log, elementwise
let t = z.tanh() # hyperbolic tangent
let s = z.sin() # elementwise sin/cos
let c = z.cos()
let m = z.clamp(-1.0, 1.0) # elementwise clip to [lo, hi]
```
### Tensor methods
Shape ops, linear algebra, and reductions use method syntax:
| Method | Returns | Description |
|---|---|---|
| `.shape() -> Array` | Array of Long | shape as an Array |
| `.dot(other: Tensor) -> Tensor` | lazy | matrix product; rank-1/2, or batched (rank >= 3, leading dims matching exactly) |
| `.linear_sigmoid(x, b) -> Tensor` | lazy | fused `sigmoid(self @ x + b)` |
| `.pow(exp) -> Tensor` | lazy | elementwise power; `exp` is Tensor or scalar |
| `.gt(other) / .lt(other) / .ge(other) / .le(other) / .eq(other) / .ne(other) -> Tensor` | lazy | elementwise comparison, `1.0` / `0.0` mask; `other` is Tensor or scalar |
| `.tanh() / .sin() / .cos() -> Tensor` | lazy | elementwise trig / hyperbolic tangent |
| `.clamp(lo, hi) -> Tensor` | lazy | elementwise clip to `[lo, hi]` |
| `.rope(pos: Long, base) -> Tensor` | lazy | rotary position embedding (half-split convention) over the last axis; `self` is `[H, D]` or `[H, T, D]`, row `r`'s position is `pos + (r % T)` |
| `.causal_attention(k, v) -> Tensor` | lazy | causal self-attention: `softmax(self·kᵀ / sqrt(D))·v` with row `t` attending rows `0..t`; `self`, `k`, `v` share one shape, `[H, T, D]` or `[T, D]` — see below |
| `.layer_norm(gamma, beta) -> Tensor` | lazy | layer normalization over the last axis: `(self - mean) / sqrt(var + 1e-5) · gamma + beta` per row; `gamma`, `beta` are `[D]` or `[1, D]` — see below |
| `.transpose() -> Tensor` | view | reverse all axes (matrix transpose for rank-2) |
| `.permute(axes: Array) -> Tensor` | view | general axis reorder; `axes[i]` names which of self's axes becomes result axis `i` |
| `.slice(start, end) -> Tensor` | view | take axis 0 in `[start, end)` |
| `.narrow(params: Array) -> Tensor` | view | `[axis, start, end]`; `.slice()` generalized to any axis |
| `.reshape(dims: Array) -> Tensor` | view | contiguous input only; new shape |
| `.unfold(params: Array) -> Tensor` | view | `[axis, win, step]`; sliding window along `axis`, adds a trailing size-`win` axis |
| `.pad(params: Array) -> Tensor` | new buffer | `[axis, before, after]`; zero-pad along `axis` |
| `.fold(params: Array) -> Tensor` | new buffer | `[axis, orig_size, step]`; `.unfold()`'s inverse — scatter-add every window overlap back onto `axis` |
| `.sum() -> Float` | scalar | sum of all elements (forces eval) |
| `.sum(axis: Long?, keepdims: Bool = false) -> Tensor` | lazy | reduce one axis; a nil axis reads as no axis, i.e. the scalar form. `keepdims: true` keeps the axis as size 1, so the result broadcasts back over the input: `x - x.mean(1, keepdims: true)` |
| `.mean() / .mean(axis, keepdims:)` | Float / Tensor | likewise |
| `.max() / .max(axis, keepdims:)` | Float / Tensor | likewise |
| `.argmax(axis: Long, keepdims: Bool = false) -> Tensor` | lazy | reduce one axis to indices stored as Float |
| `.index_select(indices: Tensor) -> Tensor` | lazy | row gather along axis 0: `out[i] = self[indices[i]]` — the embedding-table lookup |
| `.softmax_cross_entropy(targets: Tensor) -> Tensor` | lazy | fused softmax + cross-entropy over the rows of `[N, C]` logits; one loss per row (`[N]`) |
| `.to_array() -> Array` | eager | convert to a Culebra Array (forces eval) |
| `.item() -> Float` | eager | extract the lone element as a Float; throws unless the tensor holds exactly one element (any rank) |
`.item()` is the scalar exit point, complementing `.to_array()` (which is for
shaped data): use it to read a loss or any single-element result without
reshaping. `loss.item()` replaces `to_float(loss.to_array()[0])`.
#### `.softmax_cross_entropy(targets: Tensor) -> Tensor`
Softmax over the classes and cross-entropy against `targets`, fused into one
op. `self` is `[N, C]` logits; `targets` holds one class id per row (`[N]`,
Float-valued like `.index_select()`'s indices, whose element values are
likewise not range-checked). The result is one loss per row, so the caller
picks the reduction:
```culebra
# doctest: skip
let rows = logits.softmax_cross_entropy(targets)
let loss = rows.mean(0) # the plain batch mean
let sft = (rows * mask).sum(0) * (1.0 / kept) # scoring only some rows
```
Composing it by hand gives the same value by a longer route. Composed, the
forward builds a softmax, a one-hot and a clamped log over `[N, C]` before
summing each row to one loss, and `.backward()` walks all of that back.
Fused, the forward reduces each row to `logsumexp(self) - self[target]`
with nothing of `[N, C]` in between, and the VJP is the closed form
`softmax(self) - onehot` in one pass. On an RTX 3090 at `[512, 1000]`:
forward 47 µs against 91 µs, backward 59 µs against 203 µs.
The clamp the composed form needs is inside the op: a class whose
probability underflows to zero contributes `-log(1e-15)` rather than an
infinite loss. Like dezero's own fused `SoftmaxCrossEntropy`, the backward
takes the closed form and does not differentiate that clamp.
#### `.causal_attention(k, v) -> Tensor`
Causal self-attention in one op: for each head and each row `t`,
`softmax(self[t]·k[0..t]ᵀ / sqrt(D))·v[0..t]` — a row attends itself and
the rows before it, never the ones after. `self`, `k` and `v` share one
shape, `[H, T, D]` (heads, rows, head width) or `[T, D]` for a single
head. The scale is fixed at `1/sqrt(D)`; for any other factor, scale
`self` first — the product is bilinear, so that is the same op.
```culebra
# doctest: skip
let q = x.dot(Wq).reshape([T, H, D]).permute([1, 0, 2]) # [H, T, D]
let k = x.dot(Wk).reshape([T, H, D]).permute([1, 0, 2])
let v = x.dot(Wv).reshape([T, H, D]).permute([1, 0, 2])
let ctx = q.causal_attention(k, v) # [H, T, D]
```
Spelled out, the same product is a `.dot()`, a mask added to the `[T, T]`
scores, a `.softmax()` and another `.dot()`. Fused, on CUDA with `D` of
64 or 128 the forward is a single kernel that streams `k` and `v` past
each block of query rows and never writes the `[T, T]` scores out;
elsewhere it is a reference loop with the same result. The backward
rebuilds the probabilities from the unfused ops and applies the standard
attention pullback, so a fused forward trains like the composed one.
#### `.layer_norm(gamma, beta) -> Tensor`
Layer normalization over the last axis in one op: each row becomes
`(x - mean) / sqrt(var + 1e-5) · gamma + beta`, with `mean` and `var` (the
variance, divided by `D`) taken over that row. `gamma` and `beta` hold one
weight per element of the last axis, shaped `[D]` or `[1, D]`; `self` may
have any rank from 1 up. The epsilon is fixed at `1e-5`.
```culebra
# doctest: skip
let gamma = Tensor.ones(1, D)
let beta = Tensor.zeros(1, D)
let h = x.layer_norm(gamma, beta) # x: [T, D] or [B, T, D]
```
Spelled out, the same result is `.mean(last, keepdims: true)`, a
subtraction, a square, a second mean, `+ 1e-5`, `.pow(-0.5)`, two products
and a sum: nine ops, each writing a new buffer the size of `self`. Fused, it
is one pass per row on the CPU and one kernel on the GPU, and only the
result is written. The backward applies the closed-form pullback: on CUDA
three kernels recompute each row's statistics and write the three
gradients directly, elsewhere the normalized rows are rebuilt from the
unfused ops, and either way the gradients match the composed form's.
### Autograd (reverse-mode)
The Tensor primitive carries a native reverse-mode autodiff engine: the
forward graph doubles as the tape, and `.backward()` walks it in C++. No
script-level wrapper is needed — the same op that computes a value also
knows its vector-Jacobian product. The tape is recorded only for ops that
feed a `requires_grad` leaf; forward-only work (inference, or a training
loop that writes its own backward) records none, so it costs no more than
the underlying tensor library.
| Method | Returns | Description |
|---|---|---|
| `.requires_grad() -> Tensor` | self | mark a leaf to accumulate grad; chainable |
| `.backward() -> Nil` | — | seed `dL/dself = 1` and propagate to every leaf |
| `.grad() -> Tensor` | Tensor | the accumulated gradient (zeros until `backward`) |
| `.zero_grad() -> Nil` | — | clear the gradient before the next step |
| `.detach() -> Tensor` | Tensor | a materialized copy with no graph and no grad |
`requires_grad` propagates forward: any op with a grad-tracking input
produces a grad-tracking output. Differentiable ops include `+ - * /`,
`.pow()` (w.r.t. the base), `.dot()`, axis `.sum()` / `.mean()`,
`.relu()`, `.sigmoid()`, `.softmax()`, `.log()`, `.tanh()`, `.sin()`,
`.cos()`, `.clamp()`, `.transpose()`, `.permute()`, `.reshape()`,
`.slice()`, `.narrow()`, `.pad()`, `.unfold()`, `.fold()`, `.rope()`,
`.causal_attention()`, `.layer_norm()`, `.softmax_cross_entropy()`,
`Tensor.concat()`, `Tensor.where()`, `Tensor.scatter_to_axis()`, and
`.index_select()` / `Tensor.index_add()` (each other's own VJP). Gradients
un-broadcast automatically, so a bias added across a batch sums back to its
shape. `.permute()`'s own VJP is the inverse permutation, which is what
lets an attention head split (`[B, C, H, D]` to `[B, H, C, D]`) train:
`.transpose()` reverses *every* axis, so from rank 3 up it is a different
operation, not a shorthand for this one. The permutation rides in the node's
own scalar slot, four bits an axis, so `.backward()` through a `.permute()`
of rank 16 or more raises rather than storing one it has truncated.
`.rope()` needs nothing stored either: rotation by an angle has its own
transpose for an inverse, and negating the second half of the head
dimension before and after turns the forward rotation into that transpose
— so the backward re-runs the forward rather than keeping the cosines it
used. The im2col trio differentiates as two dual pairs: `.pad()`'s VJP is the
crop `.narrow()` performs and `.narrow()`'s is that pad, `.unfold()`'s is
`.fold()` and `.fold()`'s is `.unfold()`. Each stores only its axis and its
step; the window and the length to restore come off the shapes either side.
So a convolution written as pad -> unfold -> reshape -> `.dot()` trains
under `.backward()` with no hand-written backward pass anywhere — see
`examples/tensor/mnist_conv.cul`. Where windows overlap, `.fold()` sums the
copies, which is what makes it the transpose rather than the inverse.
`.gt()` / `.lt()` / `.ge()` / `.le()` / `.eq()` / `.ne()` (and `.max()` /
`.argmax()`) are never differentiable — `.backward()` through a comparison
always raises, the same as PyTorch/numpy: compose the 0/1 mask with `*`
instead (a ReLU-style backward gate is the concrete pattern). Writing a
Tanh/Sin/Cos/Clip backward by hand, composed from other native ops rather
than relying on this, stays valid too — both paths work, whichever a
caller's own autograd style needs.
```culebra
let w = Tensor.from([[2.0, 0.0], [0.0, 3.0]]).requires_grad()
let x = Tensor.from([[1.0], [1.0]]).requires_grad()
let y = w.dot(x) # [2, 1]
let loss = (y * y).sum(0).sum(0)
loss.backward()
Tensor.eval(w.grad(), x.grad())
let gw = w.grad().to_array() # dL/dw
```
A typical training step zeroes grads, runs the forward, calls
`.backward()`, reads `.grad()` for the optimizer update, then `.detach()`
on the new weights to start the next step from a clean leaf. The
`benchmarks/microgpt/microgpt_tensor.cul` transformer is a full worked
example (embeddings, attention with a KV cache, RMSNorm, MLP,
cross-entropy, Adam) built entirely on these methods.
`.backward()` reads gradients from the forward buffers, so it implies a
`Tensor.eval` of the loss. `.grad()` returns a Tensor like any other —
materialize it with `Tensor.eval` before `.to_array()`.
Gradients are kept at the leaves only. An intermediate's gradient is
released as soon as it has been passed on to that node's inputs, so its
`.grad()` reads zeros after `.backward()`, and a second `.backward()`
through the same graph adds to the leaves exactly what that pass
contributes.
`Tensor.no_grad(fn) -> Any` runs `fn` with grad tracking suppressed:
ops inside build no autograd graph (so no tape and no `requires_grad`
flow), and the call returns whatever `fn` returns. Use it for inference
or any forward you will not backprop through.
```culebra
# doctest: skip
let logits = Tensor.no_grad(fn () {
model_forward(x)
})
```
### Operator overloading
`+ - * /` are broadcasting elementwise (numpy / silarray rules).
Scalars mix automatically:
```culebra
let M = Tensor.ones(3, 4)
let v = Tensor.ones(4) # broadcasts to [3, 4]
let r = Tensor.ones(3, 1) # broadcasts to [3, 4]
Tensor.eval(M + v, M + r, M + 1.0, M * 2.0)
```
The `@` operator is not implemented (use `.dot()`).
### Compound assignment (`+=` `-=` `*=` `/=` `**=`) and in-place writes
Compound assignment writes back into the LHS Tensor's buffer (no
fresh Tensor is allocated) when **the LHS owns its buffer and the
shape matches the RHS broadcast result**. Views, unevaluated graph
nodes, and shape mismatches automatically fall back to the ordinary
path (a fresh Tensor).
```culebra
# doctest: skip
mut W = Tensor.randn('f32', 1024, 256)
let alias = W
W -= grad * lr # writes directly into W's buffer
Tensor.eval(alias) # alias.to_array() sees the updated values
```
The SGD-style update `W = W - grad * lr` allocates a W-sized buffer
every step. `-=` removes that allocation; the gap widens for larger
weights and longer loops (measured: 5000 steps on a 1024×256 f32
weight, plain `=` 5.5 s → `-=` 3.6 s).
Supported ops: `+= -= *= /= **=` (`%=` and `@=` are not — `%` has no
defined Tensor semantics, and `@` changes the output shape so
in-place is unsafe).
### dtype / shape constraints
- dtype is F32 only (float64 has no fast path on GPU backends; scalar
entry/exit points like `.item()` / `.to_array()` surface `Float`).
- `.dot()` supports batched matmul (rank >= 3): leading dims must match
between the two operands, and the matmul itself runs over each pair's
trailing two axes (GPU-dispatched per batch slice).
- `.reshape()` and `.softmax()` both accept non-contiguous input (a
`.transpose()` or `.permute()` result, say) — no manual materialize
step needed. `.reshape()` clones to a contiguous buffer first when the
source isn't already one; `.softmax()` reads strided input directly.
### Backends and device selection
Evaluation is delegated to the vendored `cpp-tensorlib` engine
which owns the lazy graph, kernel fusion,
and the device backends:
- **CPU** — vectorized kernels (AVX2 / NEON); Accelerate supplies the
BLAS-shaped kernels on macOS.
- **GPU** — Metal on macOS, CUDA on Linux / Windows.
The device is process-global (shared by VM, JIT, and AOT) and
switchable at runtime:
| Function | Effect |
| --- | --- |
| `Tensor.use_cpu() -> Nil` | evaluate every op on the CPU |
| `Tensor.use_gpu() -> Nil` | evaluate on the GPU backend |
| `Tensor.use_auto() -> Nil` | choose per op by problem size (the default) |
| `Tensor.gpu_available() -> Bool` | whether a GPU backend is compiled in and reachable |
| `Tensor.device() -> String` | the selection in effect: `'cpu'`, `'gpu'` or `'auto'` |
```culebra
inspect(type_of(Tensor.gpu_available())) # => 'Bool'
inspect(Tensor.device()) # => 'auto'
```
`device()` reports the selection, not where a given op ran — under
`'auto'` that is decided per op, and an op sent to a GPU that turned
out to be unreachable still lands on the CPU.
`use_auto` is the default because small tensors lose to kernel-launch
overhead: it keeps those on the CPU and sends only the ops big enough
to pay for the trip. Where that line falls for a matrix product is
measured on the machine itself, at the first evaluation under `auto`
(a few square products timed on both devices, a few milliseconds
once), so the split follows the CPU and GPU actually present rather
than a table from another box. Calling `use_cpu()` / `use_gpu()`
anywhere — including before the first tensor exists — pins every later
op to that device instead.
`use_gpu()` on a build with no reachable GPU falls back to the CPU
path rather than throwing, so a program stays portable; check
`gpu_available()` when the choice matters. Storage stays F32 on every
device (see the dtype constraints above).
Metal needs nothing at build time. CUDA is compiled in when `nvcc` is
found (`CULEBRA_TENSOR_CUDA=AUTO`, the default; `ON` turns a missing
`nvcc` into a configure error and `OFF` skips the backend). The CUDA
driver itself is loaded at run time, so a CUDA-enabled binary still
runs on a machine with no GPU — `gpu_available()` just reports
`false`.
### Profiling
| Function | Effect |
| --- | --- |
| `Tensor.profile(fn) -> Array` | run `fn` with the engine's profiler on; the rows it recorded |
| `Tensor.profile_scope(label, fn) -> Any` | run `fn` under a scope named `label`; `fn`'s result |
`Tensor.profile` returns what the profiler recorded while `fn` ran;
`fn`'s own result is dropped. During the session the engine opens a
scope named after each op it runs (`dot`, `sum_to`, `layer_norm`,
`adam_step`, …), and `.backward()` opens `backward` and one scope per
VJP named after the Culebra op (`Dot`, `LayerNorm`). Nested scopes
join into a path with `/`. Every kernel launch, host-device copy and
blocking wait a backend performs is attributed to the innermost open
scope.
`Tensor.profile_scope` adds a scope of your own — a training phase, a
layer — around `fn` and returns whatever `fn` returns. Outside a
session it just calls `fn`.
Each row is an Object:
| Field | Meaning |
| --- | --- |
| `kind` | `'scope'` for a scope's own row; `'launch'`, `'transfer'` or `'wait'` for work under it |
| `path` | the scope path (`'step/backward/Dot/dot'`) |
| `kernel` | a launch's kernel name; `'h2d'` / `'d2h'` for a transfer; `''` otherwise |
| `count` | scope entries, or launches / transfers / waits |
| `host_ms` | scope: inclusive wall time; transfer and wait: how long the host was blocked |
| `device_ms` | launch: summed kernel time on the device, where the backend can time it |
| `device_timed` | how many of the row's launches `device_ms` covers |
| `bytes` | transfer: bytes moved |
Rows come grouped by path, the paths in descending order of their
scope's inclusive time, each scope's own row first and its launches by
device time. `device_ms` is per launch on CUDA. Metal times a command
buffer, not a launch, so its launch rows have `device_timed` of 0 and
the batch time is not in the rows; WebGPU counts launches only. A row
with `device_timed` below `count` was not timed, which is not the same
as fast.
```culebra
let w = Tensor.from([[1.0, 2.0], [3.0, 4.0]]).requires_grad()
let rows = Tensor.profile(fn () {
Tensor.profile_scope('step', fn () {
w.dot(w).sum(0).sum(0).backward()
})
})
let paths = rows.map(|r| r.path)
inspect(paths.filter(|p| p.starts_with('step/backward/Dot')).size() > 0) # => true
```
Setting `TL_PROFILE=1` in the environment profiles a whole program
from its first evaluation and prints the same rows as a table to
stderr at exit — no code change needed.
---
## 9. `JSON`
Round-trip between Culebra values and JSON text. Both backends ship
the same surface.
### `JSON.stringify(v, indent=0, sort_keys=false, lines=false) -> String`
Serialize `v` to a JSON string.
* `indent > 0` pretty-prints with that many spaces per nesting level
(newline after each comma, `": "` separator). `indent <= 0` is the
compact form. An `indent` whose result is too large to allocate raises
`ValueError` (`JSON.stringify() result is too large`).
* `sort_keys=true` walks `Object` keys alphabetically instead of
insertion order — useful for deterministic diff / hash output.
* `lines=true` emits **JSON Lines**: each element of an `Array` /
`Tuple` / `Set` becomes its own compact line, terminated by `\n`.
An empty collection yields the empty string. Incompatible with
`indent > 0` and with non-list receivers (`TypeError` in either
case).
For backwards compatibility, a positional 2nd argument is still
accepted as `indent`: `JSON.stringify(v, 2)` is `JSON.stringify(v,
indent: 2)`.
Supported value types:
| Culebra | JSON |
|--------------------|---------------------------------|
| `Nil` | `null` |
| `Bool` | `true` / `false` |
| `Long`, `Float` | number (non-finite Float raises `ValueError`) |
| `String` | quoted string with `\n`, `\t`, `\r`, `\"`, `\\`, `\u00xx` escapes |
| `Array` | JSON array |
| `Tuple` | JSON array (same shape as `Array`) |
| `Set` | JSON array, members in insertion order |
| `Object` (String keys only) | JSON object, keys in insertion order |
`Function`, `Tensor`, and Objects carrying non-String keys are not
serializable — `stringify` throws `TypeError` for these.
### `JSON.parse(s, lines=false, number_mode='auto', jsonc=false) -> Any`
Parse a JSON string into a Culebra value.
* `number_mode='auto'` (default): integers (no decimal point or
exponent) read as `Long`; everything else as `Float`. An integer past
`Long`'s range reads as a `Float` too (JSON bounds no number); one past
`Float`'s range is a `ValueError`.
* `number_mode='float'` reads every number as `Float` — useful for
round-trip safety when the producer treats numbers uniformly.
* `lines=true` parses **JSON Lines**: split `s` on `\n`, parse each
non-empty line, return an `Array` of the per-line values.
* `jsonc=true` parses **JSONC**: tolerate `//` line comments, `/* */`
block comments, and trailing commas in objects and arrays — for
reading existing config files (`tsconfig.json`, VSCode `settings.json`)
without stripping them first. The default is strict JSON, which rejects
comments and trailing commas with a `ValueError`.
Malformed input raises `ValueError` and the structured Error Object
exposes the JSON-internal position via `e.line` / `e.col` (both
1-based, pointing at the offending character):
```culebra
let r = try {
JSON.parse('{"a": ,}')
nil
} catch e {
e
}
inspect(r.message) # => 'JSON.parse: expected value'
inspect("{r.line}:{r.col}") # => '1:7'
```
`\uXXXX` escapes decode to UTF-8. A character outside the BMP is written
the way JSON requires it, as a UTF-16 surrogate pair, and the two halves
combine into the one character; half a pair on its own names nothing and is
a `ValueError`. `stringify` writes control bytes as `\u00xx`, so its output
always reads back.
```culebra
# Go's encoding/json escapes `<`, `>` and `&` this way, so replies from a Go
# service arrive carrying them.
inspect(JSON.parse('"\u003cb\u003e"')) # => ''
inspect(JSON.parse('"\uD83D\uDE00"')) # => '😀'
```
Nesting is bounded: containers more than 1000 levels deep raise a
`ValueError` (`nesting too deep (limit 1000)`) instead of exhausting the
C stack. `stringify` applies the same bound to the value tree it walks.
Examples:
```culebra
let v = {name: 'alice', age: 30, tags: ['admin', 'staff']}
# The default is compact; `sort_keys` orders the keys alphabetically.
inspect(JSON.stringify(v)) # => '{"name":"alice","age":30,"tags":["admin","staff"]}'
inspect(JSON.stringify(v, sort_keys: true)) # => '{"age":30,"name":"alice","tags":["admin","staff"]}'
let back = JSON.parse(JSON.stringify(v))
inspect(back.name) # => 'alice'
let arr = JSON.parse("1\n2\n3\n", lines: true)
inspect(arr) # => [1, 2, 3]
let cfg = JSON.parse(
'{
// comments and trailing commas are allowed
"port": 8080,
}',
jsonc: true,
)
inspect(cfg.port) # => 8080
```
`indent` pretty-prints and `lines` emits JSON Lines; both produce
multi-line output:
```culebra
let v = {name: 'alice', age: 30, tags: ['admin', 'staff']}
inspect(JSON.stringify(v, indent: 2))
inspect(JSON.stringify([1, 2, 3], lines: true))
# => |
# '{
# "name": "alice",
# "age": 30,
# "tags": [
# "admin",
# "staff"
# ]
# }'
# '1
# 2
# 3
# '
```
JIT note: built-in `JSON.{stringify, parse}` route through the same
canonical call resolver every other namespace method uses, so every
call shape behaves identically to the VM: positional binding
(`JSON.stringify(v, 2)` sets `indent`, `JSON.parse(s, true)` sets
`lines`), keyword arguments, and both literal `**{...}` and dynamic
`**variable` splats. Used as a first-class value the binding is the
same (`let f = JSON.stringify; f(v, indent: 2)`).
---
## 10. `Args`
Declarative CLI argument parser. The spec is a culebra `Object`
listing positionals, options, and subcommands; `Args.parse` returns
an `Object` whose fields match the spec, prints help on `--help`,
and exits with status 2 on parse errors. For programmatic control,
`Args.try_parse` raises an `ArgParseError` instead, or throws an
`{kind: "ArgParseHelp", help}` value for `--help`.
### `Args.parse(argv: Array, spec: Object) -> Object`
Parse `argv` (typically `Sys.argv`) against `spec`. On `--help` or
`-h`, prints help to stdout and `Sys.exit(0)`. On any parse error,
prints `error: ` to stderr and `Sys.exit(2)`.
### `Args.try_parse(argv, spec) -> Object`
Same engine, but raises on error / help instead of exiting. Use
this from tests or when wrapping into a custom UX.
### `Args.help(spec: Object) -> String`
Produce the help text the parser would print on `--help`, without
parsing or exiting. Useful for embedding into a wider message.
### Spec format
Each argument is an `Object` with these fields:
| Field | Type | Default | Meaning |
|---|---|---|---|
| `name` | `String` | (required) | field name in the parsed result |
| `type` | `String` | `"String"` | `"String"` / `"Long"` / `"Float"` / `"Bool"` |
| `short` | `String` | (none) | short flag letter (e.g. `"v"` → `-v`). Marks it as an option. |
| `default` | `Any` | (none) | default value. Marks it as optional. |
| `doc` | `String` | `""` | help-text description |
| `repeated` | `Bool` | `false` | collect multiple values into an `Array` |
| `positional` | `Bool` | (inferred) | force positional (`true`) or option (`false`) |
Argument **type** of `Bool` means the option is a **flag** that
consumes no value (`--verbose` / `-v`). All other types consume the
next token (`--count 5` / `--count=5`).
An argument is **positional** unless it has `short` or `default`, and
**required** unless it has a `default`, is `repeated`, or is a `Bool`
flag. `positional` states which one it is where those two rules would
disagree: an optional positional needs a `default` to say what it is
worth when absent, which on its own would have made it an option.
```culebra
{name: "file", positional: true, default: "-"} # cat []
{name: "file", positional: true, default: nil} # the same, nil when absent
{name: "out", positional: false} # a required --out
```
Positionals are matched in spec order, but one that may be omitted
only takes a token when more are left than the required positionals
after it still need: `[{name: "files", repeated: true}, {name: "dest"}]`
parses `a b d` as `files == ["a", "b"]` and `dest == "d"`.
### Example
```culebra
# doctest: skip
let spec = {
name: "wc-lite",
doc: "count lines and words",
args: [
{name: "input", type: "String", doc: "input file"},
{
name: "lines",
short: "l",
type: "Bool",
default: false,
doc: "count lines",
},
{
name: "words",
short: "w",
type: "Bool",
default: false,
doc: "count words",
},
{name: "encoding", type: "String", default: "utf-8"},
],
}
let args = Args.parse(Sys.argv, spec)
inspect(args.input) # String
if args.lines {
inspect("lines: ...")
}
if args.words {
inspect("words: ...")
}
inspect("encoding: {args.encoding}")
```
```
$ ./wc-lite -l file.txt
lines: ...
encoding: utf-8
$ ./wc-lite --help
wc-lite - count lines and words
Usage: wc-lite [options] ` where each element is a
sub-spec (same shape as the top-level spec, minus its own
`subcommands` typically). When present, the first positional token
selects a subcommand; the parsed result has a `subcommand` field
naming the selected command, and the rest of the spec's args are
parsed as that subcommand's:
```culebra
# doctest: skip
let spec = {
name: "git-lite",
subcommands: [
{name: "add", args: [{name: "files", type: "String", repeated: true}]},
{name: "commit", args: [{name: "message", short: "m", type: "String"}]},
],
}
match Args.parse(Sys.argv, spec).subcommand {
"add" => stage_files(args.files),
"commit" => commit_with_message(args.message),
}
```
### Error handling
`Args.parse` exits on any error. `Args.try_parse` instead raises:
```culebra
let r = try {
Args.try_parse(["--bogus"], spec)
} catch e {
e
}
# r.kind == "ArgParseError", r.message == "unknown option '--bogus'"
```
The `kind` of what it raises is one of:
| `kind` | Meaning | Extra fields |
|---|---|---|
| `ArgParseError` | parse failure (unknown opt, type mismatch, missing required, etc.): a runtime error | `message`, `line`, `col` |
| `ArgParseHelp` | user passed `--help` / `-h`: a thrown Object, not an error | `help` (the help-text string) |
---
## 11. `Proc`
Run an external command synchronously (blocking) and capture its
output. The command is an `Array` — `cmd[0]` is the executable
(PATH-resolved) and the rest are arguments. There is no shell, so no
quoting or injection concerns: `["git", "commit", "-m", msg]` passes
`msg` verbatim however it's spelled.
Every command spawned by `Proc` starts its own process group (POSIX) or
job object (Windows), so every kill this namespace does — `timeout:`,
`Proc.all`'s `fail_fast:`, `Proc.race`'s losers, `h.kill()`, and the GC's
`SIGKILL` of an unreaped `Proc.spawn` handle — reaches the whole tree it
spawned, not just the direct child. A descendant that starts its own group
or job (a daemonizing server, for instance) opts itself back out and has to
be reached another way.
### `Proc.run(cmd: Array, cwd=nil, env=nil, stdin="", check=false, timeout=0, share=nil, inherit_env=true) -> Object`
Runs `cmd` to completion and returns a result Object:
| field | type | meaning |
|---|---|---|
| `code` | `Long` | exit status on normal exit; `-1` when killed by a signal |
| `stdout` | `String` | everything the command wrote to stdout (captured whole) |
| `stderr` | `String` | everything it wrote to stderr |
| `ok` | `Bool` | `true` iff `code == 0` and `signal == nil` |
| `signal` | `String?` | signal name (`"SIGTERM"`, `"SIGKILL"`, …) if killed, else `nil` |
| `error` | `String?` | spawn-failure message; `nil` whenever the command actually ran. Only `Proc.all` sets this (its allSettled errors) — for `Proc.run` it is always `nil`, since a spawn failure throws instead. |
| `timed_out` | `Bool` | `true` if the command was killed for exceeding its `timeout`; `false` otherwise. |
Keyword arguments:
- `cwd: String` — working directory for the child (default: inherit the parent's).
- `env: Object` — environment variables to set in the child (default: none
beyond what it inherits). Values must be `String`.
- `inherit_env: Bool` — whether the parent's own environment is the base
`env` lands on (default: `true`, so `PATH` and friends survive). With
`false` the child starts from nothing and gets exactly what `env` gives
it — the way to keep a secret that lives in the parent's environment out
of a child that has no business reading it:
```culebra
# doctest: skip
Proc.run(cmd, inherit_env: false,
env: {PATH: Sys.env("PATH"), HOME: "/tmp"})
```
The two are independent: `env` alone adds, `inherit_env: false` alone
empties, and together they build one exact environment. Buffers passed
with `share` travel as environment variables of their own and reach the
child either way.
- `stdin: String` — bytes written to the child's standard input, which
is then closed (default: empty).
- `check: Bool` — when `true`, a non-zero exit, signal death, or timeout
throws `ProcessError` instead of returning a `{ok: false}` result (default: `false`).
- `timeout: Long` — milliseconds; if the command runs longer it is killed
(`SIGTERM`, then `SIGKILL` after a short grace) and the result has
`ok: false` and `timed_out: true` (default: `0` = no limit). The whole
process tree is killed — the command runs in its own process group
(POSIX) or job object (Windows), and any descendant that hasn't broken
out of it (by starting its own group/job) dies too. A process that closes
stdout/stderr but keeps running may not be reached by the timeout.
A **non-zero exit** or **signal death** is a normal result, not an error —
branch on `ok` / `code` / `signal`. Only a **spawn failure** (e.g. the
executable doesn't exist) or a failure under `check: true` throws
`ProcessError`. Passing a non-Array, a non-String element, or an empty
command throws `TypeError` / `ValueError`.
```culebra
# doctest: skip
let r = Proc.run(["git", "rev-parse", "--abbrev-ref", "HEAD"])
if r.ok {
IO.inspect("on branch " + r.stdout.trim())
} else {
IO.print(r.stderr)
}
# Feed stdin and read the transformed output.
let up = Proc.run(["tr", "a-z", "A-Z"], stdin: "hello\n")
assert_eq(up.stdout, "HELLO\n")
# Run in a directory with an extra env var; throw on failure.
Proc.run(
["make", "install"],
cwd: "/src/app",
env: {PREFIX: "/usr/local"},
check: true,
)
```
Output is buffered in full, so a command that emits gigabytes will use
that much memory. stdout and stderr are drained concurrently, so a
command that fills both will not deadlock.
`share: {name: buf}` hands one or more `SharedBuffer.shared(...)` buffers to the
child process (which re-attaches each with `SharedBuffer.receive(name, Class)`).
The child must be a culebra process — typically `[Sys.executable, "worker.cul"]`.
See [SharedBuffer › Sharing across processes](#sharing-across-processes-zero-copy).
### `Proc.all(commands: Array>, limit: Long = , timeout: Long = 0, fail_fast: Bool = false, retries: Long = 0, share: Object? = nil, inherit_env: Bool = true) -> Array`
Runs many commands in parallel and returns their result Objects in input
order. Each command is its own `Array` (same form as `Proc.run`'s
first argument). At most `limit` run at once — `limit` defaults to the
online CPU count; pass a smaller number to throttle, a larger one to widen.
`timeout` (ms, `0` = none) applies per command, measured from each command's
own start, and flags the result with `timed_out: true` when it fires.
By default this is **allSettled**: one command failing never aborts the
others. A command that ran and exited non-zero is `{ok: false, code: N,
error: nil}`; a command that *couldn't start* (e.g. the executable doesn't
exist) is `{ok: false, error: ""}` — neither throws. An empty list
returns `[]`.
With **`fail_fast: true`** the call instead stops at the first failure
(non-zero exit, signal, timeout, or spawn failure), `SIGKILL`s the still-
running commands, and throws `ProcessError` naming the offending command —
the `Promise.all` shape (as opposed to the default `Promise.allSettled`). If
every command succeeds it returns the result array as usual.
**`retries`** re-runs a command that failed, up to that many extra times; the
reported result is the final attempt's. A retry slots back into the `limit`
pool as it becomes free. When combined with `fail_fast`, a command only counts
as failed once its retries are exhausted.
**`share: {name: buf}`** hands the same `SharedBuffer.shared(...)` buffers to
*every* child (each re-attaches with `SharedBuffer.receive`), exactly as
`Proc.run`'s `share:` does — so a pool of workers can write a shared result
buffer (use `buf.with_lock` for contended cells). `inherit_env: false` gives
every child an environment of its own, as it does for `Proc.run`.
```culebra
# doctest: skip — spawns programs this checkout does not carry
let results = Proc.all(
[["git", "fetch", "origin"], ["npm", "test"], ["cargo", "build"]],
limit: 2,
)
for r in results {
if !r.ok {
IO.print(r.error ?? r.stderr)
}
}
```
### `Proc.race(commands: Array>, share: Object? = nil, inherit_env: Bool = true) -> Object`
Starts every command, returns the result Object of the **first to finish**,
and sends `SIGKILL` to the rest (then reaps them). Useful for racing
redundant providers or "first mirror to respond wins". An empty list throws
`ValueError`. `share: {name: buf}` hands shared buffers to the children, and
`inherit_env: false` withholds the parent's environment from them, both as in
`Proc.run` / `Proc.all`.
```culebra
# doctest: skip — spawns programs this checkout does not carry
let fastest = Proc.race([
["curl", "-s", "https://mirror-a.example/file"],
["curl", "-s", "https://mirror-b.example/file"],
])
IO.print(fastest.stdout)
```
### `Proc.spawn(cmd: Array, cwd=nil, env=nil, stdin="", share=nil, inherit_env=true) -> handle`
Starts a command and returns immediately with a **live handle**, without
waiting for it. The handle has three methods:
| method | returns | meaning |
|---|---|---|
| `h.wait()` | result Object | block until the child exits, draining its output |
| `h.poll()` | result Object or `nil` | the result if it has exited, else `nil` (non-blocking) |
| `h.kill(sig = 15)` | `nil` | send a signal (`SIGTERM` by default); the next `wait`/`poll` reaps |
`wait()` / `poll()` are idempotent — once the child has been reaped, both
return the same cached result Object (the usual `{code, stdout, stderr, ok,
signal, error, timed_out}`). A spawn failure throws `ProcessError`, like
`Proc.run`, whose `cwd` / `env` / `stdin` / `share` / `inherit_env` mean the
same here. A handle that is dropped without ever being waited on is reaped by
the GC (the child is `SIGKILL`ed), so it won't linger as a zombie — but
explicitly `wait()`ing or `kill()`ing is clearer. As with `Proc.run`'s
`timeout:`, `kill()` (and the GC's own `SIGKILL`) reaches the whole process
tree, not just the direct child — see the note at the top of this section.
```culebra
# doctest: skip — spawns external programs
let server = Proc.spawn(["python", "-m", "http.server", "8000"])
# ... do work against the server ...
server.kill() # SIGTERM
let r = server.wait()
IO.inspect("server exited via " + (r.signal ?? to_string(r.code)))
# Run something and poll for completion without blocking.
let job = Proc.spawn(["make", "-j4"])
while job.poll() == nil {
IO.print(".") # ...other work...
}
```
`stdin` is fed once at spawn time and then closed. Incremental streaming I/O
and pipelines (`a | b`) are planned.
---
## 12. `Isolate`
Run a closure on its own OS thread with its own garbage-collected heap, for
real CPU parallelism. Isolates share no mutable memory: a value crosses the
boundary by being **copied**, so two isolates can never race on the same
object. This is the thread-level counterpart to [§11 `Proc`](#11-proc) (which
parallelizes across processes).
> `Isolate.spawn`, `Channel`, and `Parallel` all work under both the VM
> and `--jit` (a closure crosses as a shared code reference — the bytecode in
> the VM, the compiled `fn_ptr` in the JIT — plus copied captures, and runs
> on the child's own heap).
### `Isolate.spawn(fn, *args) -> handle`
Runs `fn` on another thread and returns immediately with a live handle. Any
positional `args` are passed to `fn`.
```culebra
# doctest: skip
let h = Isolate.spawn(|| 1 + 2)
h.join() # => 3
let h2 = Isolate.spawn(|n| n * n, 7)
h2.join() # => 49
```
The handle has:
| method | returns | meaning |
|---|---|---|
| `h.join()` | the closure's return value | block until the isolate finishes, then hand back its (copied) result |
| `h.poll()` | the result or `nil` | the result if finished, else `nil` (non-blocking) |
| `h.drop()` | `nil` | cancel the isolate if it is still running, and join it; `join()` and `poll()` then return `nil` |
If the closure throws, the exception is re-raised on the thread that calls
`join()`, with its original kind preserved. A handle that goes without `join()`
cancels and joins its isolate as `drop()` does, so no thread is left dangling.
The handle holds its isolate: it prints as `[Isolate]` (`[dropped Isolate]`
after `drop()`), has no fields, and is not Sendable.
### Sendable: what can cross the boundary
The closure and its arguments — and the value it returns — must be **Sendable**.
A violation throws `SendError` at the `spawn` site (never a silent copy):
| Sendable | Not Sendable |
|---|---|
| numbers, `String`, `Bool`, `nil` | a native handle (`Proc` / `File` / isolate handle) |
| `Array` / `Object` / `Set` / `Tuple` of Sendable values | a `Tensor` (share via a buffer instead — planned) |
| `enum` / data-class instances | a closure that captures a `mut` variable |
| a closure capturing only Sendable values | a value that refers to itself (a cycle) |
| a free function (`fn name(...)`), captured by reference | a generator (send what it yields, or call the generator function on the other side) |
What arrives is a copy that reads as the original does: an Object keeps
its entries in the order it held them, whatever their keys.
Because captures are copied, a closure that mutates a captured collection
mutates **its own copy** — the parent's value is untouched:
```culebra
# doctest: skip
let xs = [1, 2, 3]
let h = Isolate.spawn(fn () {
xs.push(99)
xs.size()
})
h.join() # => 4 (the isolate's copy)
xs # => [1, 2, 3] (parent unchanged)
```
A `mut` capture is rejected rather than silently snapshotted — pass the value
as an argument instead:
```culebra
# doctest: skip
mut total = 0
Isolate.spawn(|| total) # SendError: captures the mutable variable 'total'
Isolate.spawn(|t| t, total) # ok — passed by value
```
A class instance crosses with its class, so its methods work on the other
side — and so do the default methods of the traits it conforms to (`neq` from
`Eq`, `lt` from `Comparable`, a user trait's defaults): a worker starts with
the traits as they were declared when it was spawned. What a default captures
is copied at the spawn, as the sent closure's captures are, and rebuilt in the
worker only when it calls that default. A class value itself
(`Isolate.spawn(|| Sq(1))` naming a top-level `Sq`) is not Sendable; pass an
instance in, or declare the class inside the closure.
### Parallelism cap
Live isolates are capped (default: the machine's core count; override with the
`CULEBRA_ISOLATE_LIMIT` environment variable). A spawn beyond the cap runs
**synchronously on the current thread** instead of starting a new one — so
recursive parallelism can't explode into thousands of threads. `join()` still
returns the same result; only the timing differs.
```culebra
# doctest: skip
# Parallel map: split work across isolates, then collect.
let parts = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
mut handles = []
for p in parts {
handles.push(Isolate.spawn(|| p.reduce(0, |a, b| a + b)))
}
mut total = 0
for h in handles {
total += h.join()
}
total # => 45
```
### Cancellation
An isolate is cooperatively cancellable. Dropping its handle without `join()`
(or letting the GC collect it) signals the isolate to stop; it unwinds at its
next statement or blocking-channel boundary, so a runaway or idle isolate never
hangs the program.
### Channels — `Channel.new(cap = 1) -> (tx, rx)`
A channel is a bounded, blocking queue for passing values between isolates. It
returns a **(tx, rx)** pair: `tx.send(v)` enqueues, `rx`
yields values. A channel endpoint is the one exception to the Sendable rules —
it is shared (by reference), so a closure can capture `tx`/`rx` and carry it
into a spawned isolate. Values still cross by copy; only the channel itself is
shared.
```culebra
# doctest: skip
let (tx, rx) = Channel.new(10) # bounded; capacity 10
let prod = Isolate.spawn(fn () {
for line in source() {
tx.send(parse(line))
}
tx.drop() # release this sender
})
tx.drop() # release the parent's sender too
for record in rx {
process(record)
} # ends when every tx is dropped
prod.join()
```
| method | on | meaning |
|---|---|---|
| `tx.send(v)` | tx | enqueue `v` (blocks while the buffer is full); throws `ChannelError` if the channel is closed |
| `tx.clone()` / `rx.clone()` | both | another endpoint over the same channel (multi-producer / multi-consumer) |
| `tx.drop()` / `rx.drop()` | both | release this endpoint |
| `rx.recv()` | rx | block for one value; returns `nil` once the channel is closed and drained |
| `for v in rx { ... }` | rx | drain until closed (the clean end-of-stream form) |
| `rx.try_recv()` | rx | take one value without blocking; answers `Value(v)`, `Empty`, or `Closed` |
| `rx.drain(max = nil)` | rx | take what is queued right now, at most `max` of it, without blocking (an Array, possibly empty) |
**Auto-close is the deadlock-safety net.** The active senders are counted;
when the **last `tx` is dropped** (a producer isolate finishing, normally or by
throwing) the channel closes and any `for v in rx` ends. So a producer that
crashes won't hang its consumer — surface the failure by `join()`-ing the
producer. Note the multi-producer trap: every `tx` (including the parent's
original) must be dropped for the channel to close — drop the ones you don't
keep.
**Cloned endpoints compete; they don't mirror.** Each value goes to exactly
one endpoint — whichever `recv` reaches it first — so cloning `rx` spreads a
stream across consumers rather than copying it to each. Clone `rx` when one
consumer can't keep up: several long-running consumers on a shared queue keep
a producer from stalling on a full buffer, and absorb bursts that a single
consumer would serialize. For a copy per consumer, send to a channel per
consumer instead.
**An endpoint is a handle onto its channel.** It prints as `[Channel tx]` or
`[Channel rx]` and has no fields: `keys()` is `[]`, and setting one is a
`TypeError`. Two endpoints are `==` when they are the same end of the same
channel, so a clone equals its original. Only `Channel.new` and `clone` make
one, along with the `tx` that `Channel.fan_in(items, fn)` hands each producer;
an Object that looks like an endpoint stays an Object, here and after a trip
through a channel. A dropped endpoint is a closed
one, even while its clones keep the channel open: `send` on it throws
`ChannelError`, `recv` returns `nil`, and it prints as `[dropped Channel tx]`.
**`Channel.new(0)` is a rendezvous channel** (capacity 0): `send` does not return
until a receiver takes the value — a synchronous hand-off with no buffering.
Useful for backpressure (a producer can't run ahead of its consumer). Within a
single isolate it deadlocks (the send has no one to hand to), so use it across
isolates. Capacity must be `>= 0`.
#### Receiving without blocking — `rx.try_recv` and `rx.drain`
`recv()` and `for v in rx` block, which is right for a consumer whose only job
is the channel. A caller that must keep making progress on its own schedule —
a `Canvas.run` frame callback, a TUI key loop, any timed loop — cannot afford
to stop there: a blocked frame is a dropped frame. `try_recv()` always returns
immediately.
It answers with one of three variants, the shape of an `enum`:
```culebra
enum ChannelResult { Value(Any), Empty, Closed }
```
`Value(v)` carries a value that was queued, `Empty` means nothing is queued
*right now* (the channel is still open, so ask again later), and `Closed` means
nothing will ever arrive again (every `tx` is gone and the queue is drained).
```culebra
# doctest: skip
match rx.try_recv() {
Value(ev) => world.apply(ev),
Empty() => nil, # nothing this frame — carry on
Closed() => running = false,
}
```
Note the `()` on `Empty()` and `Closed()`. A bare `Empty` in a match arm is an
identifier pattern, which matches *anything* and binds it — the arms below it
would never run. `x: Empty` works too, if you prefer to name what you matched.
Three variants rather than a `nil` return, because `nil` would have to mean
three things at once here: `recv()` already returns `nil` both for a genuine
`nil` payload and for end-of-stream, and a non-blocking read adds "nothing yet"
on top. A loop written `while (v = rx.try_recv()) != nil` would then stop early
the first time a real `nil` came through. The variant makes each case
answerable.
`drain()` takes the whole queue at once, which is usually what a frame wants:
```culebra
# doctest: skip
for ev in rx.drain() { world.apply(ev) }
```
It returns an Array — empty when nothing is queued, and empty on a closed
channel too, so pair it with `try_recv()` when you need to notice the close.
What it hands back is what the queue held at the instant it was called, taken
in one step. That matters more than it sounds: popping one value at a time
frees a slot, which wakes a blocked sender, which refills before the next pop
looks — so a `try_recv()` loop over a capacity-4 channel with a busy producer
has been measured handing back several hundred values. `drain()` cannot run
away like that, and a `try_recv()` loop you write yourself can.
Pass `max` to bound the batch further, when the queue itself may be long:
```culebra
# doctest: skip
for ev in rx.drain(16) { world.apply(ev) } # at most 16 this frame
```
The rest stays queued for the next call. `max` must be a `Long` (a count) or
`nil` for no limit; a negative value raises `ValueError`.
Taken in one step, a batch also goes as one. A value can fail to rebuild on
the receiving side (a `Set` member whose `hash()` throws there, its fields
changed since it was added): `recv()`, `try_recv()` and `for v in rx` throw
that error with the value taken, and `drain()` throws it with the rest of its
batch taken too.
Neither method is available on a `Channel.fan_in` receiver yet; both are only
on a plain `rx`.
#### `Channel.fan_in(items: [rx]) -> rx`
Merge several receivers into one. The returned `rx` yields values from whichever
source is ready (it waits on all of them at once, like Go's `select` or
core.async's `merge` — event-driven, not polling), and ends once **every** source
is closed. This lets each producer own its **own** channel instead of sharing one
via `tx.clone()`, which sidesteps the multi-producer trap (a forgotten clone-drop
hanging the consumer) — each channel then has exactly one `tx`, dropped 1:1.
```culebra
# doctest: skip
mut handles = []
mut sources = []
for w in workers {
let (tx, rx) = Channel.new()
handles.push(Isolate.spawn(fn () {
produce(w, tx)
})) # producer's tx auto-drops on exit
tx.drop() # the parent's own tx (1:1, obvious)
sources.push(rx)
}
for v in Channel.fan_in(sources) {
consume(v)
} # one stream, all producers
for h in handles {
h.join()
} # keep handles alive; collect errors
```
The merge **takes over** the given receivers — read them only through the merged
`rx`, not directly (reading an original races the merge). Order across sources is
not preserved (it is a merge). An empty list yields an immediately-closed `rx`; a
single source is a pass-through.
#### `Channel.fan_in(items, fn) -> rx`
The all-in-one form: spawn one producer isolate per item, running `fn(item, tx)`,
and merge their streams. `fn` sends to its own `tx`; fan_in creates the channels,
drops the parent's senders, and owns the producer handles — so the consumer
writes **no `tx`, no `drop`, and no handle** at all. `fn` and every item must be
Sendable.
```culebra
# doctest: skip
let merged = Channel.fan_in(workers, fn (w, tx) {
for x in produce(w) {
tx.send(x)
}
})
for v in merged {
consume(v)
}
merged.join() # join the producers; re-raises the first one that errored
```
`merged.join()` (after the stream ends) joins the spawned producers and re-raises
the first error; without it, producer errors are dropped (as with not joining an
`Isolate.spawn` handle). Producers run on their own threads.
A merged `rx` of either form is a handle onto its merge, as an endpoint is onto
its channel: it prints as `[merged Channel rx]`, has no fields, and is not
Sendable. `drop()` lets the sources go, and cancels any producer still running.
A `for` loop already reading the merged `rx` keeps reading after a `drop()`,
until every source ends.
### Parallel — `Parallel.map` / `each` / `map_settled` / `race`
The high-level form: apply a function to every element of an array across a pool
of isolates, without managing handles yourself. `fn` and every element must be
Sendable (the same rules as `Isolate.spawn`).
```culebra
# doctest: skip
Parallel.map([1, 2, 3, 4], |x| x * x) # => [1, 4, 9, 16] (input order)
Parallel.map(urls, |u| fetch(u), limit: 8) # at most 8 live isolates
Parallel.each(jobs, |j| process(j)) # side effects; returns nil
Parallel.map_settled(urls, |u| fetch(u)) # => [{ok, value, error}, ...]
Parallel.race(mirrors, |m| download(m)) # first success wins
```
| call | returns | notes |
|---|---|---|
| `Parallel.map(items, fn, limit = )` | `Array` | one result per element, **in input order**; fail-fast |
| `Parallel.each(items, fn, limit = )` | `nil` | for side effects; no results collected; fail-fast |
| `Parallel.map_settled(items, fn, limit = )` | `Array` | one `Result` per element, in input order; **never fail-fast** |
| `Parallel.race(items, fn, limit = )` | `Any` | the **first** element to succeed; cancels the rest |
`limit` caps how many isolates run at once (default: the core count); the
elements are pulled from one shared queue, so it is `limit` isolates total, not
one per element.
**`map` / `each` are fail-fast:** the first element to throw stops the rest and
is re-raised as `ParallelError` naming the element index — e.g. `Parallel.map:
element[2] failed: ...`. What follows `failed:` is the error's message, or
what a thrown value says: its `message` when it is an Object whose `message`
is a non-empty String, otherwise the value as an uncaught `throw` prints it
(a `__str__` is not consulted). The `error` of `map_settled` and the failure
of `race` carry the same text.
**`map_settled` never fail-fasts:** every element yields a `Result` Object
`{ok: Bool, value: Any, error: String?}` (the same shape as `Proc.all`), so one
failure doesn't lose the others' work — `r.ok ? r.value : r.error`.
**`race` returns the first success** and cancels the remaining elements; if
*every* element throws it raises `ParallelError`, and on an empty array it raises
`ParallelError` (there is no result to return).
**`on_progress:` reports completion.** Every method also accepts an
`on_progress: |done, total|` callback. Unlike `fn`, it is **not** Sendable: it
runs on the calling thread, so it can freely read and mutate captured state
(e.g. a counter or a progress bar). It is called as elements finish, with the
running completion count and the total; a throwing callback cancels the run.
```culebra
# doctest: skip
Parallel.map(
urls,
|u| fetch(u),
on_progress: |done, total| IO.print("\r" + done.to_string() + "/" + total.to_string()),
)
```
(`map_reduce` is planned.)
### Signal — `Signal.notify` / `Signal.reset`
By default Ctrl+C raises a cooperative `Interrupted` (see the language guide's
*Interruption* section). For a long-running service you usually want the
opposite: catch the signal at one place and shut down on your own terms.
`Signal.notify(tx)` switches Ctrl+C from *throw* to *deliver* — each press sends
`"SIGINT"` to the channel's `tx` endpoint instead of interrupting the running
code, so the program blocks on `rx.recv()` (or `for sig in rx`) and drives its
own graceful shutdown. This is Go's `signal.Notify` model.
```culebra
# doctest: skip
let (tx, rx) = Channel.new(1)
Signal.notify(tx) # Ctrl+C now goes to the channel, not a throw
serve_in_background()
rx.recv() # blocks until the first Ctrl+C
inspect("shutting down…")
drain_and_close()
```
| call | notes |
|---|---|
| `Signal.notify(tx)` | route Ctrl+C to this channel's `tx` (a non-blocking send — if the buffer is full the extra signal is dropped, as Go does); the throw is suppressed while active. Use a buffered channel (`Channel.new(1)`). |
| `Signal.reset()` | restore the default `Interrupted`-throw behavior and release the channel `notify` was holding. |
`notify` retains its own sender on the channel, so it stays open even after you
`tx.drop()` your copy — `reset()` releases it. There is no force-kill escalation
while notify is active: the program opted to handle signals itself (call
`reset()` if you want a later Ctrl+C to abort). The delivery runs from a
background poller, so a press is observed within a few tens of milliseconds.
### SharedBuffer — zero-copy shared fixed-layout data
`SharedBuffer` holds a flat array of fixed-layout records that several
isolates read and write **without copying** — the one place the isolate
model shares mutable memory, kept safe by restricting the records to fixed
scalar fields (no references, no pointers).
A record type is an ordinary class marked `@packable`, whose fields carry a
type annotation and an optional default:
```culebra
@packable
class FloatPair {
x: Float32 = 0.0
y: Float32 = 0.0
}
```
`@packable` fixes the byte layout (C-ABI natural alignment). Every field
must be a fixed scalar — `Float32`, `Float64`/`Float`, `Int8`, `Int16`,
`Int32`, `Int64`/`Long`, `Byte`, `UInt16`, `UInt32`, or `Bool`. A non-scalar field is a
`SyntaxError` at load time. A field with no default takes the type's zero
value (`0`, `0.0`, `false`).
#### `SharedBuffer.new(count, type: Class) -> buffer`
Allocates `count` zero-initialized records laid out per `Class`. `buffer.size`
(also `.count` / `.len`) reports the element count. The bytes live in this
process's heap — shareable across isolates (threads), not across processes.
A `count` whose records cannot be addressed or allocated is a `ValueError`
(`count is too large`), for `SharedBuffer.file` and `.shared` as well.
#### `SharedBuffer.file(path, count, type: Class) -> buffer`
Same buffer, backed by a memory-mapped file (POSIX `mmap(MAP_SHARED)`, Windows
`CreateFileMapping`). Writes go to the file's pages — **persistent** (the file
outlives the process) and visible to any other process that maps the same
`path`. The handle gains a `flush()` method (flush the dirty pages to disk); the
file is an ordinary file you remove with `FS.remove(path)`. Pointing `path` at a
RAM-backed location (e.g. `/dev/shm/...` on Linux) gives shared memory without
disk durability.
```culebra
# doctest: skip — runs a worker pool over a large buffer
@packable
class Cell {
v: Int64 = 0
}
let buf = SharedBuffer.file("/tmp/grid.bin", 100, Cell)
buf[0].v = 42
buf.flush() # durable on disk
```
#### `SharedBuffer.shared(count, type: Class) -> buffer`
Same buffer, backed by **anonymous** shared memory (a name-less fd — `memfd` on
Linux, an immediately-unlinked POSIX shm object on macOS; a pagefile-backed
`CreateFileMapping` on Windows). It touches no disk; the kernel frees it once
every handle is dropped. Its purpose is to be handed to a **child process** via
`Proc.run` / `Proc.spawn` `share:`
(see [Sharing across processes](#sharing-across-processes-zero-copy) below).
#### `buffer[i] -> view`
Indexing yields a **view** over element `i` (negative indices count from the
end). Reading or writing `view.field` goes straight to the backing bytes — no
per-record object is materialized:
```culebra
@packable
class FloatPair {
x: Float32 = 0.0
y: Float32 = 0.0
}
let buf = SharedBuffer.new(3, FloatPair)
inspect(buf.size) # => 3
buf[0].x = 1.5 # writes the bytes in place
let v = buf[0] # a stored view aliases the same element
v.y = 2.5
inspect([buf[0].x, buf[0].y]) # => [1.5, 2.5]
```
Whole-element assignment (`buf[i] = ...`) is a `TypeError` — a record has no
standalone value form, so set its fields individually. An unknown field is an
`AttributeError`; an out-of-range index is an `IndexError`.
The buffer itself is a handle onto its bytes. It prints as `[SharedBuffer of
3]` and has no fields of its own: `keys()` is `[]`, and setting one is a
`TypeError`. Two handles are `==` when they are onto the same bytes. A view
holds the buffer it was read from and reads through it, so `buf.drop()` makes
every view of `buf` a `ValueError` (`SharedBuffer has been dropped`), while a
handle another isolate holds keeps the bytes for that isolate. A view sent to
another isolate takes a handle along with it.
A view is a handle too, and reads as its bytes decoded at that moment when it
is printed, spread (`{...v}`, `[...v]`), or written by `JSON.stringify` /
`TOML.stringify`: a record as an Object of its fields, printed under its
class's name, a `FixedArray` or `BoundedArray` field as an Array, a
`BoundedSet` as a Set, and a `BoundedMap` as an Object keyed by its keys (so
`JSON.stringify` refuses a record holding one, as it refuses any Object with
non-String keys). A view has no fields of its own: a record's names are its
fields (`v["x"]` reads and writes `v.x`), and setting a name on any other view
is a `TypeError`. Two views are `==` when they read the same bytes as the same
thing. A view of a dropped buffer prints as `[dropped SharedBuffer view]`.
```culebra
@packable
class Cell {
v: Int64 = 0
near: BoundedArray
}
let cells = SharedBuffer.new(2, Cell)
cells[0].v = 42
cells[0].near.push(1)
inspect("{cells[0]}") # => 'Cell {v: 42, near: [1]}'
inspect(JSON.stringify(cells[0].near)) # => '[1]'
inspect([cells[0] == cells[0], cells[0] == cells[1]]) # => [true, false]
```
#### Sharing across isolates (zero copy)
A buffer crosses an isolate boundary **by reference** — the child reads and
writes the same bytes. Unlike every other value (copied at the boundary), a
SharedBuffer is shared, the same exception channels make. This makes the
classic data-parallel pattern — workers updating disjoint elements —
allocation-free:
```culebra
# doctest: skip — runs a worker pool over a large buffer
@packable
class Cell {
v: Int64 = 0
}
let cells = SharedBuffer.new(8, Cell)
Parallel.each([0, 1, 2, 3, 4, 5, 6, 7], fn (i) {
cells[i].v = i * i
})
# cells now holds 0, 1, 4, 9, 16, 25, 36, 49
```
Workers writing **disjoint** elements need no synchronization. Two isolates
writing the **same** element concurrently is a data race — partition the work
(as above) so each element has a single writer, or guard the shared update with
[`with_lock`](#synchronizing-with-bufferwith_lockfn).
#### Sharing across processes (zero copy)
A `SharedBuffer.shared(...)` buffer can be handed to a **child process**, not
just another isolate. The parent passes it to `Proc.run` (or `Proc.spawn`) under
the `share:` keyword — an Object of `name -> buffer` — and the child re-attaches
it by that name with `SharedBuffer.receive(name, Class)`. Both processes map the
same physical pages: writes are visible without copying or serializing.
```culebra
# doctest: skip — spawns programs this checkout does not carry
# --- parent.cul ---
@packable
class Cell {
v: Int64 = 0
}
let grid = SharedBuffer.shared(4, Cell)
grid[0].v = 100
Proc.run([Sys.executable, "worker.cul"], share: {grid: grid})
inspect(grid[0].v) # the child's write, read back here
grid.drop()
```
```culebra
# doctest: skip
# --- worker.cul ---
@packable
class Cell {
v: Int64 = 0
}
let grid = SharedBuffer.receive("grid", Cell)
for i in 0..grid.count {
grid[i].v = grid[i].v + (i + 1) * 10
}
grid.drop()
```
`receive` takes only the name and the `@packable` type — the element count comes
from the parent (so `grid.count` matches). The child declares the same
`@packable` class; `receive` checks the record sizes agree and raises a
`ValueError` on a layout mismatch, an unknown name, or a buffer that wasn't a
`SharedBuffer.shared(...)` (heap and file buffers don't cross this way — a file
buffer is shared by re-opening its `path`). `Sys.executable` is the path to the
running culebra binary, for launching a worker copy of culebra.
`Proc.run` blocks until the child exits, so its writes are complete on return.
For concurrent children, `Proc.spawn` each and `wait()` them; as with isolates,
let each child own **disjoint** elements so the writes never race.
#### Synchronizing with `buffer.with_lock(fn)`
Disjoint writes need no synchronization. When two writers genuinely must touch
the **same** data — a counter, a multi-field update that has to stay consistent
— `with_lock` is the escape hatch: it runs the callback while holding the
buffer's lock and returns the callback's value. The lock is released on every
exit, including a thrown exception.
```culebra
# doctest: skip — runs a worker pool over a large buffer
@packable
class Counter {
n: Int64 = 0
}
let tally = SharedBuffer.new(1, Counter)
Parallel.each(iota(0, 8), fn (w) {
for _ in 0..1000 {
tally.with_lock(fn () {
tally[0].n = tally[0].n + 1
})
}
})
inspect(tally[0].n) # 8000 exactly — no lost updates
```
The same call works across **processes**: a `.shared` or `.file` buffer carries
a process-shared lock, so children handed the buffer via `share:` (or another
process re-opening the same file `path`) mutually exclude each other. Keep the
callback short — it serializes every holder. The lock is **not reentrant**:
calling `with_lock` again on the same buffer from inside the callback
deadlocks. A non-function argument is a `TypeError`; a dropped buffer is a
`ValueError`.
A `.file` buffer reserves a small fixed header at the front of the file for this
lock, so its bytes are culebra's container format rather than a bare array of
your records — keep that in mind if an external tool reads the file directly.
#### Fixed-count fields: `FixedArray`
A `@packable` field may be a `FixedArray` — exactly `N` elements of a
scalar `T`, laid out inline as `[T × N]` with nothing in front of them: C's
`T[N]`. Its size never changes, so there is no `push`; each element is zero
until written. For the arrays a record format fixes the length of — a
shape's three dimensions, a color's three channels, a reserved block.
```culebra
@packable
class Voxel {
pos: FixedArray
rgb: FixedArray
}
let grid = SharedBuffer.new(2, Voxel)
let v = grid[0]
v.pos[0] = -4
v.rgb[2] = 255
inspect([v.pos.size(), v.pos[0], v.rgb[2], v.rgb[0]]) # => [3, -4, 255, 0]
```
The view supports `.size()` / `.capacity()` (both `N`) / `.get(i)` /
`.set(i, v)` / `arr[i]` (read & write) / `for x in arr`; an index outside
`0...N` (negative counts from the end) is an `IndexError`. As with
`BoundedArray`, assigning the whole field is a `TypeError` — write its
elements.
#### Variable-count fields: `BoundedArray`
A `@packable` field may be a `BoundedArray` — a fixed-**capacity** inline
collection (`N` elements of a scalar `T`) whose **count** varies at runtime. It
is laid out fully inline (`[len][T × N]`, no pointers), so variable-count data
still fits a shared record — the VARCHAR(N) pattern.
```culebra
# doctest: skip
@packable class Body {
mass: Float32 = 0.0
trail: BoundedArray # up to 8 points, starts empty
}
let bodies = SharedBuffer.new(100, Body)
let b = bodies[0]
b.trail.push(1.5)
b.trail.push(2.5)
b.trail.size() # => 2 (capacity() => 8)
b.trail[0] # => 1.5
b.trail[1] = 9.0
for p in b.trail { ... }
```
The view supports `.size()` / `.capacity()` / `.push(v)` / `.get(i)` /
`.set(i, v)` / `arr[i]` (read & write) / `for x in arr`. `push` past capacity
and an out-of-range index raise `IndexError`. The element type must be a fixed
scalar. Assigning the whole field (`record.field = ...`) is a `TypeError` —
mutate it through the view. The view reads/writes the record's bytes in place,
so it shares across isolates with the buffer.
#### Text fields: `BoundedString`
A `@packable` field may be a `BoundedString` — a fixed-**capacity** inline
UTF-8 string holding up to `N` bytes (`[len][byte × N]`, no pointers). Unlike
`BoundedArray`, it is read and written as a whole `String` value — the VARCHAR(N)
pattern:
```culebra
# doctest: skip
@packable
class Row {
id: Int32
name: BoundedString<16>
}
let rows = SharedBuffer.new(100, Row)
rows[0].name = "alice" # whole-value write (≤ 16 bytes)
rows[0].name # => "alice" (a real String)
rows[0].name.upper() # => "ALICE" (all String methods apply)
rows[1].name # => "" (zero value is empty)
```
`N` is a **byte** capacity. A string of more than `N` bytes raises
`CapacityError`; assigning a non-String raises `TypeError`. The read returns a
fresh `String` copy of the stored bytes, so it shares across isolates with the
buffer (a child isolate's write is visible to the parent's read).
#### Hash collections: `BoundedSet` and `BoundedMap`
A `@packable` field may also be a `BoundedSet` (up to `N` scalar values) or
a `BoundedMap` (up to `N` scalar key→value pairs). Both are
open-addressed hash tables laid out fully inline (`[count][states][entries]`,
no pointers), mutated in place through a view:
```culebra
# doctest: skip
@packable class Bag {
tags: BoundedSet
counts: BoundedMap
}
let b = SharedBuffer.new(100, Bag)
let s = b[0].tags
s.add(5); s.add(7); s.add(5) # the duplicate is a no-op
s.size() # => 2 (capacity() => 16)
s.contains(7) # => true
s.remove(7) # => true (false if absent)
for x in s { ... }
let m = b[0].counts
m.set(42, 1)
m.get(42) # => 1 (nil if absent)
m.contains(42) # => true
m.set(42, 2) # overwrite; size() unchanged
m.remove(42) # => true
m.keys() # => [Int32, ...]
for k, v in m { ... } # yields (key, value) tuples
```
`add` / `set` past capacity raise `CapacityError`. Key and value types must be
fixed scalars; equality is by the scalar's bytes (so `BoundedSet` treats
`0.0` and `-0.0` as distinct). Assigning the whole field is a `TypeError` —
mutate through the view's methods. The view itself holds nothing but the way to
the bytes, so writing onto it (`m.note = 1`, `m[42] = 1`) is a `TypeError`.
The bytes live in the record, so the collection shares across isolates with
the buffer.
#### Optional fields: `T?`
A `@packable` field may be an optional scalar `T?` — a value-or-`nil` slot laid
out as `[present:byte][T]`. It is read and written as a whole value: `nil` when
the present byte is clear, otherwise the scalar.
```culebra
# doctest: skip
@packable
class Node {
id: Int32
parent: Int32? # a sparse "no parent" slot
}
let n = SharedBuffer.new(100, Node)
n[0].parent # => nil (zero value)
n[0].parent = 5
n[0].parent # => 5
n[0].parent = nil # clear it
n[0].parent ?? -1 # => -1
```
`0` is a real value, distinct from `nil`. Only scalar optionals are packable
(`T` must be a fixed scalar). Sparse structures — an id→optional-slot array —
are the main use, alongside packable enums for tagged payloads.
#### Tagged unions: `@packable enum`
An `@packable` enum is a fixed tagged union `[tag:i32][payload]`: every
variant's scalar payload shares one region sized to the largest variant, and
the tag selects which variant is live. A `@packable` class field may then be
that enum type. Use it for component kinds, message types, and other
discriminated payloads in a shared record.
```culebra
# doctest: skip — runs a worker pool over a large buffer
@packable
enum Shape {
Circle(Float32),
Rect(Float32, Float32),
Point,
}
@packable
class Obj {
id: Int32
shape: Shape
}
let objs = SharedBuffer.new(100, Obj)
objs[0].shape = Shape.Rect(2.0, 3.0) # write a variant value
match objs[0].shape {
# read it back and match
Rect(w, h) => w * h,
Circle(r) => 3.14 * r * r,
Point => 0.0,
}
```
Every variant payload must be a fixed scalar (a non-scalar payload is a
`SyntaxError` at the variant). Writing a value that is not an instance of that
enum raises `TypeError`. Reading reconstructs the variant instance from the
bytes (no enum namespace needed), so a value written by one isolate is
readable by another through the shared buffer.
#### Raw bytes: `Bytes`
A `@packable` field may be a `Bytes` — exactly `N` raw bytes inline, with no
length prefix, read and written as a whole byte `String`. For hash digests,
UUIDs, and other fixed binary blobs.
```culebra
# doctest: skip
@packable
class Entry {
id: Int32
digest: Bytes<32> # e.g. a SHA-256
}
let e = SharedBuffer.new(100, Entry)
e[0].digest = some_32_byte_string
e[0].digest # => the 32 bytes (binary-safe)
```
The written `String` must be **exactly** `N` bytes (a `ValueError` otherwise);
a non-String raises `TypeError`. The bytes are binary-safe (embedded NULs are
preserved). Unlike `BoundedString` (a variable-length text field with a length
prefix), `Bytes` is a fixed-size blob.
#### Nested records: a `@packable` class field
A `@packable` class field may be another `@packable` class — its record is
stored inline, and `outer.inner` yields a view over those bytes, so a nested
field assignment writes through in place:
```culebra
# doctest: skip
@packable
class Point {
x: Float32
y: Float32
}
@packable
class Line {
id: Int32
start: Point
end: Point
}
let lines = SharedBuffer.new(100, Line)
lines[0].start.x = 1.0 # writes into the inline Point's bytes
lines[0].start.y = 2.0
lines[0].start.x # => 1.0
lines[0].end = lines[0].start # copy a whole sub-record (memcpy)
```
The nested class must be declared (and `@packable`) before the class that
nests it. Assigning a whole sub-record copies the bytes of another record of
the **same** class (otherwise `TypeError`); to set individual fields, go
through the view (`outer.inner.field = v`).
#### Records as bytes: `String.pack`
A `@packable` class also describes a record in a binary file or a network
message. Handed to [`String.pack` / `String.unpack`](language.md#binary-data-stringpack--stringunpack)
as the `type`, each element is one record: its fields in declaration order,
**no padding**, each scalar in the byte order the call's `endian:` names
(little unless told). This is a separate form from the SharedBuffer layout
above, which aligns fields the C way and keeps the machine's byte order —
that one is memory isolates share, this one is bytes a file carries — so a
field at an odd offset in a file format is declared where it sits.
| Field | Bytes | Written | Read back |
|---|---|---|---|
| `Int8` … `Int64`, `Byte`, `UInt16`, `UInt32` | its width | a `Long` in the type's range (`ValueError` otherwise) | `Long` |
| `Float32`, `Float64` | 4, 8 | a `Float` or `Long`; `Float32` rounds as `Math.f32` does | `Float` |
| `Bool` | 1 | `1` / `0` | nonzero is `true` |
| `Bytes` | `N` | a String of exactly `N` bytes (`ValueError` otherwise) | the `N` bytes |
| `BoundedString` | `N` | up to `N` bytes, NUL-padded — C's `char[N]` (`CapacityError` past `N`) | up to the first NUL |
| `FixedArray` | `N` × `T`'s | an `Array` of exactly `N` elements, each as `T` above | an `Array` of `N` |
A `Bytes`, `BoundedString` or `FixedArray` field left unset
(`nil`) is written as zeros. The other kinds — `BoundedArray`, `BoundedSet`,
`BoundedMap` (a count that varies has no place in a fixed-size record), an
optional, an enum, a nested record — have no byte-stream form, and naming a
class with one is a `TypeError`: read those parts with a scalar
`String.unpack`, or as a record of their own. `pack` takes instances of that class (another
value is a `TypeError`), and its errors name the record and the field.
`unpack` builds each record **without running `new`** — a `@packable`
record's state is its declared fields, and every one of them comes from the
bytes — so the result is an ordinary instance of the class, with its
methods and its field types checked on later writes.
`String.pack_size(Class)` is one record's size, for reading exactly one:
```culebra
@packable
class Lump {
offset: Int32
size: Int32
name: BoundedString<8>
}
let l = Lump()
l.offset = 12
l.size = 4096
l.name = "E1M1"
let bytes = String.pack(Lump, [l])
inspect(String.pack_size(Lump)) # => 16
inspect(bytes.slice(8, 16).bytes().collect()) # => [69, 49, 77, 49, 0, 0, 0, 0]
let back = String.unpack(Lump, bytes)[0]
inspect([back.offset, back.size, back.name]) # => [12, 4096, 'E1M1']
```
### Shared — immutable values shared by reference
`Shared.new(value)` freezes an ordinary value — any nesting of objects,
arrays, tuples, sets, and scalars — and hands back a **read-only view**
that every isolate can use without copying. It is the lane for
variable-length read-only data (a tokenizer dictionary, a parsed config,
a search index): the channel lane would copy it per task and
`SharedBuffer` requires a fixed layout. One frozen tree, any number of
readers:
```culebra
# doctest: skip
let dict = Shared.new(JSON.parse(FS.read("vocab.json")))
let workers = [0, 1, 2, 3].map(|i| Isolate.spawn(fn () {
dict["hello"] # all isolates read the SAME frozen tree
}))
```
Reads look like ordinary collection access — `view.field`, `view[key]`,
`view[i]`, `view.size()`, `view.has(k)`, `view.keys()` / `view.values()`,
and `for ... in view` (an Object view yields `(key, value)` pairs, an
Array/Tuple/Set view yields elements). A scalar field materializes into
the reader's heap; a container field comes back as another shared view
(still no copy). `view.copy()` deep-materializes back into ordinary
mutable values when you need a local scratch copy. `view[key]` and
`view.has(key)` find a key the way the Object that was frozen does: any
key it held, a Tuple or `nil` included, and never across types.
A view also answers three questions the way the collection it froze
does: `view.get(key, fallback)` on an Object and `view.get(i, fallback)`
on an Array, `view.contains(x)` on a Set (a member, found as a key is),
an Array or a Tuple (an element, by `==`), and `view.index_of(x)` on an
Array. An Array view has the Array's `view.map(f)` and `view.filter(f)`
too: each hands back an ordinary Array, and an element that is a
container reaches `f`, and stays in what `filter` keeps, as a view. For
the rest of what a sequence answers, take the view's iterator:
`view.iter()` yields the elements one at a time with no copy of the
whole, so `view.iter().map(f)`, `.sum()` and the other iterator methods
read the frozen tree directly.
A `for` over a view, and an iterator of one, hold the frozen tree
themselves: dropping the view part-way does not end the walk.
The view's own methods (`size`, `has`, `get`, `contains`, `index_of`,
`keys`, `values`, `copy`, `iter`, `map`, `filter`, `drop`) are what those
names read as `view.name` and in a pattern. A key of the frozen Object by
one of those names is still there under `view['name']`.
Printing a view, `JSON.stringify(view)`, `TOML.stringify(view)` and a
spread (`{...view}`, `[...view]`) read the value that was frozen, the one
`view.copy()` hands back, so a spread's result is an ordinary value of
the program's own all the way down. A keyword splat takes that spread
(`f(**{...view})`); `**view` itself is a `TypeError`. A pattern reads a
view's keys as `view.name` does (`let {port} = view`). `==` does not
look inside a view: compare `view.copy()`. A view that has been dropped
prints as `[dropped Shared value]`.
A freeze takes what a send to an isolate takes, so anything Sendable
freezes, with two extra rejections: **functions** (`Shared` values are
data only) and **native handles** (channels, buffers, other `Shared`
views) raise `SendError`. Every write surface raises
`ImmutableError`; updates are copy-on-write by construction (build a new
value, `Shared.new` it again, hand out the new view). `view.drop()`
releases the reference (idempotent); the tree itself is freed when the
last view anywhere drops, and reads after that raise `ClosedError`.
| | lane | sharing | copy per reader |
|---|---|---|---|
| fixed-layout records | `SharedBuffer` | read **+ write** | none |
| variable-length, read-only | `Shared` | read | none |
| anything, mutable | channel | copy | yes |
---
## 13. Matchers
Assertion matchers for tests and runtime invariant checks. All ten
matchers are global names bound on every environment (no `import`
required). Failure throws a Culebra Object with the conventional
`{kind: "AssertionError", message: ...}` shape; user code can
`try/catch` it.
An uncaught failure reports the position of the `assert_*` call:
```
AssertionError at 12:1: assert_eq failed:
left: foo
right: bar
```
Every matcher but `assert_throws` also takes an optional trailing
`label`, which the failure message repeats. It earns its keep where the
position cannot tell two failures apart — a helper that asserts on
behalf of its callers reports every one of them at its own single line:
```culebra
let check = fn (got, want, label) {
assert_eq(got, want, label)
}
check(1, 1, "first") # passes
# A failure here would say: assert_eq failed: second
```
There is no `assert` keyword or builtin — instead, use a specific
matcher for each kind of check. For production invariants, write the
`if`/`throw` directly:
```culebra
# doctest: skip
if !cond {
throw {kind: "AssertionError", message: "invariant violated"}
}
```
### Truthiness matchers
* **`assert_true(x: Bool, label: String? = nil) -> Nil`** — pass if `x` is truthy. Throws
`AssertionError` with message `assert_true failed:\n value: {x}`
on failure. `x` must be `Bool`, `Long`, or `Float`; other types
raise `TypeError`: there is no implicit truthiness, so empty strings
and arrays are not falsy.
* **`assert_false(x: Bool, label: String? = nil) -> Nil`** — mirror of `assert_true`.
### Comparison matchers
Each comparison matcher dispatches through the same operator as the
plain expression — `assert_eq(a, b)` is equivalent to `a == b`, so
`__eq__` / `__lt__` / `__le__` on class instances are honored.
Failure message names both operands via `to_string` (respecting any
user `__str__`).
* **`assert_eq(a, b, label: String? = nil) -> Nil`** — `a == b`.
* **`assert_ne(a, b, label: String? = nil) -> Nil`** — `a != b`.
* **`assert_lt(a, b, label: String? = nil) -> Nil`** — `a < b`.
* **`assert_le(a, b, label: String? = nil) -> Nil`** — `a <= b`.
* **`assert_gt(a, b, label: String? = nil) -> Nil`** — `a > b`.
* **`assert_ge(a, b, label: String? = nil) -> Nil`** — `a >= b`.
```culebra
assert_eq(1 + 1, 2) # passes silently
let r = try {
assert_eq("foo", "bar")
nil
} catch e {
e
}
inspect(r.kind) # => 'AssertionError'
inspect(r.message)
# => |
# 'assert_eq failed:
# left: foo
# right: bar'
```
### `assert_throws(kind: String, f: Function) -> Nil`
Invoke 0-arg `f()` and assert it throws an error whose `kind` matches
`kind`. Built-in errors (`ZeroDivisionError`, `TypeError`, etc.)
expose `e.kind` directly; user `throw {kind: "X", ...}` matches the
same way. `f` must take 0 parameters — otherwise `ArityError`.
```culebra
assert_throws("ZeroDivisionError", fn () {
let _ = 1 / 0
})
assert_throws("MyError", fn () {
throw {kind: "MyError", message: "boom"}
})
```
### `assert_close(a: Float, b: Float, tol: Float, label: String? = nil) -> Nil`
Pass if `|a - b| <= tol`. NaN in `a`, `b`, or `tol` deliberately
fails (a naive `diff > tol` check would silently pass NaN). Use this
in place of `assert_eq` for floating-point comparisons.
```culebra
assert_close(3.14, 3.1415, 0.01)
```
### Implementation note
The matcher family is defined in culebra source (not native C++) and
bound via the lazy module mechanism so that all three backends
(VM / JIT / AOT) share one implementation. Operator dispatch
(`==`, `<` etc.) inside the matchers is the same operator dispatch
each backend already implements, so `__eq__` / `__lt__` semantics
agree without any matcher-specific drift logic.
---
## 14. `Regex`
Linear-time, grapheme-aware regular expressions (engine: vendored
[cpp-regexlib](https://github.com/yhirose/cpp-regexlib)).
Patterns match by Unicode **extended grapheme cluster**, not code point — `.`
consumes one user-perceived character (so `/./` matches `🇯🇵` as a single
element). Matching runs in **linear time** (Thompson NFA / Pike VM with a lazy
DFA fast path), so catastrophic backtracking cannot occur and there are no
backreferences. Offsets are **byte offsets** (Go-style), always on grapheme
boundaries.
A `Regex` is **compiled once and reused** (`Regex.compile` — the compiled
program is the expensive part), then queried with methods:
**Write patterns as single-quoted raw strings** (`'\d+'`, not `"\\d+"`): single
quotes do no escape processing and no `{...}` interpolation, so `\d` and `{n}`
pass through verbatim. For a pattern that also
contains an apostrophe (e.g. a tokenizer's `'s`/`'t`), use a backtick raw
string `` `...` `` — also raw, but it may hold `'`, `"`, and `{`. Flags are
either passed to `compile` as a string (`Regex.compile('hello', "i")`) or inline
in the pattern: `(?i)` case-insensitive, `(?m)` multiline, `(?s)` dotall.
For a constant pattern, the [`re"..."` literal](language.md#regex-literals) is
shorthand for `Regex.compile(...)` — `re'\d+'` and `re"hello"i` are the same
compiled `Regex`, with the body always raw and flags trailing the closing
quote. A literal also supports `${expr}` interpolation (escaped for a String,
spliced for a `Regex` — see `Regex.interp` below). Use `Regex.compile(...)`
directly when the *whole* pattern is built at runtime.
The pattern and subject accept either string flavor (`String` or `StringView`),
so the `StringView` results of `String.split` / `.slice` compose directly:
`Regex.compile('\d+').find_all(line.slice(0, 80))`.
| Constructor / static | Result |
| --- | --- |
| `Regex.compile(pattern)` | `Regex` — compile (reused); bad pattern raises |
| `Regex.compile(pattern, flags)` | `Regex` — `flags` a string of `"i"` / `"m"` / `"s"` |
| `Regex.escape(s)` | `String` — backslash-quote every metacharacter so `s` matches literally |
| `Regex.interp(x)` | `String` — splice helper for `re"...${x}..."`: a `Regex` → `(?:src)`, anything else → escaped to match literally |
For a single use, the namespace methods below take the pattern directly and
hide the `compile` step. Reusing one
pattern across many inputs still wants `Regex.compile(pattern)`, but the engine
caches by pattern so the one-shot forms pay no recompile. Put flags inline
(`(?i)` / `(?m)` / `(?s)`).
| One-shot | Equivalent |
| --- | --- |
| `Regex.find(pattern, s)` | `Regex.compile(pattern).find(s)` — `Match` or `nil` |
| `Regex.match(pattern, s)` | anchored match at the start |
| `Regex.find_all(pattern, s)` | `[Match]` |
| `Regex.test(pattern, s)` | `Bool` |
| `Regex.split(pattern, s)` | `[String]` |
| `Regex.replace_all(pattern, s, repl)` | `String` — template or `fn (Match) -> String` repl, every match |
| `Regex.replace_first(pattern, s, repl)` | `String` — same `repl`, only the leftmost match |
```culebra
inspect(Regex.find('(\d+)', "ab12")[1]) # => '12'
inspect(Regex.test('(?i)hello', "HELLO")) # => true
inspect(Regex.replace_all('[;；]', "a;b；c", "、")) # => 'a、b、c'
# A miss is nil, so it composes with `?.` and `??`:
inspect(Regex.find('x', "y")?.value ?? "none") # => 'none'
```
| Method | Result |
| --- | --- |
| `re.test(s)` | `Bool` — does the pattern match anywhere in `s` |
| `re.find(s)` | `Match` or `nil` — leftmost match |
| `re.match(s)` | `Match` or `nil` — match anchored at the start |
| `re.find_all(s)` | `[Match]` — all non-overlapping matches |
| `re.find_all_str(s)` | `[String]` — just the matched texts (no `Match` objects; ~12× faster on match-dense input) |
| `re.find_all_index(s)` | `[Int]` — flat byte spans `[s0, e0, s1, e1, …]` (positions only, one allocation total) |
| `re.count(s)` | `Int` — number of non-overlapping matches (no objects allocated) |
| `re.find_iter(s)` | `Iterator` — lazy; supports early exit (`.take(n)`) |
| `re.replace_all(s, repl)` | `String` — `repl` is a template (`$1` / `$` / `$$`) **or** a `fn (Match) -> String`, every match |
| `re.replace_first(s, repl)` | `String` — same `repl` grammar, only the leftmost match; unchanged if there is no match |
| `re.split(s)` | `[String]` — split `s` on matches |
**Choosing a bulk API.** `find_all` builds a full `Match` object (text, spans,
`groups`, `named`) per match; on match-dense input that object construction —
not the matching — dominates, costing tens of times more than the engine's raw
scan. When you do not need per-match captures, reach for the lean variant:
`count` when you only need the number, `find_all_index` for byte spans,
`find_all_str` for the matched texts, or `find_iter` when you stop early.
Keep `find_all` for when you actually consume `groups` / `named` per match.
A `Match` is a data object (and `nil` means no match):
| Field | Meaning |
| --- | --- |
| `m.value` | the whole-match text (`String`) |
| `m.start`, `m.end` | byte offsets |
| `m.groups` | `[Group \| nil]`; `groups[0]` is the whole match |
| `m.named` | `{name: Group}` — named captures |
Subscripts are the captures accessor: `m[i]` returns positional group `i`'s
text (`m[0]` is the whole match, negative indices wrap like an array) and
`m["name"]` returns a named group's text. Any miss — index out of range, an
unmatched optional group, or an unknown name — is `nil`, so it composes with
`?? ""`. Subscripting reaches only captures, never the record fields, so the
whole match is `m.value` or `m[0]` (not `m["value"]`). Use the dot fields when
you need spans (`m.groups[i].start`).
`Group` has `.value`, `.start`, `.end`. An invalid pattern raises `RegexError`.
```culebra
let d = Regex.compile('\d+')
inspect(d.test("abc 123")) # => true
inspect(Regex.compile('\w+').find(" hello world").value) # => 'hello'
inspect(d.find("no digits")) # => nil
inspect(d.find_all("a1 b22 c333").size()) # => 3
```
Captures reach by position (`m[1]`) and by name (`m["year"]`); `m[0]` is
the whole match and a miss reads as `nil`. The `Group` objects under
`m.groups` / `m.named` carry spans as well as the text:
```culebra
let m = Regex.compile('(?\d{4})-(\d{2})').find("2026-05")
inspect(m[1]) # => '2026'
inspect(m["year"]) # => '2026'
inspect(m[0]) # => '2026-05'
inspect(m[9] ?? "none") # => 'none'
inspect(m.groups[1].value) # => '2026'
inspect(m.named["year"].value) # => '2026'
```
Replacing and splitting — a replacement is either a `$n` template or a
function of the `Match`. `replace_all` replaces every match; `replace_first`
replaces only the leftmost one and leaves the rest of the string untouched
(a no-op — returns `s` unchanged — if there is no match):
```culebra
let d = Regex.compile('\d+')
inspect(d.replace_all("a1 b22 c333", "#")) # => 'a# b# c#'
inspect(d.replace_first("a1 b22 c333", "#")) # => 'a# b22 c333'
inspect(Regex.compile('(\w+)@(\w+)').replace_all("x@y", '$2.$1')) # => 'y.x'
inspect(d.replace_all("a1 b22", fn (m) {
"<{m.value}>"
})) # => 'a<1> b<22>'
inspect(Regex.compile('\s+').split("the quick brown")) # => ['the', 'quick', 'brown']
inspect(Regex.compile('hello', "i").test("HELLO world")) # => true
```
`find_iter` is lazy, so a scan can stop early — `for m in d.find_iter(s) { break }`
walks no further than the match it broke on:
```culebra
let d = Regex.compile('\d+')
inspect(d.find_iter("1 2 3").take(2).collect().size()) # => 2
inspect(Regex.escape("a.b(c)")) # => 'a\.b\(c\)'
```
The supported syntax (literal / `.` / character classes / `* + ? {n,m}` greedy
and lazy / `|` / capturing and named groups / `\d \w \s \b` / lookahead /
variable-length lookbehind / `\p{…}` Unicode properties) and the full matching
model and resource limits are documented in the vendored engine,
[cpp-regexlib](https://github.com/yhirose/cpp-regexlib).
---
## 15. `Http`
Synchronous HTTP/HTTPS client (engine: vendored `cpp-httplib` + OpenSSL,
statically linked). Each call **blocks** until the response arrives — there is
no async/await. The same namespace also serves: `Http.server()` below carries
routes, static files, and WebSocket, and `Http.sse` / `Http.ws` speak the two
streaming protocols as a client. TLS is automatic for `https://` URLs; the system trust store is
used to verify server certificates (macOS keychain / the platform CA bundle on
Linux). `gzip` / `deflate` responses are decompressed transparently — `body`
is always the decoded content.
Every method returns a **response Object** and raises only on a *transport*
failure:
| field | type | meaning |
|---|---|---|
| `status` | `Long` | HTTP status code (`200`, `404`, …) |
| `ok` | `Bool` | `true` iff `status` is in `[200, 300)` |
| `reason` | `String` | status reason phrase (`"OK"`, `"Not Found"`, …) |
| `body` | `String` | response body (raw bytes) |
| `headers` | `Object` | response headers, keyed by name (String → String) |
| `json()` | `Any` | parse `body` as JSON (convenience for `JSON.parse(r.body)`) |
A **4xx/5xx response is a normal result** (`ok: false`), not an error — branch
on `status` / `ok`. A **transport failure** (DNS, connection refused, TLS
handshake, timeout) throws `HttpError`. A malformed URL (no scheme/host) also
throws `HttpError`; a bad `headers` value throws `TypeError`.
| Method | Result |
| --- | --- |
| `Http.get(url, headers=nil, timeout=0, follow_redirects=true)` | response Object |
| `Http.delete(url, headers=nil, timeout=0, follow_redirects=true)` | response Object |
| `Http.head(url, headers=nil, timeout=0, follow_redirects=true)` | response Object |
| `Http.post(url, body="", content_type="text/plain", headers=nil, timeout=0, follow_redirects=true)` | response Object |
| `Http.put(url, body="", content_type="text/plain", headers=nil, timeout=0, follow_redirects=true)` | response Object |
| `Http.request(method, url, body="", content_type="text/plain", headers=nil, timeout=0, follow_redirects=true)` | response Object — any method (PATCH, OPTIONS, …), sent as written. A method is an HTTP token (letters, digits, and ``!#$%&'*+-.^_`\|~``); anything else — a space, CR/LF, the empty string — is a `ValueError` naming it, before anything is sent |
| `Http.sse(url, on_event, headers=nil, timeout=0, follow_redirects=true)` | response Object — opens a `GET` and streams Server-Sent Events to `on_event`; see below |
| `Http.client(base_url, headers=nil, timeout=0, follow_redirects=true)` | a persistent client handle (base URL + default headers + connection reuse); see below |
| `Http.server()` | an HTTP server handle (register routes + `static` + `ws`, then `listen`); see below |
| `Http.ws(url)` | connect a WebSocket client; returns a handle (`send`/`receive`/`for`/`close`/`is_open`); see below |
Keyword arguments (shared by every method):
- `headers: Object` — request headers; an `Object` whose values are all
`String`. A non-`String` value is a `TypeError` (default: none).
- `params: Object` — query-string parameters; an `Object` of `String` values,
appended to the URL percent-encoded (`?k=v&…`, preserving any query already in
the URL). A non-`String` value is a `TypeError` (default: none).
- `timeout: Long` — per-phase timeout in **seconds** for connect / read / write;
`0` uses the library default (default: `0`).
- `follow_redirects: Bool` — chase `3xx` `Location` headers (default: `true`).
- `into: String | Function` — stream the response body to a sink instead of
buffering it; see Streaming below (default: `nil` = buffer into `body`).
- `json: Any` (`post` / `put` / `request` only) — serialize the value to JSON
and send it as the body with `Content-Type: application/json`.
- `form: Object` (`post` / `put` / `request` only) — an `Object` of `String`
values sent as an `application/x-www-form-urlencoded` body (percent-encoded).
- `files: Object` (`post` / `put` / `request` only) — a `multipart/form-data`
body (text fields and file parts, with streaming); see Multipart below. At most
one of `body` / `json` / `form` / `files` may be given (else `TypeError`).
- `body: String | Function` / `content_type: String` (`post` / `put` /
`request` only) — request body and its `Content-Type` (the header is set only
when the body is non-empty and no explicit `Content-Type` was passed in
`headers`). A `String` is sent whole; a `Function` is a **producer** streamed
chunked — see Streaming below.
```culebra
# doctest: skip
let r = Http.get("https://api.example.com/users", params: {page: "2"})
if r.ok {
let users = r.json() # parse the response body as JSON
IO.inspect(users.size().to_string())
} else {
IO.inspect("request failed: {r.status}")
}
# POST JSON with a header and a timeout (`json:` serializes + sets Content-Type).
let resp = Http.post(
"https://api.example.com/users",
json: {name: "alice"},
headers: {Authorization: "Bearer " + token},
timeout: 30,
)
assert_true(resp.ok)
# A transport failure throws; a 404 does not.
let missing = Http.get("https://api.example.com/nope")
assert_eq(missing.ok, false) # 404 is a normal result
assert_eq(missing.status, 404)
```
The `get`/`post`/etc. methods return the whole body as a single `String` (read
into memory); pair it with [`JSON.parse`](#9-json) for JSON APIs.
**Streaming (download) — the `into:` argument.** For a response too large to
hold in memory, pass `into:` to stream the body to a sink instead of buffering
it. It works on any method and leaves the returned `body` empty (the bytes go to
the sink). `into:` accepts:
* a **`String`** — a file path; the body is written straight to that file.
* a **`Function`** — a `|chunk|` closure called with each chunk as it arrives.
The callback runs on the calling thread, so it may read and mutate captured
state; if it throws, the transfer is aborted and the error propagates.
```culebra
# doctest: skip
Http.get("https://example.com/big.tar.gz", into: "big.tar.gz") # → file
mut bytes = 0
Http.get(
"https://example.com/big.csv",
into: fn (chunk) { bytes += chunk.size() },
)
# any method, e.g. a POST whose response streams back:
Http.post(
"https://example.com/query",
body: q,
into: fn (chunk) { handle(chunk) },
)
```
**Streaming (upload).** Symmetrically, pass a `Function` as `body:` (on
`post` / `put` / `request`) to stream the request body chunked, so a large
upload never lives in memory at once. The producer is called repeatedly and
returns the next chunk `String`, or `nil` to end the stream:
```culebra
# doctest: skip
let f = File.open("big.bin")
Http.post(
url,
body: fn () {
let chunk = f.read(65536)
!chunk.empty() ? chunk : nil # nil signals end-of-stream
},
content_type: "application/octet-stream",
)
```
The producer runs on the calling thread (it may mutate captured state); if it
throws, the upload is aborted and the error propagates. A non-`String`/non-`nil`
return is a `TypeError`.
**Multipart uploads — the `files:` argument.** Pass `files:` (on `post` / `put` /
`request`) to send a `multipart/form-data` body; the `Content-Type` (with a
generated boundary) is set for you. `files:` is an `Object` whose each entry is
one part, keyed by field name. A part value is:
* a **`String`** — a plain text field.
* an **`Object`** with exactly one body source — `content:` (a `String` held in
memory), `path:` (a `String` file path, streamed from disk), or `stream:` (a
producer `Function`, streamed) — plus optional `filename:` and
`content_type:`. A `path:` part defaults `filename` to the file's base name.
* an **`Array`** of any of the above — repeated parts under the same field name
(e.g. multiple files in one `photos` field).
A part that uses `path:` or `stream:` is streamed chunked, so a large file or a
slow-to-produce part never has to live in memory all at once. A `stream:`
producer follows the same contract as a streaming `body:` — return the next
chunk `String`, or `nil` to end the part.
```culebra
# doctest: skip
# text field + an in-memory file part
Http.post(
url,
files: {
title: "My report",
doc: { content: "a,b,c\n1,2,3\n", filename: "data.csv", content_type: "text/csv" },
},
)
# a large file streamed straight from disk (never buffered whole)
Http.post(
url,
files: { clip: { path: "/movies/big.mp4", content_type: "video/mp4" } },
)
# a slow-to-produce part streamed from a producer
mut row = 0
Http.post(
url,
files: {
export: { filename: "rows.csv", content_type: "text/csv", stream: fn () {
row += 1
row <= 1000 ? "{row},{compute(row)}\n" : nil
} },
},
)
# repeated parts under one field name via an Array
Http.post(
url,
files: {
caption: "trip",
photos: [ { path: "./1.jpg" }, { path: "./2.jpg" } ],
},
)
```
A part value that isn't a `String` / `Object` / `Array`, an `Object` without
exactly one of `content` / `path` / `stream`, a non-`String` `content` / `path`,
or a non-`Function` `stream` is a `TypeError`; a `path` that can't be opened is
an `IOError`.
### `Http.sse(url, on_event, headers=nil, timeout=0, follow_redirects=true) -> Object`
Open a [Server-Sent Events](https://developer.mozilla.org/docs/Web/API/Server-sent_events)
(`text/event-stream`) stream — a long-lived `GET` whose `on_event` callback is
invoked once per event as it arrives. The call blocks for the life of the
stream and returns the final response Object after the server closes it.
**It opens a `GET`, and only a `GET`**, as the protocol does — a browser's
`EventSource` cannot POST either. Streaming LLM and chat APIs speak this same
wire format but ask for it over a **`POST`** carrying the request as a JSON
body, which puts them out of reach here: stream those with the `into:` argument
above and decode the frames yourself (below).
Each event is an Object with three String fields:
| field | meaning |
|---------|---------|
| `event` | the `event:` type, or `"message"` when the server sends none |
| `data` | the `data:` payload; multiple `data:` lines are joined with `\n` |
| `id` | the last `id:` field seen, or `""` |
```culebra
# doctest: skip
Http.sse("https://api.example/v1/events", fn (e) {
if e.event == "progress" {
IO.println("{e.id}: {e.data}")
}
})
```
A `POST` that streams its answer back — which is what every LLM API is — takes
the other route. `into:` hands you the body a chunk at a time, and the frames
are yours to split: `field: value` lines, one blank line between events, and a
chunk boundary that can fall anywhere (mid-line, or mid-character).
```culebra
# doctest: skip
Http.post(
"https://api.example/v1/chat",
json: {model: model, messages: messages, stream: true},
into: fn (chunk) { decoder.feed(chunk) },
)
```
`Accept: text/event-stream` is sent automatically unless you set `Accept`
yourself. Comment lines (`: ...`) and the `retry:` field are ignored. The
callback runs on the calling thread (it may mutate captured state); returning
ends handling of that event, and throwing aborts the stream and propagates the
error. A transport failure is an `HttpError`.
### `Http.client(base_url, headers=nil, timeout=0, follow_redirects=true) -> Object`
A persistent client that **reuses one keep-alive connection** across requests and
carries a **base URL** and **default headers** — for hitting the same API many
times without repeating the host, auth header, and re-doing the TLS handshake
each call. Returns a handle with `get` / `post` / `put` / `delete` / `head` /
`request` (the same kwargs as the free `Http.*` functions, but the first
argument is a path joined onto `base_url`) plus `close`.
| argument | meaning |
|---|---|
| `base_url` | scheme + host (+ optional path prefix), e.g. `https://api.example.com/v1` |
| `headers` | default headers layered under every request's (per-request wins per key) |
| `timeout` / `follow_redirects` | connection-level defaults (not per-request) |
```culebra
# doctest: skip
let api = Http.client(
"https://api.example.com/v1",
headers: {Authorization: "Bearer " + token},
timeout: 30,
)
let me = api.get("/me").json() # → GET https://api.example.com/v1/me
let user = api.get("/users/42").json() # reuses the same connection
api.post("/users", json: {name: "alice"}) # Authorization header rides along
api.get("/items", headers: {"Idempotency-Key": k}) # merged over the defaults
api.close() # release the connection
```
A request method's first argument is a path (for `request`, the second, after
the method — which is held to the same rule as `Http.request`'s): a leading-slash or bare relative
path is joined onto `base_url` (`/me` → `…/v1/me`); an absolute URL (with its own
scheme) bypasses `base_url` but still gets the default headers. Per-request
`headers` are merged over the client's defaults (a per-key match overrides).
`close()` releases the connection; the GC also closes a client that goes out of
scope (so an explicit `close` is optional). A request after `close` is an
`HttpError`. The handle is non-sendable (one connection, one thread): to fan out,
make a client per isolate.
**Parallel and racing requests** use the general [`Parallel`](#12-isolate)
combinators over `Http.get`, not an HTTP-specific API — the same shape as
`Promise.all`/`race` in JS or `Task.async_stream` in Elixir:
```culebra
# doctest: skip
let urls = ["https://api.example/a", "https://api.example/b"]
Parallel.map(urls, |u| Http.get(u).body) # all, input order (fail-fast)
Parallel.map_settled(urls, |u| Http.get(u)) # allSettled: [{ok, value, error}, ...]
Parallel.race(urls, |u| Http.get(u)) # first success wins, cancels the rest
```
TLS currently links OpenSSL statically; a future swap to BoringSSL is a build-only
change and does not affect this API (BoringSSL verifies hostnames more strictly, so
a server with a CN-only certificate that works today may then be rejected).
### `Http.server() -> Object`
An HTTP server. Register routes with `get`/`post`/`put`/`delete`/`patch`/`options`,
serve files with `static`, then `listen` to accept connections. Each handler is a
`fn(req) -> response` closure.
```culebra
# doctest: skip
let srv = Http.server()
srv.get("/", fn (req) {
"Hello, world!"
})
srv.get("/users/:id", fn (req) {
"user " + req.params["id"]
})
srv.post("/echo", fn (req) {
req.body
})
srv.get("/json", fn (req) {
{
status: 201,
body: '{"ok":true}',
content_type: "application/json",
headers: {"X-Trace": req.headers["X-Request-Id"]},
}
})
srv.static("/assets", "./public")
srv.listen(8080) # blocks; Ctrl+C to stop
```
| Method | Effect |
| --- | --- |
| `get/post/put/delete/patch/options(pattern, handler)` | register `handler` (a `fn(req)->response`) for that method and route `pattern`; returns the server (so calls chain) |
| `static(mount, dir)` | serve static files at the URL prefix `mount`; `dir` is a String path (a live on-disk directory) or any `Dir` — `Dir.embedded(...)` is baked into the binary under AOT (see [Serving a Dir](#serving-a-dir)) |
| `sink.write(chunk)` | (inside a `stream:` closure) push one chunk; returns `false` if the client has disconnected |
| `bind(port, host="0.0.0.0") -> Long` | open the listening socket and return the port it got; `port=0` asks the OS for an ephemeral one; a `port` outside 0..65535 is a `ValueError` (on `listen` / `listen_async` too). Once only, and not after the server has served |
| `serve(workers=0)` | run the accept loop on a bound socket until interrupted (blocks the calling thread). Handlers run on a worker pool, never on the accept loop, so a slow handler can't block accepting new connections — and handlers must be **Sendable**. `workers=0` (default) picks a CPU-scaled pool size; pass a positive count to fix it |
| `serve_async(workers=0)` | same, but serve on a background pool and return immediately; stop with `stop()` |
| `listen(port, host="0.0.0.0", workers=0)` | `bind` + `serve` in one call. Returns only once stopped |
| `listen_async(port, host="0.0.0.0", workers=0) -> Long` | `bind` + `serve_async`; returns the bound port |
| `stop()` | stop a background (`listen_async`) server and join its thread (can be called from another thread) |
| `close()` | stop serving and release the server (the GC also closes one that goes out of scope) |
**The request `req`** is an object with `method`, `path`, and `body` (Strings), and
`headers`, `query` (the parsed query string), and `params` (matched route path
parameters, e.g. `:id` → `req.params["id"]`) as objects of String — Express-style
names. A missing key reads as `nil`.
**The handler's return value** becomes the response:
- a `String` → `200`, `text/plain`, that string as the body;
- an `Object` `{status?, body?, headers?, content_type?}` → full control (absent
fields default to `200` / `""` / `text/plain`); `headers` is an object of String.
A name must be an HTTP token and a value free of control characters (RFC 9110
§5); either failing is a `ValueError` naming it, raised before the response is
sent — the same `500` a raised error gets;
- an `Object` with a `stream:` Function → a chunked (streaming) response — see below;
- `nil` → `200` with an empty body.
A handler that raises becomes a `500` whose body is the error message. One
that throws a value answers with what the value says: its `message` when it is
an Object whose `message` is a non-empty String, otherwise the value as an
uncaught `throw` prints it (a `__str__` is not consulted). An unmatched route
is a `404`.
**Streaming responses (SSE / chunked).** To send a response incrementally —
without buffering the whole body — return an Object whose `stream` field is a
`fn(sink)` closure. The body is sent chunked; `status`, `content_type`, and
`headers` apply as usual (`body` and `stream` together is a `TypeError`). The
closure is called with a `sink` handle; `sink.write(chunk)` pushes one chunk and
returns `false` if the client has gone away (so a long loop can stop early):
```culebra
# doctest: skip
srv.get("/events", fn (req) {
{content_type: "text/event-stream", headers: {"Cache-Control": "no-cache"}, stream: fn (sink) {
for i in 0..10 {
sink.write("data: " + i.to_string() + "\n\n")
}
}}
})
```
The stream closure runs on the worker thread, so under `workers: 1` a long-lived
stream occupies the single thread (use `workers: N` for concurrent streams).
An exception raised mid-stream aborts the connection (the status line is already
sent, so it cannot become a `500`); wrap the body in a `try` if you need to handle
it.
**WebSocket — `srv.ws(pattern, fn(req, ws))`.** Register a WebSocket route; the
handler runs a long-lived loop for the connection, holding one worker until it
returns. The `ws` handle reads and writes messages:
| Method | Effect |
| --- | --- |
| `for msg in ws { … }` | iterate inbound messages (each a `String`); ends when the peer closes |
| `ws.receive()` | the next inbound message as a `String`, or `nil` when the peer closes |
| `ws.try_receive()` | take a message without blocking; answers `Message(String)`, `Empty`, or `Closed` (needs `set_timeout`; see below) |
| `ws.send(msg)` | send a text message; returns `false` if the peer has gone away |
| `ws.set_timeout(ms)` | how long a read waits before reporting `Empty`; `0` waits forever |
| `ws.close()` | close the connection |
| `ws.is_open()` | whether the connection is still open |
```culebra
# doctest: skip
srv.ws("/echo", fn (req, ws) {
for msg in ws {
ws.send(msg)
}
})
srv.ws("/chat", fn (req, ws) {
while true {
let m = ws.receive()
if m == nil {
break
} # peer closed
ws.send(req.path + ": " + m)
}
})
```
A WebSocket connection occupies its worker for its whole lifetime, so `workers: N`
sets the number of simultaneous WebSocket connections (with `workers: 1` a single
connection blocks the server). Connect to a server from culebra with
[`Http.ws`](#httpwsurl---object). The server now covers request/response, route
parameters, static files, streaming, and WebSocket.
**Concurrency.** Handlers always run on a pool of worker threads — never on the
accept loop — so a slow handler can't block accepting new connections, and a
browser (which opens several connections in parallel) isn't serialized. `workers`
sets the pool size:
- `workers: 0` (default) — a CPU-scaled pool (at least 4, at most 8). At least 4
because a browser opens several connections to load a page and each keep-alive
connection holds a worker briefly; capped because each worker carries its own
runtime.
- `workers: N` — a fixed pool of N. Requests are handled in **true
parallel**: no global interpreter lock stands between them.
Each worker has its own runtime, so handlers must be **Sendable**: they can't
capture mutable variables or non-Sendable values. Share large read-only data with
[`Shared.new`](#12-isolate) (one copy across all workers) and open per-worker
resources (a DB connection) inside the handler. A non-Sendable handler is a
`SendError` at `listen`, naming the offending route. (To keep mutable state, route
it through `Shared` / a channel / a hub — the same rules as `Isolate`.)
```culebra
# doctest: skip
let model = Shared.new(load_weights()) # one read-only copy, shared by all workers
let srv = Http.server()
srv.post("/predict", fn (req) {
infer(model, req.body)
})
srv.listen(8080, workers: 8) # 8 handlers run in parallel
```
**Background servers — `listen_async` + `stop`.** To serve while the main thread
does other work — e.g. behind a GUI — use `listen_async`, which serves on a
background thread and returns immediately; `stop()` halts it (and can be called
from another thread). The server keeps running only while you hold the handle, so
keep `srv` referenced for as long as it should serve. Because the handlers run off
the calling thread, they must be **Sendable** (as with `workers > 1`). A bind
failure is reported synchronously as an `HttpError`. A server is single-use:
once it has served, starting it again is an `HttpError` — create a new
`Http.server()` to serve again.
```culebra
# doctest: skip — listens on a port until interrupted
let srv = Http.server()
srv.get("/health", fn (req) {
"ok"
})
srv.listen_async(8080, workers: 4) # returns immediately
# … do other work; call Http.get("http://127.0.0.1:8080/health") …
srv.stop() # stop and join the background thread
```
Alternatively, a blocking `listen` inside an isolate also works — dropping the
isolate (or `Ctrl+C`) stops the accept loop:
```culebra
# doctest: skip — listens on a port until interrupted
let srv_iso = Isolate.spawn(fn () {
let srv = Http.server()
srv.get("/health", fn (req) {
"ok"
})
srv.listen(8080, workers: 4)
})
# … use Http.get("http://127.0.0.1:8080/health") from the main thread …
srv_iso.drop() # signals the server to stop, then joins
```
**Knowing when it is up, and on which port — `bind` + `serve`.** A blocking
`listen` never returns, so it can report nothing: neither "the socket is open"
nor, for a `port 0` bind, which port the OS chose. Splitting the two gives a
point between them where both are known. `bind` returning **is** the readiness
signal — the socket is open with a backlog, so a connection made after it is
queued by the kernel even before `serve` starts accepting.
```culebra
# doctest: skip — listens on a port until interrupted
let (tx, rx) = Channel.new(1)
let srv_iso = Isolate.spawn(fn () {
let srv = Http.server()
srv.get("/health", fn (req) {
"ok"
})
tx.send(srv.bind(0)) # 0 = any free port; hand the number out
tx.drop()
srv.serve() # blocks here
})
tx.drop() # the parent's own sender copy
let base = "http://127.0.0.1:" + rx.recv().to_string()
inspect(Http.get(base + "/health").body) # => 'ok'
srv_iso.drop()
```
Nothing about that is channel-specific — the port is a plain `Long`, so log it,
put it in a `Shared`, or write it to a file. It is also what a handler must
capture to know its own address: the server handle is **non-Sendable**, so a
handler that reads `srv` is rejected, while a captured `Long` copies fine.
```culebra
# doctest: skip — listens on a port until interrupted
let srv = Http.server()
let port = srv.bind(0)
Log.info("http server listening", {port: port})
srv.get("/whoami", fn (req) {
"http://127.0.0.1:" + port.to_string()
})
srv.serve(workers: 4)
```
On the calling thread `listen_async` is enough on its own — it returns the bound
port, so the split is only needed where the call blocks. Routes may be registered
either side of `bind`; only `serve` needs them in place. A `serve` on an unbound
handle, a second `bind`, and a `listen` after `bind` are each a catchable
`HttpError`, and leave the recorded routes intact for a later attempt. A bind
that fails leaves the handle unbound rather than spent, so another port can be
tried.
### `Http.ws(url) -> Object`
Connect a WebSocket client to `url` (`ws://host:port/path`) and return a handle
with the same shape as the server-side `ws`: `send(msg)`, `receive()` (a `String`,
or `nil` once the peer closes), `try_receive()`, `set_timeout(ms)`,
`for msg in ws`, `close()`, and `is_open()`. A bad URL or a failed connect is an
`HttpError`.
```culebra
# doctest: skip
let ws = Http.ws("ws://127.0.0.1:8080/echo")
ws.send("hello")
inspect(ws.receive()) # => the echoed message
for msg in ws {
handle(msg)
} # drains messages until the server closes
ws.close()
```
#### Reading and sending on one connection — `set_timeout` + `try_receive`
A ws handle belongs to the thread that opened it (it is not Sendable, and the
registry behind it is per-thread), so both directions of a connection are
driven from one place. `receive()` and `for msg in ws` block until a message
arrives, which leaves no room to send — the loop is parked in the read.
`set_timeout(ms)` bounds that wait, and `try_receive()` reports what came of
it. It answers with the same three variants
[`Channel.try_recv`](#receiving-without-blocking--rxtry_recv-and-rxdrain) uses,
for the same reason — `nil` cannot mean a message, "nothing yet" and "closed"
at once:
```culebra
enum WsResult { Message(String), Empty, Closed }
```
```culebra
# doctest: skip
ws.set_timeout(50) # ms; 0 waits forever
while running {
match ws.try_receive() {
Message(m) => world.apply(m),
Empty() => nil, # nothing this round — fall through and send
Closed() => running = false,
}
for out in outbox.drain() {
ws.send(out)
}
}
```
Note the `()` on `Empty()` and `Closed()`, as in a `Channel` match: a bare
`Empty` is an identifier pattern that matches anything.
`receive()` and `for msg in ws` keep their meaning under a timeout — they wait
for the *next message*, and a timeout is the absence of one, not the end of the
stream. They do become interruptible with one set, since the wait between
attempts is where a Ctrl+C lands.
---
## 16. `Encoding`
Text codecs, grouped into a **sub-namespace per scheme** (`Encoding.html`,
`Encoding.base64`, `Encoding.hex`, `Encoding.url`, `Encoding.utf16`). The codec logic is shared
between the VM and the JIT/AOT backends, and every codec is
binary-safe (embedded NUL bytes survive a round trip).
### `Encoding.html`
| Function | Result |
| --- | --- |
| `Encoding.html.escape(s)` | `String` — replace the five HTML-unsafe characters (`& < > " '`) with entities |
| `Encoding.html.unescape(s)` | `String` — turn entity references back into their characters |
`escape` replaces `&` first (so the output is always safe to re-escape) and emits
`&` `<` `>` `"` `'`.
`unescape` handles numeric references `&#DDD;` (decimal) and `&#xHHH;` / `&#XHHH;`
(hexadecimal, any case), plus a practical set of named references — the common
typographic, Latin-1, Greek, math, and currency entities, **not** the full HTML5
table of ~2200 names. A reference must end in `;`; anything that is not a
well-formed, recognized reference is left **exactly as written** (browser-style
leniency), so a bare `&` and unknown entities pass through unchanged.
```culebra
inspect(Encoding.html.escape("a & b < c")) # => 'a & b < c'
inspect(Encoding.html.escape("it's fine")) # => 'it's fine'
inspect(Encoding.html.unescape("Tom & Jerry")) # => 'Tom & Jerry'
inspect(Encoding.html.unescape("café — x")) # => 'café — x'
inspect(Encoding.html.unescape("AB")) # => 'AB'
inspect(Encoding.html.unescape("あ")) # => 'あ'
inspect(Encoding.html.unescape("&unknownent;")) # => '&unknownent;'
```
### `Encoding.base64`
| Function | Result |
| --- | --- |
| `Encoding.base64.encode(s)` | `String` — base64 (RFC 4648 standard alphabet, `=` padding) |
| `Encoding.base64.decode(s)` | `String` — the decoded bytes; raises `ValueError` on invalid input |
`encode` is binary-safe (any byte string, including multi-byte UTF-8).
`decode` tolerates ASCII whitespace in the input (so line-wrapped base64
decodes) and `=` padding; an out-of-alphabet character raises `ValueError`.
```culebra
inspect(Encoding.base64.encode("user:pass")) # => 'dXNlcjpwYXNz'
inspect(Encoding.base64.decode("dXNlcjpwYXNz")) # => 'user:pass'
```
```culebra
# doctest: skip
# e.g. an HTTP Basic auth header
let cred = Encoding.base64.encode(user + ":" + password)
let r = Http.get(url, headers: {Authorization: "Basic " + cred})
```
### `Encoding.hex`
| Function | Result |
| --- | --- |
| `Encoding.hex.encode(s)` | `String` — lower-case hex, two digits per byte |
| `Encoding.hex.decode(s)` | `String` — the decoded bytes; raises `ValueError` on invalid input |
`encode` always emits lower-case digits. `decode` accepts either case and
raises `ValueError` on an odd-length string or any non-hex character.
```culebra
inspect(Encoding.hex.encode("abc")) # => '616263'
inspect(Encoding.hex.decode("616263")) # => 'abc'
inspect(Encoding.hex.decode("00FF").size()) # => 2
```
### `Encoding.url`
| Function | Result |
| --- | --- |
| `Encoding.url.encode(s)` | `String` — percent-encode (RFC 3986) |
| `Encoding.url.decode(s)` | `String` — decode percent-escapes |
`encode` keeps the unreserved set `A-Z a-z 0-9 - _ . ~` verbatim and turns every
other byte into `%XX` with upper-case hex, so a space becomes `%20` (not `+`)
and multi-byte UTF-8 is percent-encoded byte by byte. `decode` is lenient: a
`%` not followed by two hex digits is left **as written**, and a literal `+`
stays a `+` (so `encode`/`decode` round-trip exactly).
```culebra
inspect(Encoding.url.encode("a b&c")) # => 'a%20b%26c'
inspect(Encoding.url.decode("a%20b%26c")) # => 'a b&c'
inspect(Encoding.url.encode("café")) # => 'caf%C3%A9'
```
### `Encoding.utf16`
| Function | Result |
| --- | --- |
| `Encoding.utf16.encode(s: String, bom: Bool = false) -> String` | `String` — the UTF-16 bytes of `s`, big-endian; `bom: true` puts the byte-order mark `FE FF` first |
| `Encoding.utf16.decode(bytes: String) -> String` | `String` — the text, as UTF-8 |
A culebra `String` is UTF-8; these read and write the other encoding a file
or a protocol may use (an XML file written by a Windows tool, say). `decode`
takes the byte order from a leading byte-order mark (`FE FF` big-endian,
`FF FE` little-endian) and drops the mark; without one it reads big-endian,
as RFC 2781 says — which is how Java's `UTF-16` reads it, and not how
Python's `utf-16` does (that assumes the machine's order). An odd number of
bytes, or a surrogate without its pair (named by its byte offset), raises
`ValueError`; `encode` raises it for a `String` that is not UTF-8.
```culebra
let b = Encoding.utf16.encode("héllo")
inspect(Encoding.hex.encode(b)) # => '006800e9006c006c006f'
inspect(Encoding.utf16.decode(b)) # => 'héllo'
inspect(Encoding.hex.encode(Encoding.utf16.encode("😀", bom: true))) # => 'feffd83dde00'
inspect(Encoding.utf16.decode(Encoding.hex.decode("fffe6800e900"))) # => 'hé'
```
---
## 17. `Compress`
gzip / deflate (de)compression, backed by zlib. Every function is binary-safe
(embedded NUL bytes survive a round trip).
| Function | Result |
| --- | --- |
| `Compress.gzip(data: String) -> String` | gzip-compressed bytes (RFC 1952 wrapper); interoperates with the standard `gzip` tool |
| `Compress.gunzip(data: String) -> String` | the decompressed bytes; raises `ValueError` on malformed input |
| `Compress.deflate(data: String, level: Long = -1) -> String` | zlib-compressed bytes (RFC 1950 wrapper) — `gzip` minus its gzip-specific header |
| `Compress.zip(files: Object \| Dir) -> String` | the bytes of a ZIP archive holding `files` |
`gunzip` auto-detects the header, so it decompresses both `gzip` and
`deflate` output with the one function — there is no separate `inflate`.
Several gzip members back to back (as `cat a.gz b.gz` makes) decompress
to their concatenation. A truncated or unrecognized input, or bytes after
the compressed stream that start no gzip member, raise `ValueError`.
```culebra
let original = "the quick brown fox the quick brown fox the quick brown fox the quick brown fox"
let z = Compress.gzip(original)
inspect(z.size() < original.size()) # => true
inspect(Compress.gunzip(z) == original) # => true
```
```culebra
# doctest: skip
# Read a .gz file and write one back
let text = Compress.gunzip(FS.read("logs.gz"))
FS.write("out.gz", Compress.gzip(text))
```
`deflate` differs from `gzip` only in the wrapper: no filename/mtime header,
no CRC-32 trailer, just a 2-byte header and an Adler-32 checksum — the format
a PNG's `IDAT` chunk holds, and the smaller choice when the gzip envelope adds
nothing (an in-memory blob, a value embedded in another container).
```culebra
let text = "the quick brown fox the quick brown fox the quick brown fox"
let z2 = Compress.deflate(text) # zlib wrapper, not gzip's
inspect(Compress.gunzip(z2) == text) # the same decoder handles both
# => true
inspect(z2.size() < Compress.gzip(text).size()) # no gzip header to pay for
# => true
```
`level` follows zlib's own convention: `-1` (the default) picks zlib's
built-in tradeoff, `0` stores the input with no compression, and `9` spends
the most time for the smallest output. A value outside `-1..9` raises
`ValueError` at the same call.
```culebra
let text = "the quick brown fox the quick brown fox the quick brown fox"
inspect(Compress.deflate(text, level: 9).size() <=
Compress.deflate(text, level: 0).size()) # => true
```
HTTP responses are decompressed transparently by the `Http` client (it sends
`Accept-Encoding` and inflates `Content-Encoding: gzip` automatically), so
`Compress` is for data and files you handle yourself, not for `Http` bodies.
`zip` packs a set of files into an archive: an Object of one key per file
(`"sub/name.txt"` to its contents, as [`Dir.memory`](#41-dir) takes it) or any
[`Dir`](#41-dir). A path holding `\` or starting with a drive (`C:`) raises
`ValueError`, since an archive reader takes either for a separator or a root.
The entries go in sorted by path, with one fixed timestamp,
so the same files always make the same bytes. Writing the archive is
`FS.write`; reading one back, or extracting it, is
[`Dir.zip`](#zip-archives).
```culebra
let z = Compress.zip({'readme.md': '# hi', 'img/logo.png': 'PNG'})
inspect(z.slice(0, 2)) # => 'PK'
inspect(z == Compress.zip({'img/logo.png': 'PNG', 'readme.md': '# hi'})) # => true
inspect(Dir.zip(bytes: z).files()) # => ['img/logo.png', 'readme.md']
```
```culebra
# doctest: skip — writes a file
FS.write("site.zip", Compress.zip(Dir.disk("public")))
```
---
## 18. `Hash`
Message digests and HMAC, self-hosted (no OpenSSL dependency) and identical on
every backend. Each function returns the **lowercase hex** digest. Inputs are
binary-safe (embedded NUL bytes are part of the message, not a terminator).
| Function | Result |
| --- | --- |
| `Hash.sha256(data: String) -> String` | 64-char hex SHA-256 digest |
| `Hash.sha1(data: String) -> String` | 40-char hex SHA-1 digest |
| `Hash.sha512(data: String) -> String` | 128-char hex SHA-512 digest |
| `Hash.md5(data: String) -> String` | 32-char hex MD5 digest |
| `Hash.hmac_sha256(key: String, data: String) -> String` | 64-char hex HMAC-SHA-256 |
| `Hash.hmac_sha1(key: String, data: String) -> String` | 40-char hex HMAC-SHA-1 |
| `Hash.hmac_sha512(key: String, data: String) -> String` | 128-char hex HMAC-SHA-512 |
```culebra
inspect(Hash.sha256("abc"))
# => 'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad'
inspect(Hash.md5("abc"))
# => '900150983cd24fb0d6963f7d28e17f72'
inspect(Hash.hmac_sha256("Jefe", "what do ya want for nothing?"))
# => '5bdcc146bf60754e6a042426089575c75a003f089d2739839dec58b964ec3843'
```
For a raw (non-hex) digest, decode the hex with `Encoding.hex.decode`; to
re-encode in another form, pair it with `Encoding.base64`. MD5 and SHA-1 are
provided for compatibility with existing systems (checksums, legacy APIs) — use
SHA-256 or SHA-512 for new security-sensitive work.
---
## 19. `CSV`
Parse and render comma-separated values (RFC 4180-ish), shared byte-for-byte
across backends. Fields are separated by commas and records by newlines; a
field containing a comma, double quote, or newline is wrapped in double
quotes, and an embedded quote is doubled (`""`).
| Function | Result |
| --- | --- |
| `CSV.parse(text, delimiter=",", header=false, types=nil) -> Array` | rows of String fields, or (with `header`) an Array of Objects |
| `CSV.stringify(rows: Array, delimiter: String = ",") -> String` | CSV text; each row must be an Array, each field rendered like `to_string` |
The `delimiter:` option (a single byte; first byte used) selects the field
separator — pass `"\t"` for TSV. It may be given positionally or by keyword.
By default `parse` is lenient (no errors): every field comes back as a `String`
(numbers are not inferred), an empty input yields no rows, and a trailing newline
does not add an empty final row. Both LF and CRLF end a record. `stringify`
renders each field with the same conversion as `to_string`, so scalars like
numbers and `Bool` serialize naturally; `stringify(parse(text))` round-trips
well-formed input.
```culebra
let rows = CSV.parse("name,age\nalice,30\nbob,25")
inspect(rows[1]) # => ['alice', '30']
inspect(CSV.stringify([["a,b", "c"], [1, 2]]) == "\"a,b\",c\n1,2") # => true
inspect(CSV.parse("a\tb", delimiter: "\t")[0]) # => ['a', 'b']
```
**Header mode — `header: true`.** The first row names the columns, and each later
row becomes an `Object` keyed by those names (instead of a positional Array):
```culebra
let rows = CSV.parse("name,age\nalice,30\nbob,25", header: true)
inspect(rows[0]["name"]) # => 'alice'
```
A header with no data rows (or empty input) yields `[]`. A duplicate header name,
or a data row whose field count differs from the header, is a `ValueError`.
**Typed columns — `types:`.** Pass `types:` (an `Object` mapping a header name to
`"String"` / `"Long"` / `"Float"` / `"Bool"`) to coerce those columns; columns
not listed stay `String`. Coercion is **explicit, never inferred** — so a value
like a ZIP code or ID declared `String` keeps its exact text (no leading-zero or
precision loss). `types:` requires `header: true`.
```culebra
let rows = CSV.parse(
"name,age,active\nalice,30,true",
header: true,
types: {age: "Long", active: "Bool"},
)
# age is a real Long now (not a String), so arithmetic works:
inspect(rows[0]["age"] + 1) # => 31
# a ZIP code stays exact text — no number inference:
let z = CSV.parse("zip\n01234", header: true, types: {zip: "String"})
inspect(z[0]["zip"]) # => '01234'
```
An unknown type name, a `types` key that names no column, or a cell that can't be
coerced (e.g. `"hello"` as a `Long`, or an empty cell) is a `ValueError` tagged
with the record number and column. Bool accepts exactly `"true"` / `"false"`;
coercion does not trim surrounding whitespace.
Constraint validation (ranges, regex, allowed sets, uniqueness) is intentionally
*not* built in — that is a separate, general concern. Convert ad-hoc columns with
`to_long` / `to_float` if you don't want a `types:` map.
---
## 20. `Env`
Parse and load dotenv-style `.env` configuration, shared byte-for-byte across
backends.
| Function | Result |
| --- | --- |
| `Env.parse(text: String) -> Object` | parse dotenv text into an `Object` of `String` values (no side effects) |
| `Env.load(path: String = ".env", override: Bool = false) -> Object` | read a file, parse it, set each entry into the process environment, and return the parsed `Object` |
Each entry is one line, `KEY=VALUE`:
- blank lines and lines whose first non-space character is `#` are ignored;
- a leading `export ` is stripped (so shell-style `.env` files load as-is);
- the key is trimmed; a line without `=` or with an empty key is skipped;
- a double-quoted value (`"..."`) honours `\n`, `\t`, `\r`, `\\`, and `\"`
escapes; a single-quoted value (`'...'`) is raw; an unquoted value is
trimmed and an inline ` # comment` (a `#` preceded by whitespace) is removed,
while a `#` not preceded by whitespace stays literal;
- a duplicate key keeps its first position with the last value winning.
Parsing is lenient (malformed lines are skipped). Multi-line values are not
supported. `Env.load` raises `IOError` if the file cannot be opened. By default
it does **not** overwrite a variable already set in the environment (the real
environment wins); pass `override: true` to replace existing values. Loaded
variables are visible via `Sys.env` and inherited by children spawned afterward
(e.g. `Proc.run`).
```culebra
let cfg = Env.parse("# app config\nPORT=8080\nNAME=\"my app\"\nDEBUG=true")
inspect(cfg["PORT"]) # => '8080'
inspect(cfg["NAME"]) # => 'my app'
inspect(cfg["DEBUG"]) # => 'true'
```
```culebra
# doctest: skip
Env.load(".env") # set vars from ./.env (if present)
inspect(Sys.env("PORT"))
```
To use values as numbers or booleans, convert at the use site with `to_long` /
`to_float` (every value is a `String`).
---
## 21. `UUID`
Generate canonical lowercase UUIDs (the `8-4-4-4-12` hyphenated form). Two
variants:
| Function | Result |
| --- | --- |
| `UUID.v4() -> String` | random UUID (122 random bits) |
| `UUID.v7() -> String` | time-ordered UUID — a 48-bit Unix-millisecond prefix then random, so values sort by creation time (good as database keys) |
Entropy comes from the shared PRNG that `Random.*` uses, so UUIDs are
reproducible under `Random.seed` and are **not** cryptographically secure —
fine for identifiers, not for tokens or secrets. v7 orders by millisecond;
two values created within the same millisecond are not ordered relative to
each other (there is no monotonic counter).
```culebra
inspect(UUID.v4().size()) # => 36
inspect(UUID.v4() != UUID.v4()) # => true
```
---
## 22. `Term`
Terminal control for building text UIs — color, cursor positioning, the
alternate screen, terminal size, and non-blocking key input. The color and
escape helpers are pure functions that return strings, so they compose and
are easy to test; the stateful pieces (raw mode, the render loop) are wrapped
so the terminal is always restored on exit.
### Color and attributes
Each returns its argument wrapped in the matching ANSI codes (and a reset),
so calls nest:
| Function | Result |
| --- | --- |
| `Term.fg(s, n) -> String` | 256-color foreground (`n` is 0–255) |
| `Term.bg(s, n) -> String` | 256-color background |
| `Term.rgb(s, r, g, b) -> String` | 24-bit truecolor foreground |
| `Term.red(s)` / `green` / `yellow` / `blue` / `magenta` / `cyan` / `white` / `black` | named 16-color foreground |
| `Term.bold(s)` / `Term.dim(s)` / `Term.underline(s)` / `Term.reverse(s)` | text attributes |
| `Term.style(fg = nil, bg = nil, bold = false, dim = false, underline = false, reverse = false) -> String` | an SGR parameter string for a `Screen` cell; `fg`/`bg` take a 256-color index or an `(r,g,b)` tuple |
Colors adapt to the terminal's **capability level** — `0` none, `1` 16,
`2` 256, `3` truecolor — detected from `isatty`, `NO_COLOR` (present ⇒ off),
`FORCE_COLOR`, `COLORTERM`, and `TERM`. A color beyond the level is
downsampled (truecolor → nearest 256 → nearest 16), and at level 0 nothing
is emitted, so piped or `NO_COLOR` output stays plain. `Term.level()` reads
the level and `Term.set_level(n)` overrides it. The `fg`/`bg`/`rgb`/`bold`/…
helpers wrap a string for direct printing; `Term.style(...)` produces the
style passed to `screen.set` / `screen.put` for colored cells.
```culebra
inspect(Term.bold(Term.fg("alert", 196))) # bold bright-red "alert" (printed)
let st = Term.style(fg: (255, 128, 0), bold: true) # for a Screen cell
```
### Escapes, size, and width
| Function | Result |
| --- | --- |
| `Term.clear() -> String` | clear screen + home the cursor |
| `Term.move(x, y) -> String` | cursor to column `x`, row `y` (0-based) |
| `Term.hide()` / `Term.show()` | hide / show the cursor |
| `Term.cols()` / `Term.rows() -> Long` | terminal size in cells (80×24 off a tty) |
| `Term.size() -> (Long, Long)` | `(cols, rows)` |
| `Term.width(s) -> Long` | display width in columns (wide / emoji = 2, combining = 0) |
| `Term.flush()` | flush buffered output |
### Piped input
`Term.attach_tty() -> Bool` reattaches stdin to the controlling terminal,
replacing whatever stdin currently is (a pipe, a redirected file). Raw mode
and key reads (`Term.app`, `Term.poll`, `Term.read_key`) only work off a real
tty, so a script fed content on stdin (`some-command | script.cul`) can drain
that pipe first, then call `Term.attach_tty()` to switch to interactive key
input — the same pattern `less` uses. Returns `false` (stdin is left
untouched) when there is no controlling terminal at all, e.g. running fully
detached or under CI.
```culebra
# doctest: skip
let content = IO.stdin().read() # drain the pipe first
if !Term.attach_tty() {
println(content) # no terminal to be interactive on
Sys.exit(0)
}
Term.app(fn (screen) { ... })
```
### Input events
Input is a single event model: `Term.poll(timeout)` — also reachable as
`screen.poll(timeout)` inside a `Term.app` callback — returns one event
**Object**, or `nil` when nothing arrived within `timeout` seconds.
`Term.parse(raw)` turns an already-read escape sequence into the same
shape. A `kind` field discriminates; modifiers are booleans.
| `kind` | Fields |
| --- | --- |
| `"key"` | `key`, `ctrl`, `shift`, `alt` |
| `"mouse"` | `event`, `button`, `x`, `y`, `ctrl`, `shift`, `alt` |
| `"resize"` | `cols`, `rows` |
For a **key**, `key` is a printable character (`"q"`, `" "`) or a name:
`"up"` / `"down"` / `"left"` / `"right"`, `"enter"`, `"escape"`, `"tab"`,
`"backspace"`, `"insert"`, `"delete"`, `"home"`, `"end"`, `"pageup"`,
`"pagedown"`, `"f1"`…`"f12"`. Modifiers are reported in `ctrl` / `shift` /
`alt` (e.g. Ctrl+Right → `{key: "right", ctrl: true}`, Ctrl+C →
`{key: "c", ctrl: true}`, Alt+x → `{key: "x", alt: true}`).
For a **mouse** event (see below), `event` is `"press"` / `"release"` /
`"drag"` / `"scroll"`; `button` is `"left"` / `"middle"` / `"right"` /
`"wheel_up"` / `"wheel_down"`; `x` / `y` are 0-based cells.
`Term.resized() -> Bool` is the lower-level resize flag (true once after a
SIGWINCH); `poll` turns it into a `"resize"` event.
Mouse reporting is opt-in: enable it with `Term.app(..., mouse: true)` (or
print `Term.mouse_on()` / `Term.mouse_off()` yourself).
```culebra
# doctest: skip
let ev = screen.poll(0.1)
if ev != nil {
if ev.kind == "key" && ev.key == "q" { ... }
else if ev.kind == "mouse" && ev.event == "press" { ... ev.x, ev.y ... }
}
```
### `Term.app` and `Screen`
`Term.app(fn (screen) { ... }, mouse: false)` enters raw mode and the
alternate screen, hides the cursor, watches for resizes (and enables mouse
reporting when `mouse: true`), and **restores the terminal on exit**
— normal return, an exception, or Ctrl+C — via `defer`. The callback
receives a `Screen`:
| Method | Effect |
| --- | --- |
| `screen.size()` / `cols()` / `rows()` | terminal dimensions |
| `screen.clear()` | reset the back buffer to a blank frame (current size) |
| `screen.set(x, y, glyph, style = "")` | place one grapheme (with an optional `Term.style`) into the back buffer |
| `screen.put(x, y, s, style = "")` | lay the graphemes of `s` into successive cells, all in `style` |
| `screen.render() -> String` | minimal escapes to update the screen from the last frame (and advance the front buffer) |
| `screen.flush()` | print `render()` and flush |
| `screen.poll(timeout) -> Object?` | wait up to `timeout` seconds for an input event (key / mouse / resize), or `nil` |
A `Screen` is a double-buffered grid of cells, each holding a glyph and an
optional style. `flush` emits **only the cells that changed** since the last
frame, with the minimal SGR transitions between styles, so live UIs update
without flicker and with minimal output; wide glyphs occupy two cells and a
resize forces a full repaint. Build a frame with `clear` + `set` / `put`
(passing a `Term.style(...)` for color), then `flush`; read input with `poll`
(which doubles as the per-frame delay).
```culebra
Term.app(fn (s) {
s.clear()
s.put(2, 1, "hello")
s.flush()
s.poll(2.0) # wait up to 2s for a keypress
})
```
---
## 23. `Log`
Leveled, structured logging to **standard error** (so it never pollutes the
program's stdout data, which a pipe like `myscript | jq` consumes).
| Function | Result |
| --- | --- |
| `Log.debug(msg: String, fields: Object = {})` | log at `debug` |
| `Log.info(msg, fields = {})` | log at `info` |
| `Log.warn(msg, fields = {})` | log at `warn` |
| `Log.error(msg, fields = {})` | log at `error` |
| `Log.with(fields: Object) -> logger` | a child logger that binds `fields` into every record |
| `Log.set_level(level: String) -> Nil` | set the threshold (`"debug" < "info" < "warn" < "error"`) |
| `Log.set_format(format: String) -> Nil` | `"text"` (default) or `"json"` |
Each call takes a message and an optional `Object` of structured fields. Only
records at or above the threshold are emitted (default `info`, so `debug` is
dropped). The threshold and format default from the `LOG_LEVEL` / `LOG_FORMAT`
environment variables and can be overridden with `set_level` / `set_format`; an
unknown level or format raises `ValueError`. A timestamp (ISO 8601 UTC) is always included.
`text` is human-readable (the level is colored when stderr is a terminal):
```
2026-06-24T21:30:01Z info request done method=GET status=200 ms=12.4
```
`json` emits one object per line (JSON Lines — pipe to `jq` or a log shipper):
```json
{"time":"2026-06-24T21:30:01Z","level":"info","msg":"request done","method":"GET","status":200,"ms":12.4}
```
`Log.with(fields)` returns a **child logger** that carries `fields` on every
line, so request- or job-scoped context is bound once instead of repeated.
Children share the global level/format and can nest. The reserved keys `time`,
`level`, and `msg` always win over a field of the same name.
```culebra
# doctest: skip
Log.info("server started")
Log.set_level("debug")
Log.debug("cache miss", {key: "user:42"})
let log = Log.with({request_id: id}) # bind context once
log.info("received") # ...carried on every line
log.error("upstream failed", {status: 502})
```
Values are rendered with `to_string` in text mode and serialized via the JSON
namespace in json mode. For a fatal condition, log at `error` then
`Sys.exit(1)` — there is no separate `fatal` level.
---
## 24. `TOML`
Parse and render [TOML](https://toml.io) configuration, shared byte-for-byte
across backends. The grammar and serialization live in a value-neutral core,
so the VM, JIT, and AOT agree exactly.
| Function | Result |
| --- | --- |
| `TOML.parse(text: String) -> Object` | the document as a nested `Object` |
| `TOML.stringify(v: Object, sort_keys: Bool = false) -> String` | TOML text; sub-tables become `[section]` headers |
`parse` accepts the TOML v1.0 surface: bare / quoted / dotted keys,
`[table]` and `[[array.of.tables]]` headers, the four string forms (basic,
literal, and their multiline `"""` / `'''` variants), integers
(decimal / `0x` / `0o` / `0b`, with `_` separators), floats (including
`inf` / `nan`), booleans, arrays, and inline tables. Values map as:
| TOML | Culebra |
| --- | --- |
| table / inline table | `Object` (insertion order preserved) |
| array | `Array` |
| array of tables `[[x]]` | `Array` |
| string (any of the four forms) | `String` |
| integer | `Long` |
| float | `Float` |
| boolean | `Bool` |
| date-time / date / time | `String` (kept raw — there is no date type) |
Date-times come back as their raw text rather than a dedicated type, so a
round trip re-quotes them as ordinary strings. A `\uXXXX` / `\UXXXXXXXX`
escape must name a Unicode scalar value — a surrogate (`U+D800`–`U+DFFF`) or a
code point above `U+10FFFF` is a parse error, the same boundary culebra's own
string escapes enforce. Malformed input raises a `ValueError` whose
`e.line` / `e.col` (both 1-based) point at the offending character:
```culebra
let r = try {
TOML.parse("x = ")
nil
} catch e {
e
}
inspect(r.message) # => 'TOML.parse: expected value'
inspect("{r.line}:{r.col}") # => '1:5'
```
Nesting is bounded: a document whose tree runs more than 1000 levels deep —
through nested arrays / inline tables or through dotted keys — raises a
`ValueError` (`nesting too deep (limit 1000)`) instead of exhausting the
C stack. `stringify` applies the same bound to the value tree it walks.
`stringify` takes an `Object` (a TOML document is always a table) and renders
each scalar / array / inline value first, then expands sub-tables into
`[section]` headers and table arrays into `[[…]]` blocks, so every bare key
precedes the header that would otherwise capture it. Floats always carry a
decimal point so they parse back as floats. `sort_keys: true` walks keys
alphabetically for deterministic output. Functions, tensors, and Objects with
non-String keys are not serializable — `stringify` throws `TypeError`.
```culebra
let cfg = TOML.parse("""
title = "demo"
ports = [80, 443]
[server]
host = "localhost"
""")
inspect(cfg.title) # => 'demo'
inspect(cfg.server.host) # => 'localhost'
```
`stringify` renders bare keys before the headers that would capture them:
```culebra
inspect(TOML.stringify({a: 1, b: {c: 2}}))
# => |
# 'a = 1
#
# [b]
# c = 2
# '
```
---
## 25. `XML`
Read XML into plain values, search it with a small path language, and write it
back. There is no element type: an element is an `Object` with four fields.
The elements `parse` returns, and every element found in them, also have
methods: `el.find(path)` is `XML.find(el, path)`, and so on for the rest.
| Call | Result |
| --- | --- |
| `XML.parse(text, keep_space=false) -> Object` | the root element |
| `XML.events(text, keep_space=false) -> Iterator` | the document as a lazy stream of events |
| `el.find(path, namespaces={}) -> Object \| Nil` | the first match, or `nil` |
| `el.find_all(path, namespaces={}) -> Array` | every match |
| `el.find_iter(path, namespaces={}) -> Iterator` | every match, as an `Iterator` |
| `el.text(path=".", namespaces={}) -> String \| Nil` | the text of the first match, or `nil` |
| `el.stringify(indent=0, declaration=false) -> String` | XML text |
An element has four fields, in this order:
* `tag`: the name as written, prefix included (`'item'`, `'m:image'`).
* `ns`: the namespace URI the name resolves to, or `nil` when it has none.
* `attrs`: an `Object` of `String` values, keys as written and in document
order. Namespace declarations (`xmlns`, `xmlns:m`) stay here, which is how
`stringify` writes them back.
* `children`: an `Array` of elements and `String`s (text), in document order.
The methods are not fields: they live on a prototype every parsed element
shares, so `inspect`, `==`, `keys()`, `for` and `JSON.stringify` see the four
fields only, and `type_of` answers `'Object'`. `el.has('find')` is `true`, as
for a class instance's method; `el.get('find', d)` and `el['find']` see no such
field. An element built by hand, or a copy made with spread (`{...el}`), is a
plain `Object` without the methods; the `XML` functions take any `Object` of
this shape, so use them there. A parsed element keeps its methods when it is
sent to another `Isolate` or through a `Channel`. `Shared.new` refuses one,
since its methods are functions and a `Shared` value is data only.
```culebra
let doc = XML.parse('- Tea
- Mug & saucer
\n ")
nil
} catch e {
e
}
println(r.message) # => XML.parse: mismatched end tag: expected ' ', got ''
inspect("{r.line}:{r.col}") # => '2:6'
```
### `XML.events(text, keep_space=false) -> Iterator`
The same parser as `parse`, one event at a time, for a document too large to
hold as a tree or when only part of it is needed. Each event is an `Object`:
| `kind` | Other fields |
| --- | --- |
| `'start'` | `tag`, `ns`, `attrs` |
| `'end'` | `tag`, `ns` |
| `'text'` | `text` |
`boot ready boot <<<') {
inspect(ev) # => {kind: 'start', tag: 'log', ns: nil, attrs: {}}
break
}
```
### `el.find(path, namespaces={}) -> Object | Nil`, `XML.find(el, path, namespaces={}) -> Object | Nil`
### `el.find_all(path, namespaces={}) -> Array`, `XML.find_all(el, path, namespaces={}) -> Array`
### `el.find_iter(path, namespaces={}) -> Iterator`, `XML.find_iter(el, path, namespaces={}) -> Iterator`
Search the tree under `el` with `path`. `find_all` returns every match in
document order without duplicates, `find` the first of them (or `nil`), and
`find_iter` the same matches as `find_all` as an `Iterator`. The matches are
the elements themselves, not copies.
The path language is a subset of XPath: the abbreviated location paths of
XPath 1.0, plus two name forms from later versions (`*:tag` from XPath 2.0,
`Q{uri}tag` from XPath 3.0). Each form means what it means in XPath. A path is
relative to `el`; steps are separated by `/`.
| Step | Selects |
| --- | --- |
| `tag` | child elements named `tag` |
| `p:tag` | child elements named `tag` in the namespace bound to `p` |
| `Q{uri}tag` | child elements named `tag` in namespace `uri` (`Q{}tag`: in no namespace) |
| `*:tag` | child elements named `tag` in any namespace or none |
| `*`, `p:*`, `Q{uri}*` | every child element (in that namespace) |
| `.` | the element itself |
| `..` | the parent (nothing above `el`) |
| `a//b` | `b` elements anywhere below an `a`, at any depth (`.//b`: anywhere below `el`) |
| Predicate | Keeps an element that |
| --- | --- |
| `[@attr]` | has attribute `attr` |
| `[@attr='v']` | has `attr` equal to `v` (`"v"` works too) |
| `[p]` | has an element the relative path `p` selects from it (`[title]`, `[pitch/step]`, `[.//b]`, `[../note]`) |
| `[p='v']` | has an element `p` selects whose text is `v` (`[title='B']`) |
| `[p/@attr]`, `[p/@attr='v']` | has an element `p` selects with attribute `attr` (equal to `v`); `p//@attr` looks at those elements and every element below them |
| `[n]` | is the `n`th of the candidates (1-based) |
| `[last()]`, `[last()-n]` | is the last candidate, or `n` before it |
Predicates chain (`book[@lang='en'][2]`), each filtering what the one before
kept. As in XPath, a position counts among the children of one parent:
`book[1]` is the first `book` of every parent that has one, and `.//book[1]`
the first `book` child of `el` and of each element under it. Attribute names in
predicates are matched as written in the document (`[@xlink:href]`).
A path in a predicate starts from the candidate element and takes every step
the outer path takes, `.` and `..` included, with predicates of its own
(`note[pitch[step='C']]`); its `..` reaches up to `el` and no further. Its
names resolve as in the outer path. As in XPath, `[p='v']` holds when any one
of the elements `p` selects has text `v`. Predicates nest at most 1000 deep.
**Names and namespaces.** A prefix in a path is looked up in `namespaces:`
first, then in the `xmlns:p` declarations on `el` itself (not its ancestors,
which an element does not record). A prefix found in neither is a
`ValueError`. An unprefixed name matches only elements with no namespace, even
under a default `xmlns=`; to match those, map `""` in `namespaces:` to their
URI (XPath 2.0's default element namespace), or write `Q{uri}tag` or `*:tag`.
Everything else is rejected with a `ValueError` naming the construct and its
column: axes (`following-sibling::`), functions other than `last()`, `and`,
`or`, comparisons other than `=` or with anything but a literal, unions (`|`),
attribute steps (`a/@href`) other than at the end of a predicate's path, and
absolute paths (`/a`, `//a`, in a predicate too: "a path is relative to the
element"). A name in braces without the `Q` (`{uri}tag`, `{*}tag`) is not XPath
either; its error gives the XPath spelling (`Q{uri}tag`, `*:tag`).
```culebra
let lib = XML.parse('A B C First a bold word
').text()) # => 'a bold word'
inspect(XML.parse('
').text('q')) # => ''
inspect(XML.parse('
').text('q')) # => nil
```
### `el.stringify(indent=0, declaration=false) -> String`, `XML.stringify(el, indent=0, declaration=false) -> String`
Write an element as XML text: the `tag`, then `attrs` in order, then
`children`. `ns` is not read; namespace declarations come from `attrs`.
* `tag` must be a `String`. `attrs` and `children` may be left out (or `nil`);
when present `attrs` is an `Object` whose values are `String`, `Long`,
`Float` or `Bool`, and `children` an `Array` of elements and `String`s.
Anything else is a `TypeError`.
* The tag and each key of `attrs` must be names `parse` accepts (a qualified
name: name characters, at most one `:`, not at either end); another is a
`TypeError`. Text and attribute values that hold bytes which are not UTF-8,
or a character no XML document can contain (most control characters), are
a `ValueError`.
* An element with no children is written `Hi Say hi
#
# Hi
#
# Say hi
#
```
---
## 26. `SQLite`
Embedded SQL database backed by [SQLite](https://sqlite.org) (the amalgamation
is vendored and compiled in — no system library is required). `SQLite.open`
returns a stateful **Database** handle; the high-level `execute` / `query` /
`transaction` methods cover everyday CRUD, and `prepare` returns a reusable
**Statement** handle for hot loops. Both handles close deterministically when
they leave scope, so an explicit `close` / `finalize` is optional.
| Function | Result |
| --- | --- |
| `SQLite.open(path: String) -> Database` | open (or create) a database; `":memory:"` for an in-memory one |
| `SQLite.version() -> String` | the linked SQLite library version, e.g. `"3.53.2"` |
### Database
| Method | Result |
| --- | --- |
| `db.execute(sql: String, params = nil) -> Long` | run one statement; returns rows affected |
| `db.query(sql: String, params = nil) -> Array` | run a query; each row is an `Object` keyed by column name |
| `db.prepare(sql: String) -> Statement` | compile a reusable statement |
| `db.transaction(fn: Function) -> Any` | `BEGIN`, run `fn`, `COMMIT`; any throw triggers `ROLLBACK` and re-raises |
| `db.close()` | close the connection (also runs automatically on scope exit) |
### Statement
| Method | Result |
| --- | --- |
| `stmt.run(params = nil) -> Long` | execute (INSERT/UPDATE/DELETE); returns rows affected |
| `stmt.query(params = nil) -> Array` | execute a query and collect the rows |
| `stmt.finalize()` | release the statement (also runs automatically on scope exit) |
### Parameters
`params` binds placeholders in the SQL. An **Array** binds positional `?`
placeholders left to right; an **Object** binds named `:name` (or `@name` /
`$name`) placeholders by key:
```culebra
# doctest: skip
db.execute("INSERT INTO users VALUES (?, ?)", [1, "Alice"])
db.query("SELECT * FROM users WHERE id = :id", {id: 1})
```
### Type mapping
Values are mapped by the column's runtime type (reads) or the culebra value's
type (writes):
| SQLite | Culebra |
| --- | --- |
| INTEGER | `Long` |
| REAL | `Float` |
| TEXT | `String` |
| BLOB | `String` (raw bytes) |
| NULL | `nil` |
On the write side a `Bool` binds as `0` / `1`; binding any other type (an
`Array`, `Object`, function, …) raises `TypeError`. SQL or constraint errors
raise `SQLiteError` carrying SQLite's own message:
```culebra
let db = SQLite.open(":memory:")
db.execute("CREATE TABLE users (id INTEGER, name TEXT)")
db.execute("INSERT INTO users VALUES (?, ?)", [1, "Alice"])
let rows = db.query("SELECT * FROM users")
inspect(rows[0]["name"]) # => 'Alice'
# a reusable prepared statement
let ins = db.prepare("INSERT INTO users VALUES (?, ?)")
for u in [[2, "Bob"], [3, "Carol"]] {
ins.run(u)
}
ins.finalize()
# all-or-nothing
db.transaction(fn () {
db.execute("UPDATE users SET name = 'Bob!' WHERE id = 2")
})
let r = try {
db.query("SELECT * FROM missing")
nil
} catch e {
e
}
inspect(r.kind) # => 'SQLiteError'
db.close()
```
A Database / Statement handle is tied to the thread (isolate) that created it —
it is not `Sendable` and cannot be passed across an `Isolate` / `Channel`
boundary. Transactions do not nest (use `SAVEPOINT` directly if you need that).
---
## 27. `Canvas`
An immediate-mode 2D framebuffer for little games and pixel graphics: draw a
frame, `present` it, poll input, repeat. Colors are packed RGBA `Long`s and
the buffer can be any size (a WASM-4-style 160×160 is typical). In the WASM
Playground a Canvas program runs in the **Canvas tab** — frames are shown on a
``, keyboard/pointer feed the input, and sound ([`Audio`](#40-audio)) plays through WebAudio.
Natively a build **opens a real desktop window** on macOS, Linux and Windows,
using vendored static raylib + SDL3, the same backend the
`Scene` namespace links. Building it on Linux needs SDL3's documented build
dependencies present; SDL3's configure fails
outright when the X11 or audio headers it probes for are missing, so a machine
without them should configure with `-DCULEBRA_ENABLE_CANVAS_WINDOW=OFF`. Windows
needs no such packages — the mingw-w64 toolchain already has the headers SDL3's
Win32 backends want. The resulting binary still runs anywhere: SDL3 loads
X11/GL/audio (`opengl32.dll` and friends on Windows) on first use, so a window
build adds no load-time library dependency and starts fine on a headless
server. Each `present` uploads the frame, upscales it with
nearest-neighbour to a comfortable window size, and blocks to vsync at 60 fps;
the keyboard and mouse feed `Canvas.buttons`/`Canvas.mouse`, and closing the
window (its close box, or a script's own `Canvas.quit()`) ends the `run`
loop — Esc is an ordinary key here (see `Canvas.key`), not a built-in quit.
**Headless is declared, never inferred**:
in a build without the window backend — or in any run with
`CULEBRA_CANVAS_HEADLESS` set to anything but `0`/`off` — the backend is
**headless**: the pixel and sprite ops run identically (so behaviour is the same
across VM / JIT / AOT and testable via `Canvas.get_pixel`), but nothing
is displayed and input reads as "no button". That variable is
how a displayless server — or the test suite, since every `just` recipe exports
it — runs a window-capable binary; `-DCULEBRA_ENABLE_CANVAS_WINDOW=OFF` goes
further and leaves raylib out of the build entirely. A window build that
declared neither and cannot open a window (no display, no usable GL) raises a
`RuntimeError` at the first `present` naming the variable, rather than
guessing that a silent headless run is what was wanted.
### Color
`Canvas.rgba(r, g, b, a = 255) -> Long` packs four 0–255 channels into one
`Long` (byte order `[r, g, b, a]`). Every drawing call takes such a color,
and **the alpha composites**: 255 draws opaque, 0 draws nothing, and anything
between blends the shape over what is already there (integer source-over —
`(src*a + dst*(255-a) + 127) / 255` per color channel, so all three backends
round identically, and an opaque buffer stays opaque). `rgba(0, 0, 0, 128)`
is a half-dark overlay; drawing it twice darkens twice. The two exceptions
are `clear`, which replaces every pixel with the given value (a frame reset,
not a wash), and `set_pixel`, which stores the value raw so it pairs with
`get_pixel` (writing transparency is what those two are for).
`Canvas.rgb_to_hsv(r, g, b) -> Tuple` and `Canvas.hsv_to_rgb(h, s, v) -> Tuple`
convert between the two color models — the RGB side in the same 0–255
channels as `rgba`, the HSV side each of hue/saturation/value in `0.0..1.0`.
HSV is where a palette gets *derived*: boosting saturation, narrowing a
light/dark pair toward each other, or shifting hue are all one-line
expressions in HSV that have no simple RGB equivalent. `Canvas.hsv(h, s, v, a
= 255) -> Long` packs the result directly, mirroring `rgba`.
```culebra
inspect(Canvas.rgb_to_hsv(255, 0, 0)) # => (0.0, 1.0, 1.0)
inspect(Canvas.hsv_to_rgb(0.0, 1.0, 1.0)) # => (255, 0, 0)
inspect(Canvas.hsv(0.0, 1.0, 1.0) == Canvas.rgba(255, 0, 0)) # => true
# saturate a base color by 40%, in HSV, then pack it
let (h, s, v) = Canvas.rgb_to_hsv(180, 140, 200)
inspect(Canvas.hsv(h, Math.min(1.0, s * 1.4), v)) # => 4291327148
```
The pair round-trips exactly on the 0–255 grid — `hsv_to_rgb` rounds each
channel, so feeding `rgb_to_hsv`'s result straight back into `hsv_to_rgb`
reproduces the original `r, g, b` for every input, not just a
visually-close approximation.
### Drawing
| Function | Effect |
| --- | --- |
| `Canvas.init(w, h)` | allocate (or resize) the framebuffer; `Canvas.run` does this for you |
| `Canvas.clear(color)` | replace the whole target with `color` (no compositing) |
| `Canvas.set_pixel(x, y, color)` | store one pixel raw (off-buffer writes are ignored) |
| `Canvas.get_pixel(x, y) -> Long` | read a pixel (0 off-buffer) — for pixel-readback collision |
| `Canvas.rect(x, y, w, h, color, fill = true)` | rectangle, clipped |
| `Canvas.line(x1, y1, x2, y2, color)` | line, both endpoints included |
| `Canvas.circle(cx, cy, r, color, fill = true)` | circle centred on `(cx, cy)` |
| `Canvas.ellipse(cx, cy, rx, ry, color, fill = true)` | ellipse with per-axis radii |
| `Canvas.triangle(x1, y1, x2, y2, x3, y3, color, fill = true)` | triangle |
| `Canvas.polygon(points, color, fill = true)` | polygon from a flat vertex list |
| `Canvas.width()` / `Canvas.height() -> Long` | current draw-target dimensions |
| `Canvas.to_png() -> String` | the current draw target's pixels as PNG bytes |
| `Canvas.present()` | show the frame (see the loop below) |
The framebuffer and the sprite registry belong to one isolate — the first one
to touch them, by drawing or by reading (`width`, `get_pixel`). A second
isolate is refused rather than raced: it finds an empty canvas (no pixels, no
sprite handles), and the calls that can report say why — `init` and
`draw_to`/`to_png` raise `RuntimeError`. A worker can own the canvas, as long
as no other isolate has touched it.
Every shape takes `fill: false` to draw its one-pixel outline instead of the
filled interior: for `rect` that is the outermost ring of the same fill, for
`circle` / `ellipse` the connected edge, and for `triangle` / `polygon` the
closed chain of `line`-drawn edges (whose vertices, unlike the half-open
fill, are included). `line` walks its major axis one pixel at a time with the
minor coordinate rounded to nearest, and sorts its endpoints first, so
`line(a, b)` and `line(b, a)` draw identical pixels. A circle of radius `r`
spans `2r + 1` pixels across its centre row — `(cx ± r, cy)` lies on the
circle — and a radius of 0 is a single pixel; negative radii draw nothing.
Every position and size argument is `Long|Float`. A `Float` is rounded toward
−∞ (pixel *n* covers `[n, n+1)`), so adjacent spans tile without a gap and a
negative coordinate stays off-buffer instead of snapping onto column 0.
Non-finite and out-of-range values saturate rather than trap, and `Math.nan`
reads as 0. This lets a program that computes positions in floating point — a
projection, a scroll offset — pass them straight in instead of rounding each
one itself. Colors, blit flags, alpha and sprite handles remain `Long`.
`Canvas.polygon` takes `points` as a flat `[x0, y0, x1, y1, …]` list of at
least three vertices — Longs or Floats, like every other coordinate — and
closes the outline itself; a trailing half-pair is ignored. It fills by the
even-odd rule, so a concave outline hollows out the way you would draw it.
`Canvas.triangle` is the same fill from three vertices spelled out, which is
what the conventional shape call looks like (raylib, SDL and GPU rasterizers
all take three) and avoids building an `Array` per call on a hot path.
`Canvas.circle` is `ellipse` with one radius; both compute each row's
half-width with an exact integer square root, so every backend rasterizes the
identical curve.
Rows and spans are half-open, the same convention as `rect`: a row belongs to
the edge whose vertical span contains it, and each filled span covers
`[xl, xr)`. Shapes sharing an edge therefore tile with no seam and no
double-drawn pixel — a rectangle cut along its diagonal into two triangles
rasterizes back to exactly the rectangle. The interpolation is integer
throughout, so every backend produces the identical shape, and coordinates
saturate into a ±2³⁰ guard band so no input can overflow it.
### The window
A window exists only in a desktop build that opened one — headless builds and
the browser have none, and the entry points below are no-ops there. `Canvas`
is a framebuffer first; this is the one part of it the OS owns.
| Function | Effect |
| --- | --- |
| `Canvas.title(name)` | name the window; call it before the loop starts |
| `Canvas.toggle_fullscreen()` | switch fullscreen on/off |
| `Canvas.fullscreen() -> Bool` | is the window fullscreen now |
| `Canvas.resizable(enabled)` | let the OS window be dragged larger/smaller |
| `Canvas.resized() -> Bool` | the window's pixel size changed this frame |
| `Canvas.show_cursor()` / `Canvas.hide_cursor()` | show / hide the OS mouse cursor |
| `Canvas.cursor_hidden() -> Bool` | is the cursor hidden now |
| `Canvas.clipboard() -> String` | read the OS clipboard |
| `Canvas.set_clipboard(text)` | write the OS clipboard |
| `Canvas.quit()` | close the window, ending `run`'s loop the way the close box does |
| `Canvas.can_quit() -> Bool` | is there a real window to close |
The browser is a deliberate no-op rather than a missing feature: the tab's
title belongs to the page hosting the canvas, not to the program drawing on it
— and the same holds for every other entry above, all of them no-ops in the
browser and headless where there is no OS window to affect.
`Canvas` windows are otherwise a fixed size; `resizable` only lets the OS
window itself be dragged, and the framebuffer's own logical resolution does
not follow along — `resized()` is the one-frame edge a script polls to react
itself (re-`init()` at a new size, reflow its own UI, or ignore the resize).
`quit()` raises the same signal `run`'s loop stops on as the window's own
close box, one frame later; `can_quit()` says whether there is a window to
close at all, so a game can decide whether an in-game "quit?" prompt makes
sense before ever offering one.
### Sprites
`Canvas.Sprite.new(pixels, w, h, palette = nil)` uploads a sprite once and
returns a handle to blit cheaply each frame. `pixels` is a flat, row-major
array: packed-RGBA `Long`s, or — when `palette` is given — indices into that
palette (compact indexed art). A fully transparent source pixel is skipped
(the shape mask), a partially transparent one — a PNG's anti-aliased edge —
blends over what is there, so sprites composite. A sprite frees its pixels
when the last reference to it goes away.
`Canvas.Sprite.new(png: String)` instead decodes PNG bytes — `FS.read` of an
image file, say, since a `String` is a byte string — and takes the size from
the image, so there is nothing to pass alongside the data. `Canvas.Sprite.from_png(data)` is the same thing under
a name that reads as one at the call site. Greyscale, palette (with `tRNS`),
truecolor and 16-bit-per-channel images all decode to the same packed-RGBA
layout the framebuffer uses; anything undecodable raises
`ValueError: not a valid PNG image`.
| Method | Effect |
| --- | --- |
| `sprite.draw(x, y, flip_x = false, flip_y = false, transpose = false)` | blit the whole sprite to `(x, y)`; `transpose` swaps X/Y (a diagonal reflection — combine with a flip for a 90° rotation) |
| `sprite.draw_sub(x, y, sx, sy, sw, sh, flip_x = false, flip_y = false, transpose = false)` | blit a sub-rectangle (for sprite sheets) |
| `sprite.draw_scaled(x, y, w, h, flip_x = false, flip_y = false, smooth = false, alpha = 255)` | blit into the `w`×`h` rectangle at `(x, y)`, resampling to fit |
| `sprite.draw_sub_scaled(x, y, w, h, sx, sy, sw, sh, flip_x = false, flip_y = false, smooth = false, alpha = 255)` | the same from a sub-rectangle |
| `sprite.to_png() -> String` | the sprite's pixels as PNG bytes — `from_png`'s inverse |
| `sprite.width()` / `sprite.height()` | sprite dimensions |
The scaling blits sample nearest-neighbour, so pixel art scaled up stays crisp;
`smooth` box-averages the source instead when the sprite shrinks (it is ignored
when neither axis shrinks). There is no scaled `transpose`. `alpha` (0–255)
scales the whole blit: each pixel composites at its own source alpha times
`alpha`, so an opaque sprite at `alpha: 128` is half-blended and a
half-transparent edge pixel under it blends at a quarter. A fully transparent
source pixel is skipped and never contributes, whether it is sampled directly
or averaged over. A destination or source rectangle with a non-positive side
draws nothing.
A sprite's or the framebuffer's side below 0 is 0. A size past what an image
holds — `w * h * 4` bytes of 2 GiB or more — raises `ValueError`
(`Canvas.Sprite: 24000x24000 is too large`, or `Canvas.init: …` from
`Canvas.init` / `Canvas.run`).
### Offscreen drawing
`Canvas.Sprite.blank(w, h, color = 0)` creates an empty sprite, and
`Canvas.draw_to(sprite, fn () { ... })` redirects every drawing call into it
for the duration of the closure — `clear`, the shapes, `text`, other
sprites' `draw`, plus `Canvas.width()` / `height()` and `get_pixel`, which
follow the target so centring code keeps working offscreen. `present()`
always shows the framebuffer. The previous target is restored on every exit
path (including a throw), and `draw_to` nests like the call stack it is.
```culebra
# doctest: skip
let bgd = Canvas.Sprite.blank(320, 240)
Canvas.draw_to(bgd, fn () {
# render the backdrop once…
Canvas.clear(sky)
for i in 0..50 {
Canvas.circle(Random.below(320), Random.below(240), 2, star)
}
})
Canvas.run(320, 240, fn () {
bgd.draw(0, 0) # …then each frame is one blit
true
})
```
Two things refuse with a `ValueError`: drawing a sprite onto itself (a blit
would read its own writes), and freeing the sprite currently drawn to. Both
are backend-symmetric, like every Canvas error.
### Saving an image
`sprite.to_png()` and `Canvas.to_png()` answer PNG bytes, so writing one out
is `FS.write` of the result and reading it back is the `from_png` that already
existed. `Canvas.to_png()` follows the **current draw target**, the way
`width` / `height` / `get_pixel` do — inside a `draw_to` it encodes the sprite
being drawn into, outside it encodes the framebuffer.
```culebra
# doctest: skip — writes files beside it
Canvas.init(320, 240)
Canvas.clear(Canvas.rgba(24, 24, 32))
Canvas.circle(160, 120, 40, Canvas.rgba(240, 180, 90))
FS.write("shot.png", Canvas.to_png()) # a screenshot
let tile = Canvas.Sprite.blank(16, 16) # or render one offscreen
Canvas.draw_to(tile, fn () {
Canvas.clear(Canvas.rgba(80, 200, 120))
})
FS.write("tile.png", tile.to_png())
```
Output is 8-bit truecolor with alpha, one `IDAT`, each row filtered by the
choice that scores smallest — so flat, dithered pixel art compresses close to
what a dedicated encoder gets. An image with no pixels (`Canvas.init(0, 0)`)
and a sprite handle that has been freed both raise `ValueError`.
### Text
`Canvas.text(s, x, y, color, scale = 1)` draws `s` in the built-in 8×8 bitmap
font (the WASM-4 runtime font, covering printable ASCII 32–126, upper- and
lower-case; characters outside that range are skipped). Each font pixel
becomes a `scale`×`scale` block and the advance follows at `8 * scale` px per
character — `scale: 2` is the title-screen size. A non-positive scale draws
nothing. `Canvas.text_width(s, scale = 1) -> Long` gives the pixel width, for
centring or right-aligning.
### TTF fonts
`Canvas.Font(data)` parses TTF/OTF bytes — `FS.read` of a font file, or
`Dir.embedded(...).read(...)` to bake it into an AOT binary the same way
`Sprite.from_png` assets already do — and returns a handle. Unlike `Sprite`,
a `Font` has no fixed size: size is a parameter of each draw call, so one
`Font` serves every size a program uses, and rasterized glyphs are cached
internally per (font, codepoint, size). Anything stb_truetype can't parse
raises `ValueError: not a valid TTF/OTF font`. **Only load fonts from
trusted sources** — like most TTF/OTF parsers, stb_truetype does not
range-check a malformed file's internal offsets.
| Method | Effect |
| --- | --- |
| `font.draw(s, x, y, color, size)` | draw `s` at `(x, y)` (visual top-left) in `color` at `size` px |
| `font.draw_screen(s, x, y, color, size)` | the same, but drawn at the resolution the frame is presented at |
| `font.text_width(s, size) -> Long` | pixel width `s` will occupy at `size` — for right-aligning / centring |
| `font.ascent(size) -> Long` | pixel ascent above the baseline at `size` |
| `font.advance(codepoint, size) -> Long` | pixel advance of one codepoint at `size` |
`draw` walks every Unicode scalar value in `s` (full Unicode, unlike the
built-in bitmap font's ASCII-only `text`), rendering each with
antialiasing — a glyph's edge composites through the same alpha-blend every
other draw call uses, so a partially-covered pixel blends toward `color`
rather than snapping fully on or off. A codepoint outside the font falls back
to the font's own `.notdef` glyph rather than being skipped. There is no
kerning: advance is the sum of each glyph's own width.
There is no hinting either — a deliberate choice, not a gap: hinting would
make glyph shapes depend on the hinting engine's own rounding, and
`Canvas.Font` targets the same pixel-for-pixel result across native, the
browser and headless that every other Canvas primitive does.
```culebra
# doctest: skip
let font = Canvas.Font(FS.read("assets/roboto.ttf"))
Canvas.run(320, 240, fn () {
Canvas.clear(Canvas.rgba(20, 20, 24))
font.draw("Score: 42", 8, 8, Canvas.rgba(255, 255, 255), 16)
true
})
```
#### Text at the display's resolution
A frame is presented by scaling the framebuffer up with nearest-neighbour
pixels: right for sprites and the bitmap font, but it magnifies a glyph's
antialiased edge into blocks. **`font.draw_screen`** takes the same arguments
in the same coordinates as `draw`, and the text it draws lands where `draw`
would put it, but it is rasterized at the size the frame is actually shown at
and composited over the frame instead of into it — so it stays sharp at any
window size or Playground pane width. A call switches between the two by name
alone; `text_width`, `ascent` and `advance` describe both.
Two things follow from the screen layer being separate from the framebuffer:
- **It is cleared every frame.** The framebuffer persists across `present()`,
so drawing into it can accumulate; screen text has to be drawn each frame,
which a `Canvas.run` tick does anyway. (It has to work this way: the glyphs
are antialiased, and compositing them over themselves frame after frame
would darken every edge.)
- **`Canvas.to_png()` does not capture it.** `to_png` reads the draw target,
and the screen layer is not one — nor is it affected by `Canvas.draw_to`,
which redirects framebuffer drawing into a sprite. Use `draw` for text that
has to appear in a saved image.
A native window rasterizes at the display's real pixel density, not the
window's nominal size: on a 2x (Retina/HiDPI) screen a 640x380 window is
1280x760 actual pixels, and that is what the text is drawn at. The Playground
does the same, from the width the pane is displayed at times the browser's
device pixel ratio. Where there is no display to scale into — headless, or a
window that could not open — the scale is 1 and `draw_screen` lands exactly
where `draw` would.
### Input
Input is polled each frame (it reflects the current state, not an event queue).
| Function | Result |
| --- | --- |
| `Canvas.buttons() -> Long` | bitmask of held buttons |
| `Canvas.mouse() -> Object` | `{x, y, buttons}` in framebuffer pixels |
| `Canvas.key(name) -> Bool` | one named key is held now |
| `Canvas.key_queue() -> Array` | drain this frame's key presses (names) |
| `Canvas.typed() -> String` | drain the characters the user typed |
| `Canvas.wheel() -> Float` | vertical mouse wheel delta since the last frame |
Button bits are the constants `Canvas.LEFT`, `RIGHT`, `UP`, `DOWN` (arrow keys,
and WASD as a second set) and `Canvas.A`, `B` (action keys — `A` is Space/Z,
`B` is X). For edge
detection, `Canvas.Input.new()` tracks the previous frame:
| Method | Result |
| --- | --- |
| `input.update()` | sample this frame's buttons (call once per frame) |
| `input.down(btn) -> Bool` | button is held now |
| `input.pressed(btn) -> Bool` | button went down **this** frame (the flap trigger) |
Beyond the six buttons, `Canvas.key` reports any key by name — **the same
vocabulary `Term.read_key` uses**, so key handling code moves between the two
namespaces unchanged: a printable character (`"a"`, `" "`, `"-"`) or a
special-key name (`"up"` / `"down"` / `"left"` / `"right"`, `"enter"`,
`"escape"`, `"tab"`, `"backspace"`, `"insert"`, `"delete"`, `"home"`,
`"end"`, `"pageup"`, `"pagedown"`, `"f1"`…`"f12"`). `"space"` is accepted as
a readable alias for `" "`; an unknown name is simply never held. Letters
name the physical key regardless of Shift (`"a"` covers both cases).
`Canvas.key_queue()` returns the keys pressed since it was last called — the
edge events, for bindings that must not repeat while held — and
`Canvas.typed()` returns the characters typed (Shift, layout and IME
applied), which is what a name-entry screen reads. Both drain destructively
and are capped at 256 entries (oldest first out), so call each from one place
per frame and hand the result around. Natively, the first `typed()` call
turns the platform's text input on — a program that only polls keys never
sees an IME popup. Headless, nothing is held and the queues are empty.
Positive `wheel()` is away from the user (scroll up / zoom in), matching the
browser's `-deltaY` convention.
### Gamepad
| Function | Result |
| --- | --- |
| `Canvas.pad_available(index = 0) -> Bool` | a gamepad is connected at `index` |
| `Canvas.pad_axis(axis, index = 0) -> Float` | axis value (sticks, triggers) |
| `Canvas.pad_held(button, index = 0) -> Bool` | button is held now |
| `Canvas.pad_pressed(button, index = 0) -> Bool` | button went down **this** frame |
| `Canvas.pad_name(index = 0) -> String` | the pad's reported name |
| `Canvas.pad_rumble(left, right, sec, index = 0)` | vibrate both motors 0–1 strength for `sec` seconds |
| `Canvas.pad_mappings(db) -> Bool` | load extra `SDL_GameControllerDB` lines for a pad the bundled DB lacks |
`index` (0–3) picks which pad — 0 for the common single-controller case,
others for local multiplayer. `pad_pressed` is a same-frame edge the way
`input.pressed` is for `buttons()`, except raylib tracks it natively, so
there is no `update()`/previous-state bookkeeping to do. Button and axis
numbers are raylib's own `GamepadButton`/`GamepadAxis` values, named so a
script never has to know them:
| Buttons | Axes |
| --- | --- |
| `Canvas.PAD_UP` / `RIGHT` / `DOWN` / `LEFT` (d-pad) | `Canvas.AXIS_LX` / `LY` (left stick) |
| `Canvas.PAD_Y` / `B` / `A` / `X` (face buttons, Xbox naming) | `Canvas.AXIS_RX` / `RY` (right stick) |
| `Canvas.PAD_LB` / `LT` / `RB` / `RT` (shoulders/triggers) | `Canvas.AXIS_LT` / `RT` (analog triggers) |
| `Canvas.PAD_SELECT` / `GUIDE` / `START` | |
| `Canvas.PAD_L3` / `R3` (stick pressed in) | |
`pad_rumble` silently does nothing on a backend/pad without haptics (Xbox
pads on macOS have no API driving them). Gamepad state is unavailable — and
`pad_available` is always `false` — in the browser and headless.
### Collision
Axis-aligned overlap and containment tests, plain culebra with no native
backend of their own:
| Function | Result |
| --- | --- |
| `Canvas.rect_overlap(x1, y1, w1, h1, x2, y2, w2, h2) -> Bool` | two rectangles overlap |
| `Canvas.circle_overlap(x1, y1, r1, x2, y2, r2) -> Bool` | two circles overlap |
| `Canvas.point_in_rect(px, py, x, y, w, h) -> Bool` | a point is inside a rectangle |
| `Canvas.point_in_circle(px, py, cx, cy, r) -> Bool` | a point is inside a circle |
`rect`/`(x, y, w, h)` arguments match `Canvas.rect`'s own convention
(top-left corner and size), so a sprite's hitbox and its drawn rectangle
share the same numbers.
### Audio
Canvas carries no audio of its own: sound is the [`Audio`](#40-audio)
namespace, which a Canvas program uses as any other program does.
### The game loop
`Canvas.run(w, h, tick, frames = 600)` sets up a `w`×`h` framebuffer and calls
`tick()` once per frame, presenting after each. `tick` returns `false` to stop
(e.g. the player quit). In the interactive Playground build `present()` waits
for the browser's next animation frame, so the loop paces itself and yields
cooperatively. Otherwise the loop stops after `frames`, so a run nobody can end
can't spin forever. Staying unbounded takes both halves: the frames must reach a
display (not a build without the window backend, not `CULEBRA_CANVAS_HEADLESS`,
not a machine with no display) *and* the run must be interactive (not a piped
native run, not a non-JSPI browser). A headless run from a terminal meets only
the second — it shows nothing, takes no input and has no close box — so it is
capped like any other automated run.
```culebra
# doctest: skip — opens a window
let red = Canvas.rgba(220, 60, 60)
mut x = 0
Canvas.run(160, 160, fn () {
Canvas.clear(Canvas.rgba(20, 24, 40))
Canvas.rect(x, 76, 8, 8, red)
x = (x + 2) % 160
true
})
```
`tick`'s cadence is vsynced to a target frame rate — 60 by default, matching
the browser loop — so most games never touch the functions below:
| Function | Effect |
| --- | --- |
| `Canvas.dt() -> Float` | seconds since the previous `present()` |
| `Canvas.target_fps(n)` | cap the frame rate (0 uncaps it) |
| `Canvas.fps() -> Long` | the measured frame rate |
`dt()` only matters to a `tick` that wants to step by real elapsed seconds
instead of the fixed `1/60` a vsynced tick otherwise assumes (this repo's own
`lunar_lander.cul` example does). All three are native-only: the browser
build paces itself to the display's own animation frame and headless has no
frame rate at all, so there `dt()`/`fps()` read `0` and `target_fps` is a
no-op.
---
## 28. `Scene`
A retained-mode renderer for 3D built from procedural geometry: you populate a
scene graph of nodes — primitives (boxes, spheres, cylinders, planes) and
hand-built meshes — give them materials and transforms, place a camera, and
render. The lighting is physically based (metallic/roughness materials, a
directional sun with two-cascade shadows, sky/fog, and a post stack of SSAA,
ambient occlusion, bloom, and depth of field), so the output is more than
flat-shaded primitives.
`Scene` is **not a game engine**. It has no physics or collision, no model
import (geometry is procedural or built vertex-by-vertex; textures are drawn
in-process or decoded from PNG bytes), and no skeletal animation.
It targets 3D you *construct* — visualisations, procedural scenes, vehicle or
flight demos with a chase camera — rather than asset-driven games. A
racing demo — circuit mesh, chase camera, gamepad steering — is the
shape it is designed around.
`Scene` is in every build that has the `Canvas` window backend — the default
on macOS, Linux and Windows, so the released binaries carry it — on the same
vendored static SDL3 + raylib; `-DCULEBRA_ENABLE_SCENE=OFF` leaves it out. A
`Scene` program built with `culebra build` links raylib and SDL3 statically
and comes out at ~5.1 MB on macOS arm64 (a `print` alone is ~0.4 MB).
Linux is where a frame is drawn on every push (under Xvfb,
`tests/scene_api_test.sh` calling every method on both engines) and macOS is
where it is run by hand, while the Windows build is linked but has yet to open
a window. A `View` has no headless mode, so — unlike `Canvas` — `Scene`
programs run neither headless nor in the Playground; `Scene.Image`, the
CPU-side image baker, is the one part that needs no window.
### The view and the frame loop
`Scene.View.new(w, h, title)` opens a window, and raises a `RuntimeError` when
it cannot (no usable display/GL) — unlike `Canvas` there is no headless mode to
fall back to, because everything a `View` does needs the GPU. Positions and
sizes are `Float`
world units; colors are three or four `0–255` integer channels, and a channel
outside that range clamps to it. A frame is
either a 3D pass with a 2D overlay (`render_3d()` → overlay draws → `present()`)
or pure 2D (`begin_2d()` → draws → `present()`).
| Method | Effect |
| --- | --- |
| `view.target_fps(fps)` | cap the frame rate |
| `view.closing() -> Bool` | the close box was clicked, or `quit()` was called (loop until true) |
| `view.quit()` | end the loop from the script: `closing()` is true from now on |
| `view.dt() -> Float` | seconds since the previous frame |
| `view.width()` / `view.height() -> Float` | window size |
| `view.camera(px,py,pz, tx,ty,tz, ux,uy,uz, fov)` | eye position, look-at target, up vector, vertical FOV |
| `view.render_3d()` | render the scene graph, then open the frame for a 2D overlay |
| `view.begin_2d()` | open a pure-2D frame (no 3D pass) |
| `view.present()` | finish and show the frame |
| `view.fullscreen(on)` / `view.is_fullscreen() -> Bool` | switch fullscreen on or off / is it on now |
| `view.resizable(on)` / `view.resized() -> Bool` | let the user drag the edges / did the size change this frame |
| `view.size(w, h)` / `view.title(s)` | set the window's size / its title |
| `view.vsync(on)` | wait for the display's refresh at `present()` |
| `view.cursor(on)` / `view.mouse_capture(on)` | show or hide the pointer / lock it to the window for mouse-look |
| `view.clipboard() -> String` / `view.set_clipboard(s)` | the system clipboard |
| `view.fps() -> Long` / `view.time() -> Float` | measured frame rate / seconds since the window opened |
| `view.supersample(n)` | render at `n`× the window and downsample (1–4; the default 2 is the antialiasing) |
| `view.clip_planes(near, far)` | the 3D pass's clip planes in metres (default 2 and 8000; a cockpit camera pulls `near` in) |
The 3D targets follow the window: once a resize or a fullscreen toggle has been
noticed (at the `present()` after it), frames render at the new resolution
rather than being stretched from the old.
### Scene graph
`view.add_node()` adds an empty node; the `add_*` helpers add geometry and
return the new node. Nodes nest (`node.add_node()`, `node.add_box()`, …) and the
transform methods are fluent, so a subtree builds in one expression. Build
persistent geometry once and move it each frame.
| Method | Effect |
| --- | --- |
| `view.add_box(w, h, d)` / `add_sphere(r)` / `add_cylinder(r, h)` / `add_plane(w, d)` | add a primitive node (a `node` adds the same shapes as a child) |
| `view.add_mesh()` / `node.add_mesh()` | add an empty custom mesh (below) |
| `node.move(x, y, z)` | set position |
| `node.yaw(a)` / `pitch(a)` / `roll(a)` | rotate about one axis (radians) |
| `node.spin(x, y, z, a)` / `euler(x, y, z)` | axis-angle / Euler rotation |
| `node.scale(s)` / `scale3(x, y, z)` | uniform / per-axis scale |
| `node.tint(r, g, b)` | per-node color |
| `node.material(m)` | assign a `Material` (below); `nil` goes back to the tint alone |
| `node.order(n)` | draw order: lower draws first (default 0; a HUD plate in the world sits at a high one) |
| `node.opacity(a)` | 0–255, multiplied into the material's |
| `node.quat(x, y, z, w)` | orientation as a quaternion — what an integrator hands over; replaces the euler angles (`spin` still composes) |
| `node.billboard(on = true)` | face the camera every frame (a sprite in the world, a distant tree card) |
| `node.hide()` / `show()` / `name(n)` | visibility / label |
| `node.x()` / `y()` / `z() -> Float` | read back position |
| `node.world_x()` / `world_y()` / `world_z() -> Float` | position after every ancestor's transform |
| `node.child_count() -> Long` / `node.child_at(i) -> Node` | the children (out of range is a `RuntimeError`) |
| `node.find(name) -> Node` / `node.has(name) -> Bool` | a named descendant (`find` of a name nothing carries is a `RuntimeError`; `has` is the test) |
| `node.remove()` | detach from the parent — the handle stays valid, the node draws nowhere until re-added |
| `node.vertex_count() -> Long` | a custom mesh's vertices, pushed or uploaded — to start a new node before the cap |
| `node.cull_radius(r)` | the bounding sphere culling uses (0 = from the shape) |
| `node.culling(on)` | take this node out of frustum culling, for a shape whose bound is wrong |
| `view.remove(node)` / `view.find(name) -> Node` / `view.has(name) -> Bool` | the same over the whole scene, roots included |
| `view.culling(on)` | skip nodes whose bounding sphere is off screen (default on; it never changes the picture) |
A custom mesh is built from vertices and triangles, then finalised:
`m.vertex(x, y, z, nx, ny, nz)` (or `vertex_uv(…, u, v)`) adds a vertex,
`m.tri(a, b, c)` a triangle by vertex index, and `m.build()` uploads it. (raylib
uses a 16-bit index buffer, so a mesh caps at 65535 vertices; `build()` rejects
more.) Texture coordinates put `(0, 0)` at an image's top-left, as in 2D; the
primitives show an image upright, its top toward `+y` (a plane's toward `-z`,
the far edge from a camera on `+z`). An uploaded mesh belongs to the view that
uploaded it: a node kept past `view.drop()` stays usable as a transform, but
draws nothing in a later view. `build()` needs an open view to upload into,
so it refuses with no view at all and the vertices stay for a later one.
### Materials, lighting, textures
A material is a handle the view creates and the script configures with fluent
setters, so one reads as a single expression:
| Method | Result |
| --- | --- |
| `view.add_material() -> Material` | a new material: white, matte, untextured |
| `mat.rgb(r, g, b) -> Material` | base colour |
| `mat.pbr(metallic, roughness) -> Material` | PBR response, both 0–1 (the defaults are 0 and 0.85) |
| `mat.texture(tex) -> Material` | a `Texture` to sample, or `nil` for none |
| `mat.uv(us, vs, uo = 0.0, vo = 0.0) -> Material` | scale and offset the texture coordinates (tiling a road; `-1.0` in `us` mirrors) |
| `mat.normal_map(tex, strength = 1.0) -> Material` | a tangent-space normal map (`Scene.Image.to_normal` makes one); the tangent frame is derived per pixel, so a mesh needs no tangents; `nil` removes it |
| `mat.opacity(a) -> Material` | 0–255; below 255 the surface is transparent |
| `mat.cutout(threshold) -> Material` | discard where coverage (opacity × texture alpha) is below it (a leaf card); 0 turns it off |
| `mat.blend(name) -> Material` | how a transparent surface meets what is behind it: `"over"` (default), `"add"` (lamps, glows), `"multiply"`, `"screen"` |
| `mat.emissive(r, g, b, k = 1.0) -> Material` | light the surface gives off, added after lighting (it feeds the bloom) |
| `mat.unlit(on = true) -> Material` | the base colour as is — no light, shadow or reflection (a mirror image, a decal) |
| `mat.double_sided(on = true) -> Material` | draw both faces (a flag, a fence) |
| `mat.depth_write(on)` / `mat.depth_test(on) -> Material` | the depth buffer: whether the surface writes it / is hidden by it (both default on) |
| `mat.casts_shadow(on) -> Material` | in the shadow passes (default on; off for a blob shadow or a mirror plate) |
| `mat.fog(on) -> Material` | whether distance fog reaches it (default on; off for a sky dome) |
```culebra
# doctest: skip
let gold = view.add_material().rgb(230, 180, 60).pbr(0.9, 0.3)
view.add_box(2.0, 2.0, 2.0).material(gold)
let glass = view.add_material().rgb(200, 220, 255).opacity(90).depth_write(false).casts_shadow(false)
let shadow = view.add_material().rgb(0, 0, 0).opacity(110).unlit().depth_write(false).casts_shadow(false)
```
A surface is transparent when its opacity (node × material) is below 255 or
its blend is not `"over"`. The lit pass draws every opaque surface first,
front to back, then the transparent ones back to front; `node.order()` sorts
ahead of both, so a plate that must sit on top of everything (a rear-view
mirror's frame and image, its LED strip) takes a high order and draws last
whatever its distance. Transparent surfaces blend their colour but leave the
frame's depth as the opaque pass wrote it, so ambient occlusion and depth of
field see through glass to the solid geometry behind — the same reading a
blob shadow or a windscreen wants. A transparent surface casts no shadow
unless it is a cutout, whose shape is real.
### Post-processing
`render_3d()` runs the frame through a post pass: depth of field, ambient
occlusion, bloom, tonemap, saturation, and — when asked — a vignette and colour
grading. Each has a knob; the defaults are the look the shader was tuned with,
and `0` turns a pass off.
| Method | Effect |
| --- | --- |
| `view.post_process(on)` | the whole pass (off: the lit frame as is, antialiasing included) |
| `view.exposure(k)` | tonemap exposure (default 1.35) |
| `view.saturation(k)` | 1 = as lit (default 1.1) |
| `view.bloom(threshold, strength)` | glow bleeding from what is brighter than `threshold` (defaults 0.7, 1.5) |
| `view.dof(strength, range)` | depth of field around the screen centre's depth (defaults 0.85, 3.5) |
| `view.ssao(strength, radius)` | ambient occlusion (defaults 0.45, 3.0) |
| `view.vignette(k)` | darken the corners (default 0) |
| `view.lut(tex, amount = 1.0)` | colour grading through a 3D LUT, blended in by `amount`; `nil` turns it off |
A LUT is a `Texture` holding a horizontal strip of `n` slices, each `n`×`n`,
`n` being the texture's height (a 4096×64 strip is 64 slices): blue selects
the slice, red runs across it, green down it. Build one with `Scene.Image` and
upload it with mipmaps and repeat off (`view.texture(img, false, false)`); the
pass samples a cell as a cell. The identity LUT — every cell its own colour —
leaves a frame alone, and a graded one is what a colour tool exports.
### A second camera
`view.render_to(tex, px,py,pz, tx,ty,tz, ux,uy,uz, fov)` renders the scene
from another camera into a `render_target` texture — the same lit pass as
`render_3d()`, without the post stack — so a rear-view mirror is a plate
textured with the target, its material given `uv(-1.0, 1.0, 1.0, 0.0)` to
mirror the image. Call it before `render_3d()` each frame: the plate that
samples the texture is drawn during `render_3d()`, so a later call would show
the previous frame. The shadows in it are the ones the last `render_3d()`
fitted around the main camera, which a mirror looking back along the same
road shares; the first frame's mirror is unshadowed. A render target, like a
canvas, is upright wherever it is sampled — on a mesh, as a `sprite`, through
`uv()`.
```culebra
# doctest: skip
let mirror = view.render_target(512, 256)
let mirror_mat = view.add_material().texture(mirror).unlit().uv(-1.0, 1.0, 1.0, 0.0)
view.add_box(1.2, 0.4, 0.02).move(0.0, 1.6, 2.0).material(mirror_mat).order(10000)
while !view.closing() {
view.camera(0.0, 1.5, 6.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 55.0)
view.render_to(mirror, 0.0, 1.5, 1.0, 0.0, 1.5, 20.0, 0.0, 1.0, 0.0, 120.0)
view.render_3d()
view.present()
}
```
A texture is a handle too. The two handle types are the contract: a texture
where a material is expected (or the reverse) is a `TypeError` at the call, and
a dropped handle a `ClosedError`.
| Method | Result |
| --- | --- |
| `view.texture(img, mipmaps = true, repeat = true) -> Texture` | upload a `Scene.Image` (below); mipmapped and repeating for a tiled material, neither for a sprite or a LUT |
| `view.texture_png(bytes) -> Texture` | a PNG's bytes (`FS.read`, or a `Dir.embedded` asset) straight to a texture |
| `view.checker(px, checks, r1,g1,b1, r2,g2,b2) -> Texture` | a checkerboard |
| `view.grain(px, r, g, b, amt) -> Texture` | a flat colour with noise grain |
| `view.canvas(w, h) -> Texture` / `view.canvas_end()` | open a render-to-texture the 2D calls paint into, and close it |
| `view.render_target(w, h) -> Texture` | a texture the scene renders into from a second camera (below) |
| `tex.width()` / `tex.height() -> Float` | size in pixels |
| `tex.filter(name) -> Texture` | sampling: `"point"` (pixel art, a LUT's exact cells), `"bilinear"`, `"trilinear"` |
| `tex.wrap(name) -> Texture` | past the edge: `"repeat"`, `"clamp"`, `"mirror"` |
The sizes `checker`, `grain`, `canvas` and `render_target` take follow the
image rule below: a negative side, or `w * h * 4` bytes of 2 GiB or more, is a
`ValueError`.
A canvas is how a texture gets drawn with the GPU (`rect`/`text`/… between
`canvas()` and `canvas_end()`), the way a racing demo paints liveries and
signage. One is open at a time: a second `canvas()` before the close is a
`RuntimeError`, and a frame opened with one still open ends it first. The
texture is upright on the mesh whichever way its UVs run. A texture or material
outlives the view that made it as an inert handle: in a later view it samples
as white, the way a node's mesh draws nothing there.
### Images
`Scene.Image` is a CPU image — the baker for the procedural textures a scene is
dressed with (a livery, a road's grain, a lamp's glow, a colour-grading LUT)
and the way a PNG's pixels come in. It needs no window: an image can be built,
read back pixel by pixel, or written out before any `View` exists, which is
what makes it the one part of `Scene` a test runs without a display. Always
RGBA, coordinates in pixels, every colour with its own alpha (default 255).
`Scene.Image.new(w, h)` and `img.resize(w, h)` raise `ValueError` for a
negative side, and for a size past what an image holds — `w * h * 4` bytes of
2 GiB or more (`Scene.Image.new: 24000x24000 is too large`); so do
`img.rotate(degrees)` when the rotated image would be past it, and `img.text`
when the text drawn at its `size` would be.
| Method | Result |
| --- | --- |
| `Scene.Image.new(w, h) -> Image` | a transparent image |
| `Scene.Image.from_png(bytes) -> Image` | decode a PNG (`RuntimeError` when it is not one) |
| `img.width()` / `img.height() -> Float` | size |
| `img.get(x, y) -> Long` | one pixel, packed `0xRRGGBBAA` |
| `img.copy() -> Image` / `img.save_png(path) -> Bool` / `img.to_png() -> String` | a copy / write a PNG / the PNG's bytes |
| `img.fill(r, g, b, a = 255)` | every pixel |
| `img.pixel(x, y, r, g, b, a = 255)` | one pixel |
| `img.rect(x, y, w, h, r, g, b, a = 255)` / `img.rect_line(x, y, w, h, r, g, b, a = 255)` | filled / outlined rectangle |
| `img.circle(x, y, radius, r, g, b, a = 255)` / `img.circle_line(x, y, radius, r, g, b, a = 255)` | filled / outlined circle |
| `img.line(x0, y0, x1, y1, thick, r, g, b, a = 255)` | line |
| `img.triangle(x0, y0, x1, y1, x2, y2, r, g, b, a = 255)` | filled triangle |
| `img.text(s, x, y, size, r, g, b, a = 255, font = nil, spacing = 0.0)` | text in a `Font`; the built-in font (`nil`) exists only once a window does |
| `img.gradient(r1,g1,b1, r2,g2,b2, direction = 0)` | a linear gradient over the whole image; `direction` in degrees, 0 top → bottom, 90 left → right |
| `img.gradient_radial(density, r1,g1,b1, r2,g2,b2)` | centre → rim, `density` (0–1) how far out the inner colour holds |
| `img.noise(seed, scale, amount = 255)` | Perlin noise mixed in at `amount` (0–255); `scale` is the feature size |
| `img.cellular(tile, amount = 255)` | Worley cells of `tile` pixels mixed in (gravel, stone) |
| `img.blit(src, x, y, r = 255, g = 255, b = 255, a = 255)` | another image onto this one, tinted |
| `img.blit_rot(src, x, y, rot = 0.0, scale = 1.0)` | the same, turned and scaled about its centre at `(x, y)` |
| `img.blur(radius)` / `img.tint(r, g, b)` / `img.invert()` / `img.grayscale()` / `img.brightness(k)` | whole-image passes |
| `img.flip_v()` / `img.flip_h()` / `img.rotate(degrees)` / `img.resize(w, h)` / `img.crop(x, y, w, h)` | geometry |
| `img.to_normal(strength = 1.0)` | height (the red channel) to a tangent-space normal map, +Y up, edges wrapping so a tile's normals tile |
The drawing calls are fluent, so a texture is one expression:
```culebra
# doctest: skip
let asphalt = Scene.Image.new(256, 256).fill(60, 60, 64).noise(3, 4.0, 90).cellular(24, 40)
let road = view.add_material().texture(view.texture(asphalt)).pbr(0.0, 0.9)
```
`img.get` is exact — the CPU raster is the same bytes on every platform — so a
test can assert a pixel, where a rendered frame can only be compared loosely.
Lighting is set on the view:
| Method | Effect |
| --- | --- |
| `view.background(r, g, b)` | clear color |
| `view.sky(tr,tg,tb, br,bg,bb)` | zenith → horizon gradient (also the reflected environment) |
| `view.sun(dx,dy,dz, intensity, r,g,b)` | directional light (two-cascade shadows); `(0, 0, 0)` names no direction and is refused |
| `view.ambient(intensity, r, g, b)` | fill light |
| `view.fog(start, end, r, g, b)` | distance fog |
| `view.screenshot(path)` | save the frame being drawn to a PNG — call it after `render_3d()` / `begin_2d()` and before `present()`; afterwards the shown buffer is gone and the shot would be a frame or two stale |
### 2D overlay
After `render_3d()` (or `begin_2d()`) these draw on top, for a HUD — and into
an open `canvas()`, which is how a texture gets drawn. Coordinates are window
points, angles are degrees, and every call takes the shared `alpha()`.
| Method | Effect |
| --- | --- |
| `view.alpha(a)` | opacity (0–255) for the 2D draws that follow |
| `view.text(s, x, y, size, r, g, b, font = nil, spacing = 0.0, rot = 0.0)` | draw text in a `Font`, or in the built-in bitmap font when `font` is `nil`; `rot` turns it about `(x, y)` |
| `view.text_width(s, size, font = nil, spacing = 0.0) -> Float` / `view.text_height(s, size, font = nil, spacing = 0.0) -> Float` | the box `text` would cover, for aligning |
| `view.rect(x, y, w, h, r, g, b)` / `view.rect_line(x, y, w, h, thick, r, g, b)` | filled / outlined rectangle |
| `view.rect_round(x, y, w, h, roundness, r, g, b)` / `view.rect_round_line(x, y, w, h, roundness, thick, r, g, b)` | rounded corners (`roundness` 0–1) |
| `view.rect_gradient(x, y, w, h, r1,g1,b1, r2,g2,b2, horizontal = false)` | top → bottom (or left → right) gradient |
| `view.circle(x, y, radius, r, g, b)` / `view.circle_line(x, y, radius, r, g, b)` | filled / outlined circle |
| `view.circle_gradient(x, y, radius, r1,g1,b1, r2,g2,b2)` | centre → rim gradient (a lamp) |
| `view.ring(x, y, r_in, r_out, a0, a1, r, g, b)` | an arc band between two radii, from `a0` to `a1` (a gauge) |
| `view.line(x0, y0, x1, y1, thick, r, g, b)` | line |
| `view.triangle(x0, y0, x1, y1, x2, y2, r, g, b)` | filled triangle, either winding |
| `view.poly(x, y, sides, radius, rot, r, g, b)` | regular polygon |
| `view.sprite(tex, x, y, w, h, rot = 0.0, ox = 0.0, oy = 0.0, r = 255, g = 255, b = 255)` | a `Texture` into the `w`×`h` box at `(x, y)`, turned about `(ox, oy)` within it, tinted (white = as is) |
| `view.sprite_rec(tex, sx, sy, sw, sh, x, y, w, h, rot = 0.0, ox = 0.0, oy = 0.0)` | a sub-rectangle of a texture (an atlas) |
| `view.clip(x, y, w, h)` / `view.clip_end()` | clip the draws between them to a rectangle |
A path is built by scalar push — `view.path_begin()`, then `view.path_to(x, y)`
per vertex, `view.path_close()` to return to the first — and drawn by one of
`view.path_fill(r, g, b)` (a fan from the first vertex, so convex shapes),
`view.path_strip(r, g, b)` (a ribbon of left/right pairs: a minimap's track),
`view.path_stroke(thick, r, g, b)` (the outline) or `view.path_spline(thick, r,
g, b)` (a Catmull-Rom curve through four or more points). The points stay until
the next `path_begin()`, so a shape can be filled and then outlined.
### Fonts
`view.font(path, size, chars = "") -> Font` rasterizes a TTF/OTF at one pixel
size into a glyph atlas, and `view.font_bytes(data, size, chars = "") -> Font`
does the same from the file's bytes — a `Dir.embedded` asset, so a one-binary
game ships its font inside itself. `chars` names the glyphs to include (`""`
is printable ASCII): a HUD lists its digits and words and gets a small atlas,
one with Japanese lists the characters it uses. A font that cannot be loaded
is a `RuntimeError`, not a silent fall back to the built-in font with the
wrong metrics. The atlas must fit an 8192-pixel square, counting each glyph as
a square of `size`: past that — about 750 for printable ASCII, about 160 for
2000 characters — the call is a `ValueError` (`view.font: 2000 glyphs at size
300 is too large`) before anything is rasterized; list fewer characters or
use a smaller size. `font.size() -> Float` and `font.glyphs() -> Long` read back
what was made. Like a texture, a font belongs to the view that made it.
### Input
Input is polled from the view each frame. Keys are named — **the same
vocabulary `Canvas.key` and `Term.read_key` use**: a printable character
(`"a"`, `" "`, `"-"`) or a special-key name (`"up"` / `"down"` / `"left"` /
`"right"`, `"enter"`, `"escape"`, `"tab"`, `"backspace"`, `"insert"`,
`"delete"`, `"home"`, `"end"`, `"pageup"`, `"pagedown"`, `"f1"`…`"f12"`;
`"space"` is a readable alias for `" "`). An unknown name is a key that is
never pressed. Esc is an ordinary key here, not a way out of the window: the
loop ends at the close box or at `view.quit()`.
Gamepad buttons and axes are named too, by the layout SDL's mapping DB
normalizes every pad to: face buttons `"a"` `"b"` `"x"` `"y"` (the Xbox letters
— `"a"` is Cross on a PlayStation pad), d-pad `"up"` `"down"` `"left"`
`"right"`, shoulders `"lb"` `"rb"`, triggers `"lt"` `"rt"`, `"select"`
`"guide"` `"start"`, stick clicks `"l3"` `"r3"`; axes `"lx"` `"ly"` `"rx"`
`"ry"` `"lt"` `"rt"`. `index` picks the pad (0–3) and defaults to the first.
The mouse reports in window points; its buttons are `"left"`, `"right"` and
`"middle"`.
| Method | Result |
| --- | --- |
| `view.key(name) -> Bool` | key is held now |
| `view.key_pressed(name) -> Bool` / `key_released(name) -> Bool` | key went down / came up this frame |
| `view.pad_available(index = 0) -> Bool` | a gamepad is connected |
| `view.pad_axis(name, index = 0) -> Float` | axis value (sticks, triggers) |
| `view.pad(name, index = 0) -> Bool` / `pad_pressed(name, index = 0) -> Bool` | button held / just pressed |
| `view.rumble(left, right, sec, index = 0)` | haptics (Sony pads and XInput; Xbox-on-macOS is silent) |
| `view.pad_name(index = 0) -> String` / `view.gamepad_mappings(db)` | pad identity / load an SDL mapping DB |
| `view.mouse_x()` / `mouse_y() -> Float` | pointer position |
| `view.mouse_dx()` / `mouse_dy() -> Float` | pointer motion since the last frame (also while captured) |
| `view.mouse_wheel() -> Float` | wheel delta this frame |
| `view.mouse(button) -> Bool` / `mouse_pressed(button) -> Bool` | mouse button held / just pressed |
### Audio
Scene carries no audio of its own: sound is the [`Audio`](#40-audio)
namespace, which a Scene program uses as any other program does.
### A minimal scene
```culebra
# doctest: skip — opens a window
let view = Scene.View.new(960, 540, "spinner")
view.target_fps(60)
view.background(30, 34, 42)
view.sun(0.5, -0.8, -0.3, 1.2, 255, 245, 230)
view.ambient(0.4, 180, 200, 220)
let gold = view.add_material().rgb(230, 180, 60).pbr(0.9, 0.3)
let box = view.add_box(2.0, 2.0, 2.0).material(gold)
mut a = 0.0
while !view.closing() {
a += view.dt()
box.yaw(a)
view.camera(4.0, 3.0, 5.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 55.0)
view.render_3d()
view.text("culebra scene", 20.0, 20.0, 28, 235, 235, 240)
view.present()
if view.key_pressed("escape") { view.quit() }
}
view.drop()
```
---
## 29. `Net`
Raw TCP and UDP sockets, plus name resolution — the layer under
[§15 Http](#15-http). Blocking, with an optional per-socket timeout: the same
threads-not-async model as `Http`, so a server that must not block the main
thread runs inside an [`Isolate`](#12-isolate).
A transport failure — refused connect, unresolvable host, reset peer, timeout —
raises `NetError`. The message is the OS's own wording, except a timeout, which
always reads `timed out`. A blocked `read` / `accept` / `recv_from` stays
interruptible: one Ctrl+C raises `Interrupted` rather than hanging.
Every entry that takes a port (`connect`, `listen`, `udp`, `send_to`) accepts
0 to 65535 and nothing else. A port outside that range raises `ValueError`
(`Net.listen: port must be between 0 and 65535, got 70000`) before any socket
is opened.
Socket handles are **not Sendable** — a socket belongs to the thread that opened
it. To serve from another thread, open it inside that isolate.
### `Net.connect(host: String, port: Long, timeout: Long = 0) -> Socket`
Connect to `host:port`. `timeout` is in milliseconds (`0` = wait forever) and
bounds the connect, then becomes the socket's read/write timeout.
```culebra
# doctest: skip — reaches the network
let s = Net.connect("example.com", 80, timeout: 5000)
s.write("GET / HTTP/1.0\r\nHost: example.com\r\n\r\n")
s.shutdown_write() # tell the server the request is complete
inspect(s.read()) # read until the server closes
s.close()
```
**The `Socket` handle** — the reader/writer shape of [§4 File](#4-file), so
code that only reads works over either:
| Method | Effect |
| --- | --- |
| `read(n = nil)` | up to `n` bytes (a short read is normal on a socket); `nil` reads until the peer closes. `""` at EOF |
| `read_line()` | one line, terminator stripped; `nil` once the stream ends |
| `read_exact(n)` | exactly `n` bytes; a peer that closes early is a `NetError`, not a short read |
| `lines()` | line iterator, ending when the peer closes. Unlike a file's `f.lines()` it does **not** close the socket (you usually reply afterwards) |
| `write(data)` | write every byte of `data` |
| `shutdown_write()` | half-close: signal EOF to the peer while still reading its reply |
| `local_addr()` / `peer_addr()` | `{host, port}` of this end / the other end |
| `set_timeout(ms)` | read/write timeout in ms; `0` waits forever |
| `set_nodelay(on = true)` | disable Nagle's algorithm (send small writes immediately) |
| `is_open()` / `close()` | liveness; `close` is idempotent (the GC also closes one that goes out of scope) |
A line ends at `\n` only, and a trailing `\r` is stripped — so CRLF protocols
read cleanly. (A file's `f.lines()` also splits on a lone `\r`; a socket cannot, because
that would need a one-byte lookahead that can block when a CRLF arrives split
across two packets.)
### `Net.listen(port: Long, host: String = "0.0.0.0", backlog: Long = 0) -> Listener`
Bind and listen. `port: 0` asks the OS for a free port, readable back as
`listener.port` — the reliable way to avoid a fixed-port collision in tests.
```culebra
# doctest: skip — listens on a port until interrupted
let server = Net.listen(7000)
println("listening on " + server.port.to_string())
for conn in server {
# accept in a loop
conn.write("hello " + conn.peer_addr().host + "\n")
conn.close()
}
```
| Member | Effect |
| --- | --- |
| `port` / `host` | the address actually bound (an ephemeral port shows up here) |
| `accept()` | block until a connection arrives, returning a `Socket` |
| `for conn in listener` | accept in a loop; the body `break`s to stop |
| `serve(handler, workers = 0)` | accept in a loop and run `handler(conn)` on a worker pool; blocks until interrupted |
| `set_timeout(ms)` | bound how long `accept` waits before raising; accepted sockets inherit it |
| `is_open()` / `close()` | as on `Socket` |
`accept` and `for conn in listener` are sequential — the handling runs on the
accepting thread, so one slow connection blocks the next.
### `listener.serve(handler, workers = 0)` — the concurrent form
`serve` keeps accepting while handlers run on a pool of worker threads, so a
slow connection can't stall the accept loop:
```culebra
# doctest: skip — listens on a port until interrupted
let server = Net.listen(7000)
server.serve(fn (conn) {
for line in conn.lines() {
conn.write(line.upper() + "\n")
}
}, workers: 8) # blocks until Ctrl+C
```
- `workers: 0` (the default) picks a CPU-scaled pool (at least 4, at most 8);
a positive count fixes it. Requests are handled in **true parallel** — each
worker has its own runtime, with no global lock.
- Because each worker rebuilds the handler on its own heap, the handler must be
**Sendable**: it can't capture mutable variables or non-Sendable values. A
non-Sendable handler is a `SendError` raised by `serve` itself, not on the
first connection. Share read-only data with [`Shared.new`](#12-isolate) and
open per-connection resources inside the handler.
- The connection is closed when the handler returns. A handler that raises
closes **only its own** connection — the worker and the server keep going
(there is no response to turn into a `500`, unlike `Http.server`).
- `serve` blocks; Ctrl+C (or dropping the isolate it runs in) stops the accept
loop, waits for the in-flight handlers, and raises `Interrupted`. To serve
while the main thread works, run it inside an [`Isolate`](#12-isolate).
For HTTP specifically, prefer [`Http.server`](#httpserver---object) — it carries
routing, static files, streaming and WebSocket over the same worker model.
### `Net.udp(port: Long = 0, host: String = "0.0.0.0") -> UdpSocket`
Open a datagram socket bound to `host:port` (`port: 0` = ephemeral).
```culebra
# doctest: skip
let sock = Net.udp(9000)
sock.set_timeout(2000)
let msg = sock.recv_from() # {data, host, port}
sock.send_to("ack", msg.host, msg.port)
```
| Member | Effect |
| --- | --- |
| `port` / `host` | the bound address |
| `send_to(data, host, port)` | send one datagram (whole or not at all — no partial write) |
| `recv_from(max = 65536)` | receive one datagram as `{data, host, port}`; an oversized datagram is truncated to `max`, as UDP dictates |
| `set_broadcast(on = true)` | allow sending to a broadcast address |
| `set_timeout(ms)` / `is_open()` / `close()` | as on `Socket` |
UDP has no EOF and no connection: an empty datagram is data, and a peer that
went away is silence, not an error.
### `Net.resolve(host: String) -> Array`
The numeric addresses `host` resolves to, in resolver order, deduplicated. A
numeric address resolves to itself; a name that doesn't resolve is a `NetError`.
```culebra
# doctest: skip
inspect(Net.resolve("localhost")) # => ["127.0.0.1", "::1"]
```
### Not available in the Playground
The browser has no raw sockets, so every `Net` call in a WebAssembly build
raises `NetError: networking is not available in this build`.
---
## 30. `Desktop` / `Webview`
A desktop GUI written in web tech: a local HTTP server supplies the UI, a
**native WebView** window displays it, and the whole thing ships as one
binary. `Webview` wraps the OS's own engine (WKWebView on macOS, WebKitGTK
on Linux, WebView2 on Windows) — no bundled browser. Both namespaces are
in the default build (`-DCULEBRA_ENABLE_WEBVIEW=OFF` opts out; they
self-disable when the engine's headers are absent — the GTK4 / WebKitGTK dev
packages on Linux, `WebView2.h` on Windows), and `culebra build` links the
WebView frameworks only for programs that actually reference them.
Those headers are a build-time requirement only. The binary loads its engine
when a window is created — through the vendored header's own loader on Windows,
through `dlopen` on Linux — so it starts on a machine that has none, and a
program that never opens a window runs there unaffected. Asking for a window
without an engine raises `webview: failed to create window`.
### How the pieces fit together
Three different things share this one feature, and it's easy to conflate
them:
| Component | What it is |
| --- | --- |
| `Desktop` | culebra's one-call facade — starts an HTTP server, opens a window pointed at it, blocks until the window closes, then stops the server |
| `Webview.Window` | the raw native window binding — one OS window, backed by the platform's WebView engine, that can navigate to a URL or load raw HTML |
| `window` | the standard JS **global object** — it exists inside the web page the native window is displaying. Plain JavaScript, not a culebra API |
`Desktop.run` is built on `Webview.Window`: it starts a local HTTP server,
then creates a window and navigates it to that server's loopback URL
(`http://127.0.0.1:PORT`). From there, the native window and the page it
renders talk to each other over two independent channels:
- **Loopback HTTP.** The page's JS can call
`fetch('/__quit', {method: 'POST'})` against the server `Desktop.run`
started (the route is registered for you), and culebra code on the server
side can call `Desktop.quit()` / `Webview.Window.quit()` directly. This is
how server-side or page-side code tells the *native window* to close.
- **Two properties on the page's `window`.** Before honoring the frame's own
close button, the native side checks whether the page's `window` defines
`__culebra_before_close__` — that function alone then decides whether and
when to actually close, by calling `window.__culebra_close__()` once ready.
This is how the *page* tells the *native window* it's safe to close, e.g.
after a confirmation dialog. See [The page's `window`
object](#the-pages-window-object) below.
In short: `Desktop` orchestrates, `Webview.Window` is the native window it
drives, and `window` is the ordinary browser global inside whatever page
that window happens to be showing — culebra never creates or owns it, it
only looks for two conventionally-named properties on it.
### `Desktop.run(config: Object) -> Nil`
The one-call facade: start the server, open the window on it, block until
the window closes, then stop the server.
| Key | Default | Meaning |
| --- | --- | --- |
| `title` | `'culebra'` | window title |
| `size` | window default | `[width, height]` in pixels |
| `assets` | — | static root served at `/`: any `Dir` ([Serving a Dir](#serving-a-dir)), typically `Dir.embedded('dist')`, which reads from disk in dev and is baked into the binary under `culebra build` |
| `routes` | — | `fn (srv) { ... }` to register the app's own routes on the `Http` server (§15) |
| `port` | `8731`, then OS-assigned | loopback port the server binds. Unset: try `8731`; if it's taken (say, another culebra desktop app), fall back to an OS-assigned free port — note the page's origin, and so its `localStorage`, changes with the port. Set explicitly: bind exactly that port or fail |
| `workers` | `4` | server worker threads |
A `POST /__quit` route is registered for you, so the page can close the
app (`fetch('/__quit', {method: 'POST'})`); `Desktop.quit()` does the same
from culebra code.
```culebra
# doctest: skip — opens a window
Desktop.run({title: 'culebra desktop', size: [
720,
560,
], assets: Dir.embedded('dist'), routes: fn (srv) {
srv.get('/api/hello', fn (req) {
{content_type: 'application/json', body: JSON.stringify({message: 'hello'})}
})
}})
```
### `Webview.Window` — the raw binding
Use it directly when you don't want a server at all (inline HTML, or a
remote URL).
| Method | Effect |
| --- | --- |
| `Webview.Window.new()` | create a window |
| `w.set_title(title)` | set the title |
| `w.set_size(width, height)` | resize |
| `w.set_html(html)` | load an HTML string |
| `w.navigate(url)` | load a URL |
| `w.run()` | run the native event loop; blocks until terminated |
| `w.terminate()` | end this window's `run()` |
| `Webview.Window.quit()` | terminate whichever window is currently in `run()` — callable from another thread, e.g. an HTTP handler |
| `Webview.Window.is_running()` | `true` while a window is pumping its event loop. Not needed before `quit()` — a quit that arrives earlier is held — but lets another thread wait for the loop to be up |
```culebra
# doctest: skip — opens a window
let w = Webview.Window.new()
w.set_title('hello')
w.set_size(480, 320)
w.set_html('hi from culebra ')
w.run()
```
**When the window becomes visible.** On macOS the window stays transparent
until the page puts its first frame on screen, so an app opens on its own
content instead of flashing an empty white rectangle first. A page that never
reports a frame — JavaScript turned off, a navigation that never completes —
gets the window shown anyway after 1.5 s, blank rather than invisible. Windows
and Linux show the window as soon as `set_size` runs, so the empty frame is
still briefly visible there.
### The page's `window` object
The native window and the page it displays coordinate a close through two
conventionally-named properties culebra looks for on the page's own
`window` — the standard JS global object, not anything culebra defines or
owns. Nothing needs importing on the page side; culebra simply reads and
calls whatever the page happens to put there.
Clicking the frame's own close button does not close the window outright:
1. Before closing, the native side checks whether the page defined
`window.__culebra_before_close__`.
2. If it did, that function alone now decides whether and when to actually
close — show a confirmation dialog, run an unsaved-changes check,
whatever the page needs — and calls `window.__culebra_close__()` once
it's ready to proceed. That's the same function an in-page "Quit" button
can call directly, whether or not the app uses `Desktop.run`.
3. A page that never defines `__culebra_before_close__` keeps closing
immediately, exactly as if this hook didn't exist.
```js
// in the page's own JS — not culebra code
window.__culebra_before_close__ = () => {
if (confirm('Discard unsaved changes?')) {
window.__culebra_close__()
}
// returning without calling __culebra_close__() cancels the close
}
```
---
## 31. `Vector2`
A minimal 2D float vector for graphics/game code — an ordinary culebra
class, not a language builtin. Elements are
always `Float`: `new` accepts `Long | Float` and coerces, so
`Vector2.new(1, 0)` works, but `.x` / `.y` are never `Long`.
`Vector2` also stands in for a "Point" — position vs. direction is not
distinguished, matching Unity / Godot / three.js rather than CGAL /
nalgebra's stricter split, and matching the rest of the standard library's
preference for the fewest types that cover the need. Distinct from the
`@packable class` pattern used for `SharedBuffer`'s fixed-layout records
(§12) — that's a raw byte-layout descriptor with no operators; `Vector2` is
the general-purpose math type.
```culebra
let a = Vector2.new(3, 4)
inspect(a.length()) # => 5.0
inspect(a.normalized()) # => (0.6, 0.8)
inspect(a + Vector2.new(1, 1)) # => (4.0, 5.0)
# Equal vectors are one Set member and one Object key.
inspect(Set(Vector2.new(1, 2), Vector2.new(1, 2)).size()) # => 1
```
| Member | Returns |
|---|---|
| `Vector2.new(x, y)` | `Vector2` |
| `a + b` / `a - b` | `Vector2` (component-wise) |
| `a * k` / `k * a` | `Vector2` (scalar; `k: Long \| Float`) |
| `-a` | `Vector2` |
| `a == b` | `Bool` — nominal: a non-`Vector2` (including a same-shaped `Vector3`) is always `false`, never a thrown error |
| `a.hash()` | `Long` — equal vectors hash equally, so a `Vector2` can be a `Set` member or an Object key |
| `a.dot(other)` | `Float` |
| `a.length()` / `a.length_squared()` | `Float` |
| `a.normalized()` | `Vector2` (unit length; a zero vector raises `ZeroDivisionError`, same as any `Float / 0.0`) |
| `a.distance_to(other)` | `Float` |
| `a.distance_squared_to(other)` | `Float` — skips the square root, for ranking or thresholding distances |
| `"{a}"` / `to_string(a)` | `String` — `"(x, y)"` |
`cross()` is intentionally not provided (on either `Vector2` or `Vector3`
— see §32): a 2D cross product returns a scalar (the perp dot product)
while a 3D cross product returns a vector, and no example in this codebase
needed one.
## 32. `Vector3`
The 3D counterpart of `Vector2` (§31) — same design (Float-only, no
separate "Point" type, nominal `==`), same member set, with `x`, `y`, `z`
fields and `Vector3.new(x, y, z)`.
```culebra
let a = Vector3.new(1, 2, 3)
inspect(a.length()) # => 3.7416573867739413
inspect(a + Vector3.new(1, 1, 1)) # => (2.0, 3.0, 4.0)
```
| Member | Returns |
|---|---|
| `Vector3.new(x, y, z)` | `Vector3` |
| `a + b` / `a - b` | `Vector3` (component-wise) |
| `a * k` / `k * a` | `Vector3` (scalar; `k: Long \| Float`) |
| `-a` | `Vector3` |
| `a == b` | `Bool` — nominal, same as `Vector2` |
| `a.hash()` | `Long` — same as `Vector2` |
| `a.dot(other)` | `Float` |
| `a.length()` / `a.length_squared()` | `Float` |
| `a.normalized()` | `Vector3` (unit length; a zero vector raises `ZeroDivisionError`) |
| `a.distance_to(other)` | `Float` |
| `a.distance_squared_to(other)` | `Float` — skips the square root, for ranking or thresholding distances |
| `"{a}"` / `to_string(a)` | `String` — `"(x, y, z)"` |
## 33. `Deque`
A double-ended queue — an ordinary culebra class
, not a language builtin. Backed by a
growable ring buffer (array + head index + count), so `push`/`pop` at
either end are O(1) amortized. `Array` (language spec §18) only ever
grows or shrinks at its end (`push`/`pop`); a FIFO queue built on
`Array` needs `remove_at(0)`, which shifts every remaining element —
O(n) per dequeue. Reach for `Deque` whenever the front matters: a FIFO
queue (`push_back` + `pop_front`), a sliding window, or a stack where
"which end" should stay explicit (`push_back` + `pop_back`, mirroring
`Array.push`/`pop`).
```culebra
let q = Deque.new()
q.push_back(1)
q.push_back(2)
q.push_front(0)
inspect(q.pop_front()) # => 0
inspect(q.pop_front()) # => 1
inspect(q.size()) # => 1
```
| Member | Returns |
|---|---|
| `Deque.new()` | `Deque` |
| `d.push_back(x)` | `Nil` |
| `d.push_front(x)` | `Nil` |
| `d.pop_back()` | `Any` — removed element, `nil` if empty |
| `d.pop_front()` | `Any` — removed element, `nil` if empty |
| `d.peek_back()` | `Any` — `nil` if empty, does not remove |
| `d.peek_front()` | `Any` — `nil` if empty, does not remove |
| `d.size()` | `Long` |
| `d.empty()` | `Bool` |
| `d.to_array()` | `Array` — snapshot, front to back |
| `d.iter()` | `Iterator` — so `for x in d { ... }` works, front to back |
| `"{d}"` / `to_string(d)` | `String` |
Like `Array.pop()`, `pop_front`/`pop_back`/`peek_front`/`peek_back`
return `nil` on an empty `Deque` rather than throwing — the same
ambiguity `Array.pop()` already accepts (a pushed `nil` and an empty
`Deque` are indistinguishable from the return value alone).
## 34. `PriorityQueue`
A binary min-heap over an `Array` — an ordinary culebra class
, not a language builtin.
`push`/`pop` are O(log n); the naive alternative — an `Array` kept
sorted, or scanned for the minimum on every pop — is O(n log n) or
O(n) per operation.
Ordering follows the same convention as `Array.sort`/`sort_by`
(language spec §18): elements compare with `<` (so a class's
`__lt__` is honored) unless `key:` is given, and `reverse: true` flips
it into a max-heap.
```culebra
let pq = PriorityQueue.new()
pq.push(5)
pq.push(1)
pq.push(3)
inspect(pq.pop()) # => 1
inspect(pq.peek()) # => 3
let jobs = PriorityQueue.new(key: |j| j[0])
jobs.push((2, 'retry'))
jobs.push((1, 'ping'))
inspect(jobs.pop()) # => (1, 'ping')
```
| Member | Returns |
|---|---|
| `PriorityQueue.new(*, key: Function \| Nil = nil, reverse: Bool = false)` | `PriorityQueue` |
| `pq.push(x)` | `Nil` |
| `pq.pop()` | `Any` — removes and returns the minimum (maximum if `reverse: true`), `nil` if empty |
| `pq.peek()` | `Any` — like `pop()` but does not remove, `nil` if empty |
| `pq.size()` | `Long` |
| `pq.empty()` | `Bool` |
| `"{pq}"` / `to_string(pq)` | `String` — `"PriorityQueue(size=n)"`, not the heap array (its element order is not priority order) |
Like `Array.pop()` and `Deque`, `pop`/`peek` return `nil` on an empty
queue rather than throwing.
## 35. `PEG`
A parser generator: you write a **PEG** (parsing expression grammar) and parse
with it. The engine is the vendored
[cpp-peglib](https://github.com/yhirose/cpp-peglib) — the same one culebra's own
front end runs on, so a grammar that `peglint` accepts behaves identically here.
The API is cpp-peglib's too: `PEG.parser` is its `peg::parser`, with the same
methods, the same defaults and the same callbacks
([where it differs](#where-it-differs-from-cpp-peglib) is listed at the end).
```culebra
let calc = `
Additive <- Multiplicative '+' Additive / Multiplicative
Multiplicative <- Primary '*' Multiplicative / Primary
Primary <- '(' Additive ')' / Number
Number <- < [0-9]+ >
%whitespace <- [ \t\r\n]*
`
let parser = PEG.parser(calc)
parser.enable_ast()
let (ok, ast) = parser.parse("1 + 2")
inspect(ok) # => true
inspect(ast.name) # => 'Additive'
inspect(parser.parse("1 +")) # => (false, nil)
```
**Write grammars as backtick raw strings** (`` `...` ``): they do no escape
processing and no `{...}` interpolation, so `\t`, `[0-9]{2}` and `'` all reach
the grammar verbatim. A single-quoted raw string works too, but a PEG is full of
`'literal'` quoting, so the backtick form is usually the one that fits.
### Grammar notation
The full notation is [cpp-peglib's](https://github.com/yhirose/cpp-peglib#syntax);
the parts you need most:
| Form | Meaning |
| --- | --- |
| `Name <- e` | define a rule |
| `'text'` / `"text"` | literal |
| `[a-z]` | character class; `.` is any character |
| `e1 e2` | sequence — `e1` then `e2` |
| `e1 / e2` | prioritized choice — try `e1`, and only if it fails try `e2` |
| `e*` `e+` `e?` | repetition (greedy, no backtracking into it) |
| `&e` `!e` | lookahead: succeed / fail without consuming |
| `< e >` | token boundary — the node keeps the matched text |
| `%whitespace <- e` | what to skip between tokens |
| `~e` | drop this element from the tree |
| `{ no_ast_opt }` | keep this rule's node even when it has one child |
| `{ ast_name: Tag }` | emit the node under `Tag` instead of the rule name |
A PEG is unambiguous by construction: `/` is ordered, so the first alternative
that matches wins and there is no ambiguity to resolve. That also means the
order matters — `'a' / 'ab'` never matches `ab`.
### The parser object
| Making one | Result |
| --- | --- |
| `PEG.parser()` | a parser with no grammar yet |
| `PEG.parser(grammar)` | a parser of `grammar`; when the grammar is malformed, one that parses nothing |
| `PEG.parser(grammar, start)` | start from rule `start` instead of the grammar's first |
| `p.load_grammar(grammar)` | `Bool` — whether the grammar loaded; the logger is told why not |
| `p.load_grammar(grammar, start)` | same, with a start rule |
| Method | Result |
| --- | --- |
| `p.parse(text)` | `(ok, val)` — `(false, nil)` for a subject that does not parse |
| `p.parse(text, path)` | same, naming the subject for the callbacks (`vs.path`, `c.path`) |
| `p.parse(text, path, dt)` | same, with the [user data](#user-data) the callbacks are handed |
| `p.enable_ast()` | the parser; `val` is from now on [the tree](#the-tree) |
| `p.enable_ast(collapse_mode, opt_mode)` | same, with single-child nodes folded away as the tree is built |
| `p.optimize_ast(ast)` | the tree with single-child nodes folded away |
| `p.optimize_ast(ast, opt_mode)` | same; `opt_mode` `false` folds only the `no_ast_opt` rules |
| `p.enable_packrat_parsing()` | `Nil` — turns [memoization](#packrat-parsing) on |
| `p.set_logger(f)` | `f(line, col, msg)` or `f(line, col, msg, rule)` is told each error |
| `p[rule]` | the rule, whose `action`, `predicate`, `enter` and `leave` are assignable |
| `p[rule] = f` | `p[rule].action = f` |
| Tree helper | Result |
| --- | --- |
| `PEG.walk(node)` | `Iterator` — the node and its descendants, depth-first, in source order |
| `PEG.find(node, name)` | the first node named `name`, or `nil` |
| `PEG.find_all(node, name)` | `[Node]` — every node named `name` |
| `PEG.ast_to_s(node)` | `String` — an indented dump, the shape `peglint --ast` prints |
`parse` answers `(ok, val)`: cpp-peglib's `bool parse(s, val)`, with its out
parameter as one more returned value. What `val` is depends on what the parser
was asked for:
| The parser has | `val` |
| --- | --- |
| an [action](#semantic-actions) on any rule | the start rule's value |
| `enable_ast()` and no action | the tree |
| neither | `nil` |
The same steps in both:
| culebra | cpp-peglib |
| --- | --- |
| `let p = PEG.parser(grammar)` | `peg::parser parser(grammar);` |
| `let ok = p.load_grammar(grammar)` | `bool ok = parser.load_grammar(grammar);` |
| `let (ok, val) = p.parse(s)` | `bool ok = parser.parse(s, val);` |
| `let (ok, val) = p.parse(s, dt: d)` | `bool ok = parser.parse(s, d, val);` |
| `p.enable_ast()` / `p.optimize_ast(ast)` | `parser.enable_ast()` / `parser.optimize_ast(ast)` |
| `p.enable_packrat_parsing()` | `parser.enable_packrat_parsing()` |
| `p.set_logger(f)` | `parser.set_logger(f)` |
| `p["R"] = f` (the same as `p["R"].action = f`) | `parser["R"] = f` |
| `p["R"].predicate = f` | `parser["R"].predicate = f` |
| `p["R"].enter = f` / `.leave = f` | `parser["R"].enter = f` / `.leave = f` |
| assigning `nil` | assigning an empty function |
| `PEG.ast_to_s(ast)` | `peg::ast_to_s(ast)` |
What is set on a grammar goes with it. `load_grammar` starts the parser over
with the new grammar: the callbacks, `enable_ast()` and
`enable_packrat_parsing()` are gone, and only the logger stays. Called on a
parser with no grammar, the two `enable_` methods do nothing.
A loaded grammar is kept per thread, so a second parser of the same grammar
costs a lookup. A parser crossing an isolate boundary carries the grammar text
and loads it again on the other side; nothing is shared. The callbacks
assigned to its rules cross as any closure does, so one that captures a `mut`
binding stops the parser from crossing (`SendError`).
### The tree
After `enable_ast()` a parse hands back the tree. A node is a plain `Object`,
so `match` takes it apart with no helper API:
```culebra
let calc = `
Additive <- Multiplicative '+' Additive / Multiplicative
Multiplicative <- Primary '*' Multiplicative / Primary
Primary <- '(' Additive ')' / Number
Number <- < [0-9]+ >
%whitespace <- [ \t\r\n]*
`
fn eval(n) {
match n {
{name: "Number", token} => to_long(token),
{name: "Additive", nodes: [a, b]} => eval(a) + eval(b),
{name: "Multiplicative", nodes: [a, b]} => eval(a) * eval(b),
{nodes: [only]} => eval(only),
_ => throw "unexpected {n.name}",
}
}
let parser = PEG.parser(calc)
parser.enable_ast()
let (ok, ast) = parser.parse("(1 + 2) * 3")
inspect(eval(ast)) # => 9
```
| Field | Meaning |
| --- | --- |
| `name` | the rule that produced the node, or its `ast_name` override |
| `token` | a token node's matched text; `""` on a branch node |
| `nodes` | child nodes, in source order |
| `line` / `column` | 1-based position of the node's start in the subject |
| `position` / `length` | byte offset and byte length, so `s.slice(position, position + length)` is the node's whole text |
| `is_token` | whether the node came from a `< ... >` token boundary |
| `choice` | which alternative of a prioritized choice matched (0-based) |
There is no `parent` link: a tree of plain Objects with parent pointers would be
a reference cycle. Walk downward and carry what you need.
```culebra
let g = `
Doc <- Item+
Item <- < [a-z]+ > _
_ <- [ ]*
`
let parser = PEG.parser(g)
parser.enable_ast()
let (ok, doc) = parser.parse("ab cd")
inspect(PEG.find_all(doc, "Item").map(|n| n.token)) # => ['ab', 'cd']
inspect(PEG.find(doc, "Nothing")) # => nil
inspect(PEG.ast_to_s(doc))
# => |
# '+ Doc
# - Item (ab)
# - Item (cd)
# '
```
#### Optimizing the tree
The tree `enable_ast()` builds has a node for every rule that matched.
`optimize_ast` replaces a node that has exactly one child by that child, so
the tree carries structure rather than the grammar's bookkeeping:
```culebra
let g = `
Wrap <- Inner
Inner <- < [0-9]+ >
`
let parser = PEG.parser(g)
parser.enable_ast()
let (ok, ast) = parser.parse("1")
inspect(PEG.walk(ast).map(|n| n.name).collect())
# => ['Wrap', 'Inner']
inspect(PEG.walk(parser.optimize_ast(ast)).map(|n| n.name).collect())
# => ['Inner']
parser.enable_ast(true)
inspect(PEG.walk(parser.parse("1")[1]).map(|n| n.name).collect())
# => ['Inner']
```
`enable_ast(true)` builds the same tree `optimize_ast` would, without handing
the program the nodes that go. It is the form for large inputs: the full tree
is several times the size, and deep enough to pass the
[tree-depth bound](#errors-and-bounds) on input the optimized one fits.
A grammar keeps a rule's node with `{ no_ast_opt }`, and `{ ast_name: Tag }`
renames a node so two productions can converge on one tag. `opt_mode` `false`,
the second argument of both `optimize_ast` and `enable_ast`, turns the choice
round: only the nodes of the `no_ast_opt` rules go.
### Packrat parsing
Memoization is off until `enable_packrat_parsing()`, as in cpp-peglib. A PEG
backtracks, so alternatives that share a prefix — `A <- B '+' A / B`, the
shape a first grammar usually has — parse the same prefix once per alternative
and cost exponential time without it. Measured on the calculator above: ten
levels of nesting take 0.04 ms memoized and 2.0 s not.
```culebra
let parser = PEG.parser(`
Additive <- Multiplicative '+' Additive / Multiplicative
Multiplicative <- Primary '*' Multiplicative / Primary
Primary <- '(' Additive ')' / Number
Number <- < [0-9]+ >
`)
parser.enable_packrat_parsing()
inspect(parser.parse("((((((((((((((((1))))))))))))))))")) # => (true, nil)
```
The cost is a table proportional to subject length × rule count (roughly 1 KB
per input byte), which is why it is asked for. A grammar written without
shared prefixes (left-factored, or using `*`/`+` where a naive one recurses)
does not need it. It also changes how often a callback runs: a rule tried
again at a position it was tried at before reuses its earlier result instead
of running its callbacks again.
### Errors and bounds
A grammar or a subject that is wrong is not an error raised. `load_grammar`
and `parse` answer `false`, and why is told to the logger, once per error:
```culebra
let parser = PEG.parser()
let mut errors = []
parser.set_logger(|line, col, msg| errors.push("{line}:{col}: {msg}"))
inspect(parser.load_grammar("A <- B")) # => false
inspect(errors) # => ['1:6: 'B' is not defined.']
errors = []
inspect(parser.load_grammar(`N <- < [0-9]+ >`)) # => true
inspect(parser.parse("x")) # => (false, nil)
inspect(errors)
# => ['1:1: syntax error, unexpected 'x', expecting .']
```
Positions are inside the grammar or the subject. A logger of four parameters
is also handed the name of the rule the error belongs to, when it has one.
Without a logger a failure says nothing more than `false`; `set_logger(nil)`
takes one off. A grammar that loads can still be warned about (a rule nothing
refers to), and the logger is told that too.
`PEG.parser(grammar)` loads the grammar before a logger can be set, as
cpp-peglib's constructor does. To hear why a grammar did not load, make the
parser with no grammar, set the logger, and call `load_grammar`.
The logger is not told the subject's name. `path` goes to the callbacks, and a
program that wants it in its messages puts it there:
```culebra
let parser = PEG.parser(`N <- < [0-9]+ >`)
let path = "input.txt"
parser.set_logger(|line, col, msg| println("{path}:{line}:{col}: {msg}"))
parser.parse("x", path)
# => input.txt:1:1: syntax error, unexpected 'x', expecting .
```
What does raise: a callback assigned to a rule the grammar lacks (`PEGError`)
or a rule member that is not a `Function` (`TypeError`), at `parse`; an
argument of the wrong type (`TypeError`); a tree past the tree-depth bound
(`ValueError`); and whatever a callback throws.
Two bounds keep adversarial input from being fatal:
* **Rule nesting** — a parse that nests more than 4000 rule matches, or
builds a tree deeper than 4000 levels, fails with
`nesting too deep (limit 4000)`: a failed parse like any other, told to the
logger. Machine-written nesting would otherwise overflow the C stack inside
the parser. This is the same bound, at the same limit, that culebra applies
to its own grammar.
* **Tree depth** — a shallower tree still deeper than 1000 levels raises
`ValueError` (`nesting too deep (limit 1000)`), the
[value-nesting bound](language.md#the-value-nesting-bound) `JSON` and `TOML`
apply to their own trees.
### Semantic actions
An action is a `Function` assigned to a rule. With an action on any rule the
parse interprets the subject directly, without building a tree: each action
receives its rule's own reduction, a `SemanticValues`, and returns whatever
value that rule contributes to its parent. This is cpp-peglib's
[semantic action](https://github.com/yhirose/cpp-peglib#semantic-actions)
(`parser["Rule"] = [](const SemanticValues &vs) { ... }`):
```culebra
let parser = PEG.parser(`
Additive <- Multiplicative '+' Additive / Multiplicative
Multiplicative <- Primary '*' Multiplicative / Primary
Primary <- '(' Additive ')' / Number
Number <- < [0-9]+ >
%whitespace <- [ \t\r\n]*
`)
parser["Number"] = |vs| vs.token_to_number()
parser["Additive"] = |vs| vs.size() < 2 ? vs[0] : vs[0] + vs[1]
parser["Multiplicative"] = |vs| vs.size() < 2 ? vs[0] : vs[0] * vs[1]
inspect(parser.parse("1 + 2 * 3")) # => (true, 7)
inspect(parser.parse("1 +")) # => (false, nil)
```
An action of two parameters is also handed `dt`, the
[user data](#user-data) its parse was called with.
`vs` reads the way cpp-peglib's `SemanticValues` does:
| Member | Result |
| --- | --- |
| `vs[i]` / `vs.size()` / `vs.empty()` | the values of the rule's children; `vs[i]` is what child *i*'s action returned |
| `vs.transform(beg = 0, end = -1)` | `Array` — the values from `beg` up to `end`; all of them by default |
| `vs.iter()` | the values, one at a time (`for v in vs`) |
| `vs.token(id = 0)` | the text of the rule's `id`-th `< ... >`; the whole match for a rule with none |
| `vs.token_to_string(id = 0)` | the same text |
| `vs.token_to_number()` | the number the token starts with: a `Long`, or a `Float` when a fraction or an exponent follows; `0` when it starts with none |
| `vs.tokens` | `Array` — the text of each of the rule's `< ... >` |
| `vs.sv()` | the whole match, the whitespace skipped after it included |
| `vs.name()` | the rule |
| `vs.line_info()` | `(line, column)` of the match's start, 1-based |
| `vs.choice()` / `vs.choice_count()` | which alternative of a prioritized choice matched (0-based), and how many it has |
| `vs.path` | the subject's name (`parse`'s `path`) |
A rule's tokens are its own `< ... >`. A rule that only refers to a token rule
has none, so its `token()` is its whole match, with the whitespace the parser
skipped after it; wrapping the reference gives the rule a token of its own:
```culebra
let parser = PEG.parser(`
Pair <- Plain Boxed
Plain <- Word
Boxed <- < Word >
Word <- < [a-z]+ >
%whitespace <- [ ]*
`)
parser["Plain"] = |vs| vs.token()
parser["Boxed"] = |vs| vs.token()
parser["Pair"] = |vs| vs.transform()
inspect(parser.parse("ab cd ")) # => (true, ['ab ', 'cd'])
```
A rule with no action takes cpp-peglib's own default: the first child's value,
or `nil` for a rule with no children (a leaf token rule with no action gets no
value at all, the same as in cpp-peglib — write an action for any leaf whose
text you need). A rule can build anything, not just a scalar — an action's
return value is an ordinary culebra value, so one grammar can drive an
evaluator, a pretty-printer, or its own custom tree shape, by choosing what
each action returns.
The actions belong to the parser object, not to the grammar: another parser of
the same grammar carries none. Assigning `nil` takes an action off. With an
action on any rule the parse reduces, whether or not `enable_ast()` was
called.
An action on a rule the grammar lacks is caught when `parse` is called — a
typo'd rule name is not a silent no-op:
```culebra
let parser = PEG.parser(`N <- < [0-9]+ >`)
parser["Nope"] = |vs| vs
inspect(try {
parser.parse("1")
} catch e {
e.message
}) # => 'PEG: no such rule 'Nope''
```
Whatever an action itself throws — a culebra `throw`, a `TypeError` from
misusing `vs`, anything — propagates unchanged; there is no action-specific
catch to work around.
### User data
`dt` is cpp-peglib's user data: the value one `parse` call is given, handed to
every callback that parse runs. It comes where cpp-peglib puts it:
| Callback | Called as |
| --- | --- |
| action | `f(vs)`, `f(vs, dt)` or `f(vs, dt, predicate_data)` |
| predicate | `f(vs)` or `f(vs, dt)` |
| `enter` | `f(c, s, n)` or `f(c, s, n, dt)` |
| `leave` | `f(c, s, n, matchlen, value)` or `f(c, s, n, matchlen, value, dt)` |
A callback names only the parameters it reads, and is called with that many.
```culebra
let p = PEG.parser(`
List <- NUMBER (',' NUMBER)*
NUMBER <- < [0-9]+ >
`)
p["NUMBER"] = fn (vs, dt) {
dt.count += 1
vs.token_to_number()
}
p["List"] = |vs| vs.transform().sum()
let first = {mut count: 0}
let second = {mut count: 0}
inspect(p.parse("1,2,3", dt: first)) # => (true, 6)
inspect(p.parse("40", dt: second)) # => (true, 40)
inspect([first.count, second.count]) # => [3, 1]
```
What a callback keeps in `dt` belongs to one parse. What a closure captures
belongs to the parser object: it is carried from one parse to the next, and a
parse that a callback starts on the same parser shares it.
`dt` is the value itself, not cpp-peglib's reference to the caller's variable.
A callback changes what is inside the Object or Array it was handed and cannot
put another value in its place, so a count is passed inside an Object, as
above. A parse called without `dt` hands `nil`.
### Predicates
A predicate runs when its rule has matched and decides whether the match
stands. When it answers `false` the rule fails there, and the parser goes on
to the next alternative as for any rule that did not match. That is how a
grammar takes a decision its notation cannot express. Here it is C's rule that
`T * x;` declares a pointer only when `T` names a type:
```culebra
let p = PEG.parser(`
Unit <- Stmt+
Stmt <- Typedef / Decl / ExprStmt
Typedef <- 'typedef' 'int' NewType ';'
Decl <- TypeName '*' Ident ';'
ExprStmt <- Ident '*' Ident ';'
NewType <- < Ident >
TypeName <- < Ident >
Ident <- < [a-zA-Z_][a-zA-Z0-9_]* >
%whitespace <- [ \t\r\n]*
`)
p.enable_ast()
p["NewType"].predicate = fn (vs, dt) {
dt.typedefs[vs.token_to_string()] = true
true
}
p["TypeName"].predicate = |vs, dt| dt.typedefs.has(vs.token_to_string())
let (ok, unit) = p.parse("typedef int T; T * x; a * b;", dt: {typedefs: {}})
inspect(unit.nodes.map(|stmt| stmt.nodes[0].name))
# => ['Typedef', 'Decl', 'ExprStmt']
```
The type names are kept in `dt`, so each subject starts with none.
cpp-peglib's predicate returns a `bool` and has two out parameters, `msg` and
`predicate_data`. Here they are returned with it:
| A predicate returns | Meaning |
| --- | --- |
| `true` / `false` | the match stands / the rule fails |
| `(ok, msg)` | the same, and `msg` is the error if the parse ends up failing here |
| `(ok, msg, predicate_data)` | the same, and the rule's action receives `predicate_data` as its third argument |
Anything else is a `TypeError`. A message replaces the syntax error the parser
would otherwise report from the alternatives it tried next, and is reported
where the rule started:
```culebra
let p = PEG.parser(`
NUMBER <- < [0-9]+ >
`)
p["NUMBER"].predicate = fn (vs) {
let n = vs.token_to_number()
n <= 100 ? (true, "", n) : (false, "number out of range")
}
p["NUMBER"] = |vs, dt, predicate_data| predicate_data
let mut errors = []
p.set_logger(|line, col, msg, rule| errors.push([line, col, msg, rule]))
inspect(p.parse("100")) # => (true, 100)
inspect(p.parse("200")) # => (false, nil)
inspect(errors) # => [[1, 1, 'number out of range', 'NUMBER']]
```
In a parse that builds the tree a predicate's `vs` has the rule's text and
tokens but no values (`vs.size()` is `0`): the values there are the tree being
built.
### `enter` and `leave`
`enter` runs when the parser starts on a rule and `leave` when it is done with
it, matched or not:
| Callback | Arguments |
| --- | --- |
| `enter` | `(c, s, n, dt)` |
| `leave` | `(c, s, n, matchlen, value, dt)` |
| Argument | Meaning |
| --- | --- |
| `c` | `{path, s, l}` — the subject's name, the subject, and its length |
| `s`, `n` | the rest of the subject from where the rule was entered, and its length (`c.l - n` is the byte offset) |
| `matchlen` | the bytes the rule matched, whitespace skipped after it included; `-1` when it did not match |
| `value` | the rule's value in a reducing parse; `nil` otherwise, and when the rule did not match |
| `dt` | the parse's [user data](#user-data) |
```culebra
let p = PEG.parser(`
Block <- '{' (Block / Word)* '}'
Word <- < [a-z]+ >
%whitespace <- [ ]*
`)
let mut depth = 0
let mut deepest = 0
p["Block"].enter = fn (c, s, n) {
depth += 1
deepest = Math.max(deepest, depth)
}
p["Block"].leave = fn (c, s, n, matchlen, value) {
depth -= 1
}
inspect(p.parse('{ a { b { c } } d }')) # => (true, nil)
inspect(deepest) # => 3
```
`leave`'s `value` is the value itself, not cpp-peglib's reference to it: a
handler cannot put another value in the rule's place.
Whether, and how many times, a callback runs in an attempt the parser later
abandons is not fixed, as in
[cpp-peglib](https://github.com/yhirose/cpp-peglib#semantic-actions): an
alternative that cannot start at the next byte is skipped without entering
its rules, and packrat parsing reuses a rule's earlier result at a position
instead of trying the rule again. So keep what a callback does safe to repeat
or to skip (undo in `leave` what `enter` did, as above), or build the state
from the value `parse` returns. When what a callback throws leaves the parser,
the `leave` handlers of the rules it passes through do not run.
### Where it differs from cpp-peglib
What a C++ signature cannot be here:
* An out parameter is one more returned value: `parse` answers `(ok, val)`,
and a predicate `ok`, `(ok, msg)` or `(ok, msg, predicate_data)`.
* `dt` and `leave`'s `value` are values, not references. What is inside an
Object or an Array can be changed; the value itself cannot be replaced.
* `token_to_number()` and `transform()` take no type: a token's own text
decides between `Long` and `Float`, and the values are whatever the actions
returned.
What is narrower:
* A tree node is a plain `Object` with the fields [above](#the-tree). It has no
`parent`, `tag`, `original_name`, `original_choice`, `choice_count` or
`path`.
* In a parse that builds the tree, a predicate's `vs` has no values and
`leave`'s `value` is `nil`.
* With an action on any rule the parse reduces; `enable_ast()` then has no
effect.
* `enable_ast` takes no list of rules as a third argument.
* User-defined rules, grammar serialization, `set_error_reporter`,
`enable_trace`, `disable_eoi_check` and `enable_left_recursion` are not
provided. Left recursion is on.
What is stricter:
* Rule nesting is bounded at 4000 in every parse; there is no
`set_max_depth`.
* A callback assigned to a rule the grammar lacks is a `PEGError` at `parse`.
The text of a subject's syntax error is put together here, in the form
culebra's own diagnostics use; a grammar's errors are cpp-peglib's.
## 36. `CodeGen`
A closed intermediate representation and the register-bytecode compiler and
executor that run it — [cpp-vmlib](https://github.com/yhirose/cpp-vmlib),
vendored the same way `PEG` vendors cpp-peglib. Where `PEG` gets you a syntax
tree, `CodeGen` gets you a place to put what you do with one: build a small
language's IR by hand, from a `match` over the tree `PEG.parse` returned, and
run it.
```culebra
let m = CodeGen.Module.new()
let forty_two = m.binary(op: 'add', lhs: m.literal(v: 40, line: 1, col: 1),
rhs: m.literal(v: 2, line: 1, col: 1), line: 1, col: 1)
let args = m.list_new()
m.list_push(args, forty_two)
let print_stmt = m.intrinsic(name: 'print', args: args, line: 1, col: 1)
let stmts = m.list_new()
m.list_push(stmts, print_stmt)
let body = m.block(stmts: stmts, line: 1, col: 1)
m.add_func(name: 'main', num_locals: 0, num_captures: 0, num_cells: 0, num_params: 0, body: body)
m.verify()
m.run() # => 42
```
### IR nodes are `Long`s, not objects
Every builder call on a `Module` returns a plain `Long` — an index into the
module's own node table, not a handle. Build a small expression tree and
thread the numbers through:
```culebra
let m = CodeGen.Module.new()
let a = m.literal(v: 3, line: 1, col: 1)
let b = m.literal(v: 4, line: 1, col: 1)
let sum = m.binary(op: 'add', lhs: a, rhs: b, line: 1, col: 1) # sum is a Long
```
A `Block`'s statements, a `Call`'s arguments and an `Object`'s keys and
values arrive as an **array of node ids**:
```culebra
let m = CodeGen.Module.new()
let a = m.literal(v: 3, line: 1, col: 1)
let b = m.literal(v: 4, line: 1, col: 1)
let args = [m.binary(op: 'add', lhs: a, rhs: b, line: 1, col: 1)]
let body = m.block(stmts: [m.intrinsic(name: 'print', args: args, line: 1, col: 1)],
line: 1, col: 1)
```
`list_new()` and `list_push(list, value)` build the same thing one element
at a time, and every method taking an array takes such a list id too. That
is the older form: it costs a call per element, and nothing needs it now
that the array goes straight in.
### Positions: `at:` instead of `line:` and `col:`
Every builder call records where the node came from. Pass `at:` any Object
with `line` and `column` — a `PEG.parse` node, or one a front end made up
for code it invented — instead of picking the two numbers out yourself:
```culebra
let m = CodeGen.Module.new()
let node = {line: 7, column: 3}
let sum = m.binary(op: 'add', lhs: m.literal(v: 40, at: node),
rhs: m.literal(v: 2, at: node), at: node)
inspect([m.node_line(sum), m.node_col(sum)]) # => [7, 3]
```
`at` sits ahead of `line`/`col`, right after the arguments saying what the
node is, so a front end holding a node writes it positionally:
```culebra
let m = CodeGen.Module.new()
let node = {line: 7, column: 3}
let a = m.literal(1, node)
let sum = m.binary('add', a, m.literal(2, node), node)
inspect(m.node_col(sum)) # => 3
```
`line:`/`col:` still work, and `at:` wins if both are given. Prefer `at`: a
front end holds a node at nearly every builder call, and the two numbers
are then noise on every line. It is the faster spelling too, since one
argument crosses the boundary instead of two and the fields are read on the
other side.
All three are optional. A node built with none of them is at 1:1, which is
what a front end wants for code no source line produced.
### Building a program
| Call | Builds |
| --- | --- |
| `CodeGen.Module.new()` | an empty module |
| `m.literal(v:, at:)` | an integer constant |
| `m.bool_literal(v:, at:)` / `m.double_literal(v:, at:)` / `m.nil_literal(at:)` / `m.str_literal(s:, at:)` | the other scalar constants |
| `m.var_ref(kind:, index:, at:)` | a read of local/capture slot `index` |
| `m.unary(op:, operand:, at:)` | `op` is `'neg'`/`'bitnot'`/`'not'` — `not` yields the Bool inverse of the same truthiness a condition tests, so it answers for every value rather than only for a Bool — or one of `wrapi8 wrapi16 wrapi32 wrapu8 wrapu16 wrapu32` — truncates a Long to that width and sign- (`wrapi*`) or zero-extends (`wrapu*`) it back, for a front end lowering a fixed-width `int`/`uint` (a C#/Java/Go subset, say): the operand must already be a Long, and every one of these keeps a slot's value in the normalized form `lt`/`le`/`gt`/`ge`/`div`/`mod`/`shr` already assume without a wrap of their own |
| `m.binary(op:, lhs:, rhs:, at:)` | `op` is one of `add sub mul div mod eq ne lt le gt ge bitand bitor bitxor shl shr` (bitwise ops are Long-only; shift counts are masked to the low six bits, and `shr` is arithmetic), or one of `udiv umod ushr ult ule ugt uge` — the unsigned forms a `uint64` needs, since its bit pattern does not sit in a signed Long's own ordering the way a normalized narrower unsigned width already does |
| `m.assign(kind:, index:, value:, at:)` | a write to local/capture slot `index` |
| `m.make_if(cond:, then_branch:, at:)` | `if` with no `else` |
| `m.make_if_else(cond:, then_branch:, else_branch:, at:)` | `if`/`else` |
| `m.make_switch(subject:, arms:, at:)` / `m.make_switch_default(subject:, arms:, default_body:, at:)` | branches on `subject` against `arms` — key, body, key, body, … (each key a Long/String literal node, all one kind, pairwise distinct — `verify()` enforces both); `make_switch_default` adds the arm taken when nothing matches. No match and no default yields `nil`, the same as an `if` with no `else` taken; a subject whose type does not match the keys' fails. A `break` inside an arm still targets the enclosing `while` — a switch does not catch it, the same as `if` |
| `m.make_while(cond:, body:, at:)` | a loop |
| `m.block(stmts:, at:)` | a sequence, consuming a staging list |
| `m.call(func:, cmap:, at:)` | a call to function index `func`, forwarding captures via capture-map `cmap` |
| `m.make_closure(func:, cmap:, at:)` | a closure *value* over function `func` — storable, passable, callable later |
| `m.call_value(callee:, args:, at:)` | a call of whatever `callee` evaluates to, consuming a staging list of arguments |
| `m.declare_native(name:)` | declares that the module calls a host function called `name`, and answers its index. The module carries the name alone — what it resolves to is supplied per run, by `Program.run`'s `natives:` — so the same module runs against whatever implementations the caller has that day, and against none at all only if it declares none |
| `m.native_ref(index:, at:)` | the value naming declared native `index`; call it with an ordinary `call_value`. `verify()` rejects an index no `declare_native` handed out |
| `m.intrinsic(name:, args:, at:)` | `name` is `'print'`/`'len'`/`'tostr'`/`'typeof'`/`'toint'`/`'todouble'`/`'tofloat32'` (1 arg each), `'readint'` (0 args), or `'fmod'`/`'pow'` (2 args); `'printraw'` is `'print'` without the trailing newline; the container primitives are `'arraypush'`/`'objecthas'`/`'objectremove'` (2 args) and `'arraypop'`/`'objectkeys'` (1 arg) — pop from empty fails, keys come back in insertion order, and `'len'` answers a map's entry count as well as a string's or an array's length; `'arrayconcat'` (2 args) appends every element of the second array onto the first in place and yields nothing, and `'arrayfill'` (2 args: a count and a value) makes a new array of that many of that value — the two shapes otherwise written as a loop over `arraypush`; `'mapnew'` (0 args) makes a map, a second keyed container whose keys are whole values rather than names — nil, bools, ints, doubles and strings compared by value, everything else by identity — read and written with the same `index`/`set_index`, and answered by the same `'len'`/`'objecthas'`/`'objectkeys'`/`'objectremove'` an object takes; `'strslice'`/`'arrayslice'` (3 args: the string or array, then a start and an end index) copy out a half-open range, and `'strbyte'` (2 args) / `'strfrombyte'` (1 arg) read one byte out of a string and build a one-byte string back; `'strupper'`/`'strlower'` (1 arg) map ASCII A-Z and a-z and leave every other byte alone, so a multi-byte character passes through whole and what case means above ASCII stays the front end's own decision — bytes, not code points, so a front end with its own text model does the decoding itself; `'same'` (2 args) answers reference identity — the same heap object, or for scalars the same tag and payload — which `eq` refuses for two objects; `'argcount'` (0 args) is the count of arguments the running function was called with (see `set_lenient_arity`); `'genresume'`/`'genreturn'`/`'genthrow'` (2 args) drive a generator activation (see `set_generator`); `'enqueue'` (1 arg, a 0-argument closure) queues it on the run's job queue — drained in FIFO order once the entry function has returned, each job run to completion before the next, so a job's own enqueues land behind every job already waiting — and yields `nil`; `'fnarity'` (1 arg) is a closure's declared parameter count (a non-function fails); `'collect'` (0 args) runs a full tracing collection now and yields the number of objects freed, running the drop hook of every condemned object first, newest first; `'heapstats'` (0 args) yields `{live_objects, heap_bytes}` for the runtime's heap. `tostr` formats whole doubles bare — `4`, not `4.0` — so a front end with other display rules post-processes; `typeof` yields the tag as a string (`'int'`, `'double'`, `'string'`, …); `toint` truncates toward zero and fails on NaN/∞/out-of-range; `fmod` is IEEE fmod with the integer-mod zero-divisor failure; `pow` is over doubles; `tofloat32` rounds a Double (an int argument widens first, like `todouble`) to the nearest `float` and back to a Double holding exactly that value — the rounding a front end with both a `float` and a `double` type cannot produce itself — saturating to `float`'s own max/min rather than overflowing early, and going to +-infinity only past the true `float` overflow boundary; NaN passes through unchanged |
| `m.array_lit(items:, at:)` / `m.object_lit(kv:, at:)` | an array from a staging list of items; an object from a staging list holding key, value, key, value, … |
| `m.index(recv:, key:, at:)` / `m.set_index(recv:, key:, value:, at:)` | reads and writes, dispatching on what `recv` turns out to be: an array takes a Long index (out of range fails), an object a String key (a missing key reads as `nil`), and a string reads one byte out as a string (writes are refused) |
| `m.field_get(recv:, slot:, name:, at:)` / `m.field_set(recv:, slot:, name:, value:, at:)` | a struct field at `slot`, a front end's own numbering (the same contract a local's slot index already is) — read/written directly rather than `index`/`set_index`'s key comparison, so `recv` must already have every field built through `object_lit`, in that same slot order, before either reaches it; `name` is carried only for a trap message ("field 'next' of …"), never read to execute anything |
| `m.scope(first_local:, end_local:, body:, at:)` | a lexical region owning local slots `[first_local, end_local)` — released when the region exits, however it exits; defers registered inside run then, LIFO |
| `m.scope_release(first_local:, end_local:, body:, release:, at:)` | the same region with its release order spelled out: `release` holds `var_ref` nodes (`'local'` within the range, or `'cell'`), released in that order at every exit — a front end lists reverse declaration order, captured slots included, so a captured binding is released in its own turn rather than with the frame. A released cell is re-created, so a closure that captured it keeps the old one with its value |
| `m.make_return(value:, at:)` | returns `value` from the running function (a bare `return` is spelled with an explicit `nil_literal`) |
| `m.make_break(line:, col:, depth: 0)` / `m.make_continue(line:, col:, depth: 0)` | leave / re-test a `while`: `depth` is how many enclosing loops to skip, `0` (the default) being the innermost. A front end that resolves its own labels answers with a depth, so the IR needs no label table — and the scopes between here and the target are left on the way out either way. `verify()` rejects either outside a loop body, or naming a loop deeper than the ones actually open |
| `m.make_throw(value:, at:)` | raises any value |
| `m.make_try(caught_local:, body:, handler:, at:)` | guards `body`; a throw (or an executor trap — divide by zero, a wrong-typed operand) lands what it carried in local slot `caught_local` and resumes in `handler`. A trap's value is an object `{message, line, col}`. Yields the value of whichever child finished |
| `m.make_defer(value:, at:)` | registers a 0-arity callable to run when the enclosing `scope()` exits — on fall-through, `break`, `continue`, `return`, and unwinding throws alike; `verify()` rejects a defer outside a scope |
| `m.cell_fresh(cell:, at:)` | replaces one frame cell with a fresh box — what a loop iteration's own binding means: closures from earlier iterations keep the old cell |
| `m.make_yield(value:, at:)` | suspends the running generator function, handing `value` to whoever resumed it; the node's own value is what the next resume sends in. `verify()` rejects it outside a generator |
| `m.set_generator(func:)` | marks function `func` as a generator: calling it packages a suspended activation instead of running the body. `'genresume'` (the activation and a value to send) runs it to its next yield and answers `{value, done}`; `'genreturn'` (the activation and a value) closes it early, running the parked body's pending defers innermost-first, and answers `{value, done: true}`; `'genthrow'` (the activation and a value) delivers the value at the parked yield as if the yield expression had thrown it — the body's own handlers get the first look, its defers run as the throw crosses them, and one the body does not catch comes out of the call with the generator done; a body that catches it and yields again answers `{value, done: false}` like any resume. A generator that never started or already finished has no frame for the value to land in, so it is thrown at the call itself |
| coroutine intrinsics | a coroutine is an ordinary function driven through six intrinsics rather than a node shape of its own — where a generator (`set_generator` above) suspends its one frame at a `yield`, a coroutine suspends its whole call stack, so the `'coroyield'` may sit any number of calls deep inside it. `'corocreate'` (1 arg, a function value) packages one; `'cororesume'` (2 args: the coroutine and a value to send) runs it until it yields or finishes and answers `{value, done}`; `'coroyield'` (1 arg) suspends the running coroutine, handing the value out, and answers whatever the next resume sends in; `'coroclose'` (1 arg) ends a suspended one early, running its parked frames' defers; `'corostatus'` (1 arg) answers `'start'`/`'suspended'`/`'running'`/`'done'`; `'corocurrent'` (0 args) answers the coroutine the running code is inside, or nil at the top. Resuming one that is already running, or yielding from inside a native's callback (there is no C++ frame to park), fails rather than misbehaving. `'enqueue'` takes a coroutine as readily as a closure, which is what makes the job queue a scheduler: a coroutine that yields is parked, and the queue comes back to it |
| `m.set_lenient_arity(func:)` | calls of `func` accept any argument count: extras are dropped, a parameter nothing arrived for starts as `nil`, and the body reads the count actually supplied with `'argcount'` — so a front end raises its own missing-argument diagnostic or fills a default. Without it, a count mismatch is an executor trap |
| `m.set_tail_calls(func:)` | a call of `func`'s in tail position — a `make_return`'s operand, or a body's final value through a block, `if`, `switch` or scope — replaces `func`'s own frame instead of stacking on it, so a loop written as a call chain runs in one frame however long it goes and the depth bound stops mattering. The one thing this changes is *when* the frame exits: before the callee runs, so a local's drop hook precedes the callee's own output where an ordinary call has it follow. A call inside a `make_try` body, or crossing a scope that declares defers or its own release order, stays an ordinary call — each of those needs the frame to still be there |
| `m.set_singleton(func:)` | every `make_closure` of `func` yields the *same* closure object, built at the first one to run and kept for the rest of the run — so a helper library called through `make_closure` + `call_value` stops paying an allocation per call site. Opt-in rather than automatic for every capture-free function because the sharing is observable: `'same'` compares closures by identity, so two evaluations of one function literal would go from distinct to identical. Set it on a function whose closure has no identity of its own — a runtime helper, a top-level procedure — and leave it off for a lambda the source can capture and compare. `verify` refuses it on a function with captures; a generator is fine, since its activation is built by the call, not by the closure |
| `m.set_entry_frame_drops(on:)` | whether the entry function's own bindings run their drop hooks when the program ends (on by default). A front end whose top-level scope is released without destructors turns it off; the entry function's defers still run, and every nested scope still drops as usual |
| `m.list_new()` | a staging list, for `stmts:`/`args:` above |
| `m.list_push(list:, value:)` | appends a node id to a staging list |
| `m.add_func(name:, num_locals:, num_captures:, num_cells:, num_params:, body:)` | a function; returns its index (`funcs[0]` is the entry point). A negative count, or one past 2^31 − 1, is a `ValueError` |
| `m.set_local_name(func:, index:, name:)` | names a local, for diagnostics only |
| `m.set_capture_name(func:, index:, name:)` | names a capture, for diagnostics only |
| `m.capture_map_new()` | a staging capture map |
| `m.capture_map_push(cmap:, kind:, index:)` | appends one forwarded variable to it |
| `m.add_capture_map(cmap:)` | finishes it; the result is what `call()`'s `cmap:` takes |
| `m.verify()` | checks the module structurally; raises `IrError` if malformed |
| `m.run()` | verifies, then executes |
| `m.dump_ir()` | a readable dump of the tree, for debugging |
| `m.dump_bc()` | a readable dump of the compiled bytecode, for debugging |
The executor also honors a drop contract: an object whose `"\x01drop"` key holds a closure gets it called — with the object as its one argument — the moment the object's refcount reaches zero, before it is freed. A throwing destructor is reported to stderr and swallowed; a destructor that stores its argument resurrects the object and the free is skipped.
`kind:` is `'local'` (this function's own slot), `'cell'` (also this
function's own slot, but boxed so a call this function makes can capture it),
or `'capture'` (borrowed from an enclosing one). A variable reference is
never a name lookup at run time — see "Variables are captures" below.
A staging list or capture map is consumed — and gone — the moment it's handed
to `block()`/`intrinsic()`/`call()`; reusing its id afterward is undefined.
### Reading the IR back
Every builder above has a reader: a script's own constant folder, or a test
asserting a node's shape rather than substring-matching `dump_ir()`, walks
the tree the same way the Compiler and Dumper do. A bad node/func/cmap id, or
a reader called on the wrong tag or literal kind, raises `IrError` — the same
failure class `verify()` itself reports.
| Call | Reads |
| --- | --- |
| `m.num_nodes()` | how many nodes exist so far |
| `m.node_tag(node:)` | the tag name — `'literal'`, `'varref'`, … — the same vocabulary `dump_ir()` prints |
| `m.node_line(node:)` | the node's own source line |
| `m.node_col(node:)` | the node's own source column |
| `m.num_children(node:)` | how many children `node` has |
| `m.child(node:, index:)` | the `index`-th child, as a node id |
| `m.const_kind(node:)` | which of the five kinds a `literal` node holds — `'nil'`/`'int'`/`'bool'`/`'double'`/`'str'` |
| `m.int_const(node:)` | an int literal's value; fails on any other kind |
| `m.bool_const(node:)` | a bool literal's value; fails on any other kind |
| `m.double_const(node:)` | a double literal's value; fails on any other kind |
| `m.str_const(node:)` | a str literal's value; fails on any other kind |
| `m.node_op(node:)` | the operator name for a `unary`/`binary`/`intrinsic` node — the same vocabulary `op:`/`name:` accept; a `varref`/`assign` is redirected to `var_kind()` instead |
| `m.var_kind(node:)` | `'local'`/`'capture'`/`'cell'`, for a `varref` or `assign` node |
| `m.var_index(node:)` | the slot index, for a `varref` or `assign` node |
| `m.switch_subject(node:)` | a `switch` node's subject, as a node id |
| `m.switch_arm_count(node:)` | how many `(key, body)` arms `node` has |
| `m.switch_key(node:, index:)` / `m.switch_body(node:, index:)` | the `index`-th arm's key or body, as a node id |
| `m.switch_has_default(node:)` | whether `node` has a default arm |
| `m.switch_default_body(node:)` | its default arm, as a node id; fails if there is none |
| `m.field_slot(node:)` / `m.field_name(node:)` / `m.field_receiver(node:)` | for a `fieldget` or `fieldset` node: its slot, its diagnostics-only name, and its receiver as a node id |
| `m.field_set_value(node:)` | a `fieldset` node's value, as a node id |
| `m.scope_first_local(node:)` | a `scope` node's first owned local slot |
| `m.scope_end_local(node:)` | one past a `scope` node's last owned local slot |
| `m.try_caught_local(node:)` | a `try` node's caught-value local slot |
| `m.closure_func(node:)` | a `makeclosure` node's function index |
| `m.closure_cmap(node:)` | a `makeclosure` node's capture-map index |
| `m.cell_index(node:)` | a `cellfresh` node's cell index |
| `m.num_funcs()` | how many functions exist |
| `m.func_name(func:)` | its name |
| `m.func_num_locals(func:)` | its local-slot count |
| `m.func_num_captures(func:)` | its capture-slot count |
| `m.func_num_cells(func:)` | its cell-slot count |
| `m.func_num_params(func:)` | its parameter count |
| `m.func_body(func:)` | its body, as a node id |
| `m.func_is_generator(func:)` | whether `set_generator` was called on it |
| `m.func_lenient_arity(func:)` | whether `set_lenient_arity` was called on it |
| `m.func_singleton(func:)` | whether `set_singleton` was called on it |
| `m.func_local_name(func:, index:)` | the name `set_local_name` gave that local |
| `m.func_capture_name(func:, index:)` | the name `set_capture_name` gave that capture |
| `m.num_capture_maps()` | how many capture maps exist |
| `m.num_capture_entries(cmap:)` | how many entries `cmap` has |
| `m.capture_kind(cmap:, index:)` | that entry's `kind:` |
| `m.capture_index(cmap:, index:)` | that entry's slot index |
`node_op`/`var_kind`/`capture_kind`/`const_kind` return the same strings the
builders above accept as `op:`/`kind:`/`name:` — a value read back from one
of these can be fed straight into another builder call.
### Compiling once, running many times
`m.compile()` does everything `m.run()` does except the run itself:
verifies `m` and returns a `CodeGen.Program`, so a script can run the same
compiled program more than once, or inspect the heap in between runs —
neither is possible through `m.run()` alone.
| Call | Does |
| --- | --- |
| `m.compile()` | verifies `m`; returns a `CodeGen.Program` |
| `CodeGen.Runtime.new()` | an empty heap a program can run on |
| `p.run(rt: nil, max_call_depth: 10000, natives: nil)` | runs the compiled program. `rt` is a `CodeGen.Runtime` to run on, or nil for a throwaway heap; `max_call_depth` bounds the frame stack. `natives` supplies what the module's `declare_native` names refer to — an Object mapping each declared name to an ordinary culebra `Function`, which the program calls like any other value. Every name the module declared must be there (`IrError` naming the one that is not, before the run starts); a name in the table the module never declared is simply unused, so one table can serve several programs. Values crossing that boundary are copied, and only `nil`, `Bool`, `Long`, `Float` and `String` can: the program's heap and culebra's own are collected independently, so an array, object, map, closure or coroutine would need the two collectors to agree about it. Anything else — as an argument, or as what the function returns — is a `TypeError` naming the slot, catchable by the program's own `make_try`. So is a wrong argument count (checked against the function's own arity), and so is whatever the function throws: a `nil`/`Bool`/`Long`/`Float`/`String` arrives as itself, anything else as the `{kind, message, line, col}` object a culebra error carries |
| `p.dump_bc()` | `m.dump_bc()`'s output for the already-compiled program |
| `rt.live_objects()` | the heap's current object count |
| `rt.heap_bytes()` | the heap's current size in bytes |
| `rt.collect()` | a full collection now — the same one the `'collect'` intrinsic runs from inside a program, run from outside instead; answers how many objects it freed |
```culebra
let m = CodeGen.Module.new()
let forty_two = m.binary(op: 'add', lhs: m.literal(v: 40, line: 1, col: 1),
rhs: m.literal(v: 2, line: 1, col: 1), line: 1, col: 1)
let args = m.list_new()
m.list_push(args, forty_two)
m.add_func(name: 'main', num_locals: 0, num_captures: 0, num_cells: 0, num_params: 0,
body: m.intrinsic(name: 'print', args: args, line: 1, col: 1))
m.verify()
let p = m.compile()
let rt = CodeGen.Runtime.new()
p.run(rt: rt) # => 42
p.run(rt: rt) # => 42
inspect(rt.live_objects()) # => 0
```
A `Runtime` that still holds objects from a *different* compiled program
refuses to run one: a `Program`'s closures index their own chunk table, and
reading another program's would misread it. `rt.collect()` — or simply
using a fresh `Runtime` — clears the way.
`entry_frame_drops` is a property of the compiled `Program`, not a `run()`
argument: call `m.set_entry_frame_drops(on:)` before `compile()` to decide
it, the same flag that already decides what `m.run()` does.
### Variables are captures, not static links
A variable reference names either a slot in the running function's own frame
(`kind: 'local'`) or a slot borrowed from an enclosing one (`kind: 'capture'`).
There is no "level" the way a textbook static-scoping VM might have one: a
static link assumes the frame that declared a variable is still on the stack,
which a closure breaks, so capture forwarding is what survives one arriving
later without `var_ref`'s meaning having to change.
The forwarding table for a call belongs to the **call site**, not the
function being called: a self-recursive procedure forwards its own captures
unchanged on its recursive call, while its first caller forwards its own
cells — two different tables for the same callee, because a per-function
table can't express what a self-recursive call needs (see cpp-vmlib's
README for the fuller version of this point).
A plain `'local'` cannot be forwarded this way: it dies with the frame that
owns it, and the call being built may outlive that frame. A local a call
captures has to be promoted to `'cell'` first — a boxed slot the frame and
every call over it share — before `capture_map_push` can name it; `verify()`
rejects a capture map entry with `kind: 'local'` rather than letting it read
freed memory later. Deciding which slots need promoting is
`CodeGen.Resolver`'s job (below) rather than one to do by hand, and every
read and write of a promoted slot switches to `kind: 'cell'`, not only the
one at the capture site.
### Closure conversion: `CodeGen.Resolver`
Deciding which slots need promoting to cells, and what each function
captures, is the same analysis in every language, and `CodeGen.Resolver` is
it. A variable is an id rather than a (function, slot) pair: `declare` hands
one out, the front end keeps whatever else its language records beside it,
and `use` records a read.
```culebra
let rs = CodeGen.Resolver.new()
let main = rs.new_fn() # no parent: nothing builds main's closure
rs.push_scope()
let x = rs.declare('x', main)
let outer = rs.new_fn(main) # outer's closure is built in main's frame
let inner = rs.new_fn(outer)
rs.use(x, inner) # inner reads a variable main owns
inspect(rs.num_captures(outer)) # => 1
```
That last line is the whole point. `outer` never names `x`, but a closure's
capture map is written in the frame that builds it, and a frame two levels
down cannot name a cell it does not own — so `use` records the read in
*every* function between the reader and the owner. Get that wrong by hand
and the program reads the wrong variable at run time.
Once the reads are in, `number_captures()` assigns every function's capture
indices and every owner's cell indices, and `access_kind`/`access_index`
give a name's `var_ref` arguments:
```culebra
let rs = CodeGen.Resolver.new()
let main = rs.new_fn()
rs.push_scope()
let x = rs.declare('x', main)
let outer = rs.new_fn(main)
rs.use(x, outer)
rs.set_func_index(main, 0)
rs.set_func_index(outer, 1)
rs.set_var_slot(x, 0)
rs.number_captures()
inspect(rs.access_kind(main, x)) # => 'cell'
inspect(rs.access_kind(outer, x)) # => 'capture'
inspect(rs.num_cells(main)) # => 1
```
The counts go to `add_func` and the names to `set_capture_name`, rather
than into the module here: a function's body cannot be emitted until the
capture indices it reads through exist, so the numbering has to come before
the func it describes.
`capture_map(m, builder, target)` writes the forwarding table for a closure
of `target` built in `builder`'s frame. It refuses a frame that cannot
supply what the closure captures, rather than emitting a table of nothing:
build a closure in the frame its function is written in, or record the
captures there first.
| Call | Answers |
| --- | --- |
| `CodeGen.Resolver.new()` | an empty resolver |
| `rs.new_fn(parent:)` | a function id. `parent` is the frame its closure is built in, `-1` when closures are only built at reference sites |
| `rs.set_func_index(fn:, index:)` / `rs.func_index(fn:)` | where in the module's funcs this one's body lands |
| `rs.push_scope()` / `rs.pop_scope()` / `rs.depth()` | the open blocks |
| `rs.declare(name:, owner:)` | a new binding in the innermost scope, owned by `owner` |
| `rs.declare_in(scope:, name:, owner:)` | the same, into a scope that is not the innermost — a `global`, or a hoist into a function's outermost block |
| `rs.alias(name:, v:)` | a second name for a binding that already exists. Not a declaration, so it joins no order table |
| `rs.declared_here(name:)` / `rs.declared_at(scope:, name:)` | what that one scope binds the name to, or `-1` |
| `rs.declared_count(scope:)` / `rs.declared_index(scope:, i:)` | what the scope declared, in order, one entry per `declare` |
| `rs.lookup(name:)` / `rs.lookup_from(name:, from_scope:)` | the innermost scope that binds it, or `-1` |
| `rs.use(v:, fn:)` | records that `fn` reads `v` |
| `rs.var_name(v:)` / `rs.var_owner(v:)` / `rs.var_slot(v:)` / `rs.set_var_slot(v:, slot:)` | what a variable id stands for |
| `rs.number_captures()` | assigns the capture and cell indices. Once, after the reads are in |
| `rs.read(m:, fn:, v:, at:)` / `rs.write(m:, fn:, v:, value:, at:)` | that read or write as an IR node. The pair every front end builds by hand otherwise |
| `rs.name_captures(m:, func:, fn:)` | names `func`'s captures in index order |
| `rs.access_kind(fn:, v:)` / `rs.access_index(fn:, v:)` | how `fn` reaches `v`: `'local'`, `'cell'` or `'capture'`, and the index, for a read this pair does not cover |
| `rs.num_captures(fn:)` / `rs.num_cells(fn:)` / `rs.cell_of(fn:, v:)` | what `add_func` wants, and which cell holds a variable (`-1` for none) |
| `rs.closure(m:, builder:, target:, at:)` | a closure of `target` built in `builder`'s frame, as one node |
| `rs.capture_map(m:, builder:, target:)` | that closure's forwarding table alone, for a front end assembling the node itself |
| `rs.reaches(fn:, v:)` | whether `fn` can name `v` at all |
| `rs.mark()` | everything the resolver holds right now, as a token |
| `rs.rollback(mark:)` | gives back the functions, variables and free entries added since |
| `rs.reset_fn(fn:, parent:)` | starts one kept function over |
A lookup that finds nothing answers `-1`, not `nil`: every id here is a
non-negative index.
#### A prelude bound once
A front end that writes part of its runtime in its own language binds that
prelude and then compiles program after program against it. `mark()` after
the prelude and `rollback(mark)` before each program give back what the
last one added -- its functions, its variables, and any free-set entry
naming one of them -- while what the prelude captured stays. `reset_fn` is
for the entry function, which a rollback keeps and the next program rebuilds
anyway.
#### When one walk is not enough
`use` closes the free sets in a single walk, which is right for a language
where a function value comes from an expression — the closure is built where
the function is written, so the propagation follows the nesting. A language
where a function is a *static entity* built afresh at every reference to it
(PL/0's procedures, a hoisted `fn` that can be a whole overload family) has
a different obligation: every referrer must carry what the referee captures,
and recursion makes that a fixpoint rather than a walk.
A front end in that position declares its call edges and asks for the
closure, between the reads and `number_captures`:
```culebra
# doctest: skip
rs.note_call(caller, callee) # once per reference, while binding
rs.close_over_calls() # then, before number_captures
```
It is a method to ask for rather than something `use` does, and that is the
whole difference. The propagation `use` already does costs nothing: a
closure built where its function is written rides the nesting. Lifting
along call edges instead makes every frame between a referrer and an owner
carry a capture it has no use for, and — the part that matters — makes a
lowering that builds closures in the wrong frame work anyway. A front end
that declares no call edges pays for none of it.
### Errors, interruption, and recursion
A runtime failure — divide by zero, an uninitialized read, a wrong-typed
operand, a runaway recursion — is catchable *inside* the built program
first: a `make_try` region whose body raises it resumes in its handler with
an object `{message, line, col}`. Only a failure no `make_try` guards (and
every structural `verify()` failure) escapes to the culebra level, as
`CulebraError` with `kind: 'IrError'`:
```culebra
let m = CodeGen.Module.new() # no funcs at all -- verify() catches it
inspect(try {
m.run()
} catch e {
e.kind
}) # => 'IrError'
```
A `CodeGen`-built program is interruptible (Ctrl+C, an isolate's own cancel)
the same way any other running culebra code is, and a runaway recursive call
raises `IrError` (`"recursion limit exceeded"`) rather than overflowing the
process's own stack.
### Scope
Values are `nil`, booleans, integers, doubles, strings, arrays, objects,
closures, and generator activations (see `set_generator`); a variable's
capture is resolved once, when a front end builds the IR, not at run time. A closure
built with `make_closure` is a first-class value: storable in a variable,
passable as a `call_value` argument, returnable as a function's result.
Neither `CodeGen.Module`, `CodeGen.Program` nor `CodeGen.Runtime` can cross
an isolate boundary (`Isolate.spawn`/`Parallel.map`'s workers each build
their own).
## 37. `StateMachine`
A hierarchical state machine — a *statechart* — as an ordinary culebra
class, not a language builtin. States
nest, an event the active state does not handle bubbles to its ancestors,
and leaving a composite state runs its `exit` on the way out.
Two front ends build the same machine. `StateMachine.new` takes a
description `Object`, so guards and actions are ordinary functions written
beside it. `StateMachine.parse` takes the text DSL, where guards and
actions are *names* resolved against the `guards:` / `actions:` tables —
the same "name → function" table §35's semantic actions use.
```culebra
let vending = `
Vending {
idle initial {
coin: add_coin -> ready
}
ready {
coin: add_coin -> ready
select[enough]: dispense -> idle
select[!enough]: reject -> ready
}
}
`
let m = StateMachine.parse(vending,
guards: {enough: fn (ctx, ev) { ctx.balance >= 100 }},
actions: {
add_coin: fn (ctx, ev) { ctx.balance += ev.payload },
dispense: fn (ctx, ev) { ctx.balance -= 100; ctx.log.push('dispensed') },
reject: fn (ctx, ev) { ctx.log.push("only {ctx.balance}") },
},
context: {mut balance: 0, log: []})
m.fire('coin', 60)
inspect(m.state()) # => 'ready'
m.fire('select')
m.fire('coin', 60)
m.fire('select')
inspect(m.state()) # => 'idle'
inspect(m.context.log) # => ['only 60', 'dispensed']
```
Write the machine as a **raw string** (backtick or single-quoted): a
`"..."` string would read its braces as interpolation.
### The text DSL
A body line is `event ['[' guard ']'] [':' action] ['->' target]` and a
state header is `name [initial] [entry: action] [exit: action]`. Nothing in
a body is a reserved word — `entry`, `exit` and `initial` mean something
only in a header — so an event may be called `entry`. `#` starts a comment,
and newlines are ordinary whitespace, so `a { x -> b }` on one line is the
same machine as the spread-out form.
| Form | Meaning |
| --- | --- |
| `name { ... }` | a state; nesting the block makes it composite |
| `name initial { ... }` | the state its parent enters by default |
| `name entry: f exit: g { ... }` | run `f` on entering the state, `g` on leaving |
| `e -> s` | on `e`, go to `s` |
| `e: f -> s` | …running `f` between the exits and the entries |
| `e: f` | an internal transition: `f` runs, no state is left or entered |
| `e -> own-name` | an external self-transition: `exit` then `entry` do run |
| `e[g] -> s` | only when guard `g` passes |
| `e[!g] -> s` | only when it does not |
State names are global to one machine, so `-> normal` needs no path. Two
lines for the same event are tried in order and the first whose guard
passes wins; if none does, the event keeps bubbling outward.
### Hierarchy
A transition exits and re-enters exactly the states below its **domain** —
the deepest state that is a proper ancestor of both the state that declared
the transition and its target. So a move between siblings inside a
composite state leaves the parent alone, while a move out of it runs the
parent's `exit`; and entering a composite state descends through `initial`
to a leaf, running each `entry` on the way down.
```culebra
let player = StateMachine.parse(`
Player {
stopped initial { play -> normal }
playing entry: start_clock exit: stop_clock {
stop -> stopped
normal initial { seek -> seeking }
seeking { done -> normal }
}
}
`, actions: {
start_clock: fn (ctx, ev) { ctx.log.push('start') },
stop_clock: fn (ctx, ev) { ctx.log.push('stop') },
}, context: {log: []})
inspect(player.state()) # => 'stopped'
player.fire('play') # descends to the initial leaf
inspect(player.state()) # => 'normal'
inspect(player.in_state('playing')) # => true
player.fire('seek') # stays inside `playing`
inspect(player.context.log) # => ['start']
player.fire('stop') # declared on `playing`: bubbles up
inspect((player.state(), player.context.log)) # => ('stopped', ['start', 'stop'])
```
### The description form
The same machine as an `Object`: state name → `{initial, entry, exit, on,
states}`, where `on` maps an event to one candidate or an `Array` of them,
each `{guard, negate, action, target}`. Every key is optional. Guards and
actions here are usually the functions themselves; a `String` is looked up
in `guards:` / `actions:` exactly as the DSL's names are.
```culebra
let bump = fn (ctx, ev) { ctx.n += 1 }
let m = StateMachine.new({
off: {initial: true, on: {flip: {action: bump, target: 'on'}}},
on: {on: {flip: {target: 'off'}}},
}, context: {mut n: 0})
m.fire('flip')
m.fire('flip')
inspect((m.state(), m.context.n)) # => ('off', 1)
```
### Members
| Member | Returns |
|---|---|
| `StateMachine.new(desc: Object, *, name: String = "", guards: Object = {}, actions: Object = {}, context = nil)` | `StateMachine` |
| `StateMachine.parse(text: String, *, guards: Object = {}, actions: Object = {}, context = nil, path: String = "")` | `StateMachine` — the same, from the text DSL; `path` names the subject in a `PEGError` |
| `m.context` | the `context:` value, unchanged — guards and actions receive it |
| `m.state()` | `String` — the active leaf state |
| `m.in_state(name: String)` | `Bool` — true for the active leaf and for every state containing it |
| `m.fire(event: String, payload = nil)` | `Bool` — whether a transition was taken; an event no state handles is ignored, not an error |
| `m.can_fire(event: String, payload = nil)` | `Bool` — whether `fire` would take one (guards run, so they must be free of side effects) |
| `m.reset()` | `Nil` — leave the active configuration and enter the initial one again; the context is yours and is left alone |
| `"{m}"` / `to_string(m)` | `String` — `"StateMachine(Name, state=s)"` |
Guards and actions are `fn (ctx, ev)`, where `ctx` is the `context:` value
and `ev` is `{name, payload}` — an `Object`, so a later field cannot break
an existing closure. The machine enters its initial configuration twice
without an event of its own — when it is built and on `reset()` — and the
`entry` actions that run then see `ev.name` as `nil`. Every state, guard
and action name is resolved when
the machine is built, so a typo raises `StateMachineError` there rather
than on the first event that would have reached it; so do a machine with no
initial state, a composite state without one, and two initial siblings.
Not modelled: parallel regions and history states (SCXML's `` and
``). A machine is in exactly one leaf state at a time.
## 38. `FST`
A **compiled read-only dictionary** — a set of keys, or a map from keys to
values, built once into a compact byte string and then queried (engine: the
vendored [cpp-fstlib](https://github.com/yhirose/cpp-fstlib), a finite state
transducer). The structure shares both prefixes and suffixes between keys, so
a large word list compresses well and stays queryable in place; the price is
that it cannot be modified after it is built.
Reach for it when you have many keys and want more than membership:
autocomplete (`predictive_search`), longest-match tokenizing
(`longest_common_prefix_search`), or spelling correction
(`edit_distance_search`, `suggest`). For plain membership of a set that
changes at runtime, use a language `Set` or `Object` instead.
Building and querying are separate steps, because the point of the structure
is to build once and query many times — often in another run, or another
process:
```culebra
let words = ["hello", "hell", "help", "world"]
let bytes = FST.compile_set(words) # a String, safe to write to a file
let dict = FST.Set.new(bytes) # open it for querying
inspect(dict.contains("hell")) # => true
inspect(dict.predictive_search("hel")) # => ['hell', 'hello', 'help']
```
The byte code is an ordinary `String` (binary-safe, like `Compress` output),
so it needs no special file API. It carries its own length and checksum, which
is how a truncated or corrupted file is refused when opened rather than read
(byte code written before v0.7.0 has neither and is refused too — rebuild it
from the words with `FST.compile_*`):
```culebra
# doctest: skip
FS.write("words.fst", FST.compile_set(words))
let dict = FST.Set.new(FS.read("words.fst"))
```
### Building
Every builder returns the byte code as a `String`. `sorted: true` promises the
input is already in sorted order and skips the internal sort — cheaper on a
large pre-sorted list, and an `FSTError` if the promise does not hold.
| Builder | Result |
| --- | --- |
| `FST.compile_set(keys: [String], sorted: Bool = false)` | keys only → open with `FST.Set` |
| `FST.compile_map(entries: Object, sorted: Bool = false)` | `{key: String}` → open with `FST.Map` |
| `FST.compile_index_map(entries: Object, sorted: Bool = false)` | `{key: Long}` → open with `FST.IndexMap` |
| `FST.compile_auto_index(keys: [String], sorted: Bool = false)` | keys only, each valued by its rank in sorted order → open with `FST.IndexMap` |
`compile_index_map` values are whole numbers in `0..4294967295`; anything
outside that raises `ValueError` at the call. An empty key, a duplicate key,
or input that breaks a `sorted: true` promise raises `FSTError` naming the
offending index.
`compile_auto_index` is the compact way to map a word list onto positions:
the value of each key is its 0-based rank once the keys are sorted, so it
numbers the dictionary itself rather than any array you keep alongside it.
```culebra
let ranked = FST.IndexMap.new(FST.compile_auto_index(["cherry", "apple", "fig"]))
inspect(ranked.get("apple")) # => 0
inspect(ranked.get("cherry")) # => 1
inspect(ranked.get("fig")) # => 2
```
### Querying
Three classes, one per kind of compiled dictionary. Each takes the byte code
and raises `FSTError` if it is corrupt — or if it was built for a different
one of the three, so opening a `Set`'s bytes as a `Map` is caught rather than
silently answering nonsense.
| Constructor | Result |
| --- | --- |
| `FST.Set.new(bytes)` | `Set` — keys only |
| `FST.Map.new(bytes)` | `Map` — each key carries a `String` |
| `FST.IndexMap.new(bytes)` | `IndexMap` — each key carries a `Long` |
The three share a method surface; only what a result carries differs. `Set`
answers with bare keys and lengths, while `Map` / `IndexMap` add the stored
`value` to each:
| Method | `Set` | `Map` / `IndexMap` |
| --- | --- | --- |
| `contains(key)` | `Bool` | — |
| `get(key)` | — | the value, or `nil` if the key is absent |
| `common_prefix_search(text)` | `[Long]` — the length of every key that prefixes `text` | `[{length, value}]` |
| `longest_common_prefix_search(text)` | `Long` or `nil` | `{length, value}` or `nil` |
| `predictive_search(prefix)` | `[String]` — every key starting with `prefix` | `[{key, value}]` |
| `edit_distance_search(word, max_edits, insert_cost = 1, delete_cost = 1, replace_cost = 1)` | `[String]` | `[{key, value}]` |
| `suggest(word)` | `[{ratio, key}]` | `[{ratio, key, value}]` |
Lengths are **byte** counts, like `Regex`'s offsets. A miss reads as `nil`
(`get`, `longest_common_prefix_search`) or an empty array, so results compose
with `??` and `?.`.
`common_prefix_search` walks the keys that are prefixes of the subject —
the tokenizer's question, "what does the dictionary match here" — while
`predictive_search` walks the keys the subject is a prefix *of*, which is the
autocomplete question.
```culebra
let d = FST.Set.new(FST.compile_set(["hell", "hello", "help", "world"]))
inspect(d.common_prefix_search("helpless")) # => [4]
inspect(d.longest_common_prefix_search("helpless")) # => 4
inspect(d.longest_common_prefix_search("zebra")) # => nil
inspect(d.predictive_search("hel")) # => ['hell', 'hello', 'help']
```
A `Map` answers the same shapes with the stored value attached:
```culebra
let m = FST.Map.new(FST.compile_map({hello: "こんにちは", world: "世界"}))
inspect(m.get("hello")) # => 'こんにちは'
inspect(m.get("missing")) # => nil
inspect(m.predictive_search("w")) # => [{key: 'world', value: '世界'}]
inspect(m.longest_common_prefix_search("worldly")) # => {length: 5, value: '世界'}
```
### Fuzzy search
`edit_distance_search` returns every key reachable within `max_edits`, and
the three costs price the edit kinds separately. They are named **from the
search word's side**: a `delete` drops a byte the key has and the word lacks,
an `insert` adds a byte the word has and the key lacks, and a `replace`
changes one. Raising a cost pushes that kind of neighbour out of a given
budget:
```culebra
let d = FST.Set.new(FST.compile_set(["hell", "hello", "help"]))
inspect(d.edit_distance_search("helo", 1)) # => ['hell', 'hello', 'help']
# Reaching "hello" from "helo" means dropping the key's second "l":
inspect(d.edit_distance_search("helo", 1, 1, 2)) # => ['hell', 'help']
# Only the same-length keys pay a replace:
inspect(d.edit_distance_search("helo", 1, 1, 1, 2)) # => ['hello']
```
`suggest` is the spellchecker's form: it ranks candidates by similarity and
returns them best first, so a caller can take the top few rather than pick a
distance up front. The `ratio` is the engine's similarity score in `0..1`.
```culebra
let d = FST.Set.new(FST.compile_set(["their", "there", "third", "tier"]))
inspect(d.suggest("thier")[0].key) # => 'their'
```
Keys are compared as **bytes**, so matching is exact about case and about
Unicode normalization, and an edit distance counts bytes rather than
characters — a one-character change to a multi-byte character costs more than
one edit. Normalize (`.lower()`, and NFC via `String`) before building and
before querying when that matters.
## 39. `Search`
A **full-text index**: add documents under keys you choose, then search them
with a query language and get ranked hits back (engine: the vendored
[cpp-searchlib](https://github.com/yhirose/cpp-searchlib), an inverted index
with BM25 ranking). The surface here is a narrow subset of what the engine
does, and it will grow.
Reach for it when a substring scan is not enough: many documents, queries that
combine terms, and results that need an order. For finding one pattern in one
string, use `Regex`; for prefix lookup over a fixed word list, use `FST`.
```culebra
let idx = Search.Index.new()
idx.add("doc-1", "The quick brown fox jumps over the lazy dog.")
idx.add("doc-2", "A quick brown dog outpaces a lazy fox.")
for hit in idx.search("quick -dog") {
inspect(hit.key)
}
```
An index is a live handle holding its documents in memory. It is closed at
scope exit like a file, or explicitly with `close()`.
### Keys are yours
`add` takes the key you name the document by, and `search` hands that same key
back. Keys are `String`s: a `Long` is refused rather than stringified, so the
key that comes out is the key that went in. Adding under a key that already
exists replaces that document.
```culebra
let idx = Search.Index.new()
idx.add("a", "first")
idx.add("a", "second") # replaces it
inspect(idx.search("first")) # => []
inspect(idx.search("second").size()) # => 1
```
`remove(key)` deletes a document; an unknown key is a no-op. Deletion is
logical — the space is not reclaimed until the index is rebuilt.
### The query language
| Syntax | Meaning |
|---|---|
| `apple banana` | AND — documents with every term |
| `apple \| banana` | OR |
| `apple -banana` | NOT — excludes documents with the term |
| `"apple tree"` | phrase — adjacent terms |
| `apple ~ tree` | NEAR — within 4 term positions |
| `app*` | prefix — every indexed term starting with `app` |
| `a*e`, `*ana` | wildcard — `*` anywhere in the term |
| `apple~2` | fuzzy — terms within 2 edits |
| `( ... )` | grouping |
`NOT` needs at least one positive term, since the index cannot enumerate every
document. A malformed query raises `SearchError` rather than matching nothing.
Documents and queries are cut into terms the same way: Unicode word boundaries
([UAX #29](https://unicode.org/reports/tr29/)), keeping the segments that
contain a letter or a number, lowercased. So `version 2.0` is `version` and
`2.0`, `don't` and `1,234.56` are one term each, spaces and punctuation between
words are not terms, and case does not matter. Scripts written with spaces
come out word by word; the ones written without them (Han, Hiragana, Thai …)
come out one character at a time, a run of Katakana being one term — the
unigram baseline, where UAX #29 itself defers to a dictionary. `東京タワー` is
indexed as `東` `京` `タワー`, and a query for `東京` is the phrase `東` + `京`,
so it is found; a dictionary-based splitter (below) is what makes `東京` one
term.
### The analyzer
An index can be given its own way of turning text into terms. The same one
runs on both sides — when you `add` a document and when you `search` — because
an index built with one set of term boundaries and queried with another finds
nothing.
```culebra
let stop_words = fn (t) {
if ["the", "a", "of"].contains(t) { nil } else { t }
}
let idx = Search.Index.new(analyzer: { filters: [stop_words] })
idx.add("doc-1", "The quick brown fox")
inspect(idx.search("quick").size()) # => 1
inspect(idx.search("the")) # => []
```
| Field | | |
|---|---|---|
| `splitter` | `fn (text: String) -> Array` | cuts text into terms. Each element is `{term, position, length}` — the term string plus the byte range it came from. Omit it for the built-in splitting described above. Also takes a `Search.segmenter` (below) or a splitter written in C++ (next paragraph). |
| `normalizer` | `fn (term: String) -> String` | runs on every term, before the filters. Omit it for the built-in lowercasing. |
| `filters` | `Array` of `fn (term: String) -> String \| Nil` | applied in order after the normalizer. Return a replacement term, or `nil` to drop it. |
The emitted term need not be the bytes its range points at, which is what lets
a morphological analyzer index a base form while highlighting still lands on
the text as written. Range starts must never decrease. Two terms whose ranges
start at the same byte are alternatives at one term position — a compound
beside its parts, so that a search for either finds the document and a phrase
runs through either — and a hit on any of them highlights the first one's
range. A term starting later takes the next position, whether or not the
ranges overlap. The query side runs the same splitter, so a token it splits
this way is searched as a phrase with an alternative at each position.
A filter turning one term into several is deliberately not expressible: several
outputs mean alternatives, which belong on the query side, and a *sequence* of
pieces is the splitter's job.
**These hooks are called once per term.** A `normalizer` written in Culebra
runs on every token of every document, which is why the default is native; a
`splitter` costs one call per document and per query token. They are the
convenient path, not the fast one — a thousand-word document means a thousand
crossings back into the program.
What a hook raises leaves `add` or `search` as it was raised: an error keeps
its kind, and a thrown value reaches the program's `catch` as that value.
A splitter written in C++ runs natively instead: a class deriving from
`ISplitter` and declared with `wrap` gives a handle the `splitter` slot
accepts as it is. The contract and the handle's lifetime are in [Plugging a
splitter into Search](deployment.md#plugging-a-splitter-into-search).
`Search.Index.load(path, analyzer)` takes one too, and it has to be shaped
like the one that built the index — a splitter cuts the terms an index holds,
and a query cut differently finds nothing. The file records that shape and
`load` refuses a mismatch (see [Saving and loading](#saving-and-loading)).
### Japanese: a model-based splitter
The built-in splitting leaves Han and Hiragana one character at a time
(above). `Search.segmenter(model)` loads a word-segmentation model (engine:
[cpp-segmentlib](https://github.com/yhirose/cpp-segmentlib), bundled with
cpp-searchlib) and returns a splitter that cuts runs of Japanese script into
words and leaves everything else to the built-in rules — `iPhone 15を買った`
indexes as `iphone` `15` `を` `買っ` `た`, and `東京タワー` as `東京` `タワー`.
Hand it to the analyzer as its `splitter`. It runs natively, with no call
back into the program per document.
```culebra
# doctest: skip
let seg = Search.segmenter("ja-ud-gsd")
let idx = Search.Index.new(analyzer: { splitter: seg })
idx.add("doc-1", "私は東京タワーに行った")
inspect(idx.search("東京タワー").size()) # => 1
inspect(idx.search("京")) # => [] 東京 is one term now
```
`model` is a name from the table below, or the path of a model file in a
format cpp-segmentlib loads. A file that cannot be loaded raises
`SearchError`. The segmenter is a handle like an index: closed at scope exit
or with `close()`. An index copies the splitter when it is built, so closing
the handle afterwards does not affect the index — it only stops new indexes
from being built with it. The model is part of the analyzer in the sense
above: the same one on both sides, and a re-index when it changes.
| Name | Model | Size | License |
|---|---|---|---|
| `ja-ud-gsd` | cpp-segmentlib's reference model v0.3.0, trained on UD Japanese-GSD | 2.1 MB | CC BY-SA 4.0 |
A name resolves to a copy cached under `Sys.data_dir("culebra")/models`.
The first time, culebra asks on the terminal before fetching it; a program
whose stdin or stderr is not a terminal — a script, a CI job, a pipe — is not
asked and raises `SearchError` instead, with the URL, the path the copy is
expected at, and the two ways around the fetch (put the file there yourself,
or pass a path). The download is checked against a digest pinned in this
release before it is written, and the model's NOTICE goes beside it. With
`CULEBRA_OFFLINE` set to anything but `0`, nothing is fetched. An AOT binary
that never names `Http` cannot fetch either; the cache is shared, so one
`culebra` run on the same machine fills it for both.
The version is part of the cached file's name (`ja-ud-gsd-v0.3.0.mod`), so
an upstream update never replaces the model an index was built with: word
boundaries that change mean a re-index, and it is a release of culebra, not
the network, that decides when.
### The language is the index's
Which model an index uses, if any, is the analyzer's setting, so it is the
index's: every document and every query go through the same one. There is no
per-document language and no guessing from the text. A query is a few
characters where a document is a page, so a guess made from each side would
disagree: `東京都庁` inside a document with kana around it cuts as `東京`
`都庁`, but alone as a query it has no kana to guess from and would stay
whole, matching nothing. Mixed scripts are handled by script, not by
language: a model claims the runs of Japanese script and the built-in rules
take the rest, on both sides alike. A corpus in several languages that need
different models is several indexes.
### Hits
`search(query, limit = 10)` returns an Array of Objects, best score first:
| Field | |
|---|---|
| `key` | the document key you added it under |
| `score` | BM25 relevance; comparable within one result, not across searches |
| `ranges` | where the matched terms are, as `{position, length}` byte offsets into the text you added |
The ranges are what highlighting needs:
```culebra
let text = "A quick brown dog outpaces a lazy fox."
let idx = Search.Index.new()
idx.add("doc", text)
let r = idx.search("outpaces")[0].ranges[0]
inspect(text.slice(r.position, r.position + r.length)) # => 'outpaces'
```
### Saving and loading
```culebra
# doctest: skip
idx.save("notes.idx")
let reopened = Search.Index.load("notes.idx")
```
The file holds the index, not the documents: `search` still answers with keys
and byte ranges, and it is on the caller to keep the original text around to
resolve them against. The format is culebra's, wrapping the engine's. It
carries a version, so an index written by a culebra that wrote a different one
is reported as such rather than misread — rebuild it from the documents, which
the index does not keep.
It records how the analyzer that built it was shaped, and `load` refuses an
analyzer shaped differently:
```culebra
# doctest: skip
let idx = Search.Index.new(analyzer: { splitter: Search.segmenter("ja-ud-gsd") })
idx.add("doc-1", "東京タワーに行った")
idx.save("notes.idx")
Search.Index.load("notes.idx") # !! SearchError: built with a different analyzer
```
The shape is coarse, because it is what a file can honestly record: which
kind of splitter cut the terms (the built-in rules, a model — named by its
file and size —, a wrapped C++ class, or a Culebra closure), whether the
normalizer was the built-in one, and how many filters ran. A Culebra function
has no identity that survives the process, so every closure reads the same:
swapping one stop-word list for another of the same length is not caught. What
is caught is the mismatch that has a name — an index built with a model opened
without one, a filter that is no longer there — which is the shape most of
these mistakes take.
The file records the layout it was written in, and `load` refuses another
with `SearchError` rather than misreading it: an index saved before v0.7.0 has
to be rebuilt with `Search.Index.new`.
The file is written with Elias-Fano compressed postings, and `load` expands
them back into the form `new` builds, so a loaded index searches at full speed
and can still be added to. `readonly: true` skips the expansion and searches
the compressed structures where they lie:
```culebra
# doctest: skip
let idx = Search.Index.load("notes.idx", readonly: true)
```
Such an index opens in a fraction of the time and holds a fraction of the
memory, and `add`, `remove` and `save` raise `SearchError` on it. Searching is
slower, most of all for phrases. Measured on 311,030 documents (the KJV, ten
copies; a 36 MB index), best of five:
| | `load` | memory | one common word | two words | phrase |
|---|---|---|---|---|---|
| default | 1,592 ms | 376 MB | 35 µs | 23 µs | 205 µs |
| `readonly: true` | 83 ms | 102 MB | 122 µs | 83 µs | 1,346 µs |
So it suits an index that is built once and searched by short-lived processes
— a CLI that opens the file per invocation pays 83 ms rather than 1.6 s — and
the default suits a long-lived process that searches the same index many
times, or one that keeps adding to it.
### Signatures
| | |
|---|---|
| `Search.Index.new(analyzer: Object = nil) -> Index` | an empty index |
| `Search.Index.load(path: String, analyzer: Object = nil, readonly: Bool = false) -> Index` | reopen a saved one; `readonly` searches it compressed (above) |
| `idx.add(key: String, text: String) -> Nil` | index a document, replacing any with the same key |
| `idx.remove(key: String) -> Nil` | delete one; unknown keys are ignored |
| `idx.search(query: String, limit: Long = 10) -> Array` | ranked hits |
| `idx.save(path: String) -> Nil` | write the index |
| `idx.close() -> Nil` | release it; idempotent |
| `Search.segmenter(model: String) -> Segmenter` | a model-based Japanese splitter for `analyzer.splitter`; `model` is a catalog name or a file path |
| `seg.close() -> Nil` | release it; indexes built with it keep working |
An index is not `Sendable`: results alias it, so it stays on the thread that
built it. Give each isolate its own.
## 40. `Audio`
Sound for any program: WASM-4-style tones, one-shot samples, streamed music,
PCM the script synthesises, the microphone, and songs written in Kauai. `Audio` owns the sound device on its own and
needs no window, so it works the same whether a program draws with `Canvas`,
with `Scene`, or not at all. The device opens on first use; a program that
never plays anything never opens it.
No device is not an error. On a machine without one (a server, CI), or in any
run with `CULEBRA_AUDIO` set to `off` or `0` (as every `just` recipe sets it),
`Audio.available()` is `false`, nothing plays, and every call answers as this
section says: handles are made, controls do nothing, `playing()` is `false`,
and a `PCM` still counts what it is given. The first call that wanted a
missing device prints one warning.
| Function | Effect |
| --- | --- |
| `Audio.available() -> Bool` | whether an audio device is open (opening it if nothing has yet) |
Volumes are `0.0..1.0`, `1.0` being the source's own level; pitch is a
playback rate, `1.0` unchanged; pan runs from `-1.0` (left) through `0.0` to
`1.0` (right). `tone` alone keeps WASM-4's units, below.
### Tones
`Audio.tone(freq, dur, vol = 100, wave = 0, end_freq = nil, attack = 0,
decay = 0, release = 0, peak = nil, duty = 2)` plays a note through a small
WASM-4-style APU, in WASM-4's own units, so a WASM-4 program's calls carry
over unchanged. In its simplest form `Audio.tone(freq, dur)` is a `dur`-tick
note (a tick is 1/60 s) at `freq` Hz. The optional arguments expose the full
envelope: the pitch slides `freq → end_freq` while an ADSR envelope
(`attack`/`decay`/`release` in ticks, `dur` as the sustain length) shapes the
level from 0 up to `peak`, down to the sustain `vol` (both `0..100`), and back
to 0. `wave` selects the channel — `Audio.PULSE` / `PULSE2` (with a `duty`
cycle: `Audio.DUTY_EIGHTH` / `DUTY_QUARTER` / `DUTY_HALF` /
`DUTY_THREE_QUARTER`), `Audio.TRIANGLE`, `Audio.NOISE`, or the culebra
extension `Audio.SAWTOOTH`. Each channel is monophonic (a new note cuts the
previous one). A synthesised waveform is raw and four channels can sound at
once, so `tone` mixes its `0..100` well under full scale.
In the browser, `tone` plays through WebAudio (an oscillator per channel, a
`PeriodicWave` for the pulse duty cycle, a filtered noise buffer). Natively it
plays through a small software synth mixed on the audio thread (naive, not
band-limited, oscillators; noise is a filtered PRNG). Either way a note is
placed on the audio stream's own sample clock rather than at the edge of
whatever buffer the device was filling, so a sequencer keeps its spacing
whatever buffer length the device chose. The two are independent
implementations tuned to sound alike, not sample-identical: unlike pixel ops,
`tone` isn't required to produce bit-identical output across backends.
```culebra
Audio.tone(440, 8)
Audio.tone(700, 0, 10, Audio.PULSE, 870, 10, 5, 3, 20, Audio.DUTY_QUARTER)
```
### Sound
`Audio.Sound.new(data)` decodes a one-shot sample from its bytes (WAV, MP3 or
Ogg Vorbis, a `String` from `FS.read`, a `Dir` or built by the program) and
plays it per call: the recorded-sample counterpart to `tone`'s synthesis.
| Method | Effect |
| --- | --- |
| `sound.play()` | play from the start (restarts if still playing) |
| `sound.stop()` | stop the voice |
| `sound.playing() -> Bool` | still audible? |
| `sound.volume(v)` / `sound.pitch(p)` / `sound.pan(p)` | as above; they apply from the next `play` and to the voice now sounding |
Each `Sound` is one voice, and the decoded sample is freed with the last
reference, which also stops it: keep the handle for as long as it should be
heard. Bytes that are none of the three formats raise
`ValueError: not a valid WAV, MP3 or Ogg audio stream` on every backend; past
that check, a stream that fails to decode stays silent. In the browser the
bytes are decoded by WebAudio in the background, so a `play()` made right
after the `Sound` is built starts once the decode finishes, a moment later;
natively the sample is decoded when the `Sound` is built.
```culebra
let WAV = Encoding.hex.decode("524946462800000057415645666d7420100000000100010040" +
"1f0000401f000001000800646174610400000080c8803c")
let blip = Audio.Sound.new(WAV)
blip.volume(0.5)
blip.play()
```
### Music
`Audio.Music.new(data, loop = true)` streams an MP3 or Ogg Vorbis file from
its bytes. Several can play at once (a loop under a sting), and the runtime
keeps each one fed from a thread of its own, so music plays on through a long
frame, a loading screen, or a program with no frame loop at all.
| Method | Effect |
| --- | --- |
| `music.play()` | start playing |
| `music.stop()` | stop and rewind to the start |
| `music.pause()` / `music.resume()` | pause, and pick up where it left off |
| `music.seek(seconds)` | jump to a position (negative and NaN mean the start) |
| `music.playing() -> Bool` | audible right now? |
| `music.volume(v)` / `music.pitch(p)` / `music.pan(p)` | as above |
Like a `Sound`, a `Music` stops when its last reference goes. Bytes that are
neither MP3 nor Ogg raise `ValueError: not a valid MP3 or Ogg audio stream` on
every backend; a stream that passes that check but fails to decode stays
silent. In the browser the file is decoded by WebAudio, and Ogg support there
depends on the browser (Safari historically decodes only MP3); natively both
formats always work.
```culebra
# doctest: skip
let bgm = Audio.Music.new(FS.read("music.ogg"))
bgm.volume(0.6)
bgm.play()
```
### PCM
`tone`, `Sound` and `Music` all play something already shaped as a note, a
sample or a file. `Audio.PCM` is the fourth kind: PCM (pulse-code modulation)
— sound as a list of numbers, the height of the wave measured `rate` times a
second — which the script synthesises a block at a time: an emulator's own APU
mixer, a chiptune resampled from its native rate, any signal a program builds
itself. `Audio.PCM.new(rate, channels, buffer)` opens a stream of `rate` Hz, 1
or 2 channels, fed `buffer` frames at a time (1024 is a reasonable default;
below ~512 is not supported). A `rate` or `buffer` past 2^31 − 1, or a
`buffer` too large to allocate, raises `ValueError` (`Audio.PCM: buffer is
too large`).
| Method | Effect |
| --- | --- |
| `pcm.ready() -> Bool` | `false` with no audio device: nothing plays, and the methods below still answer as they say |
| `pcm.needed() -> Long` | frames the stream can take now: `buffer` once a block has drained, `0` while both are still full, and always `0` with no device |
| `pcm.push(samples: Array) -> Long` | take the frames in `samples` (each value `-1.0..1.0`, clamped) into the pending block — mono is 1 value per frame, stereo is interleaved L,R pairs; answers how many frames it took, fewer than `samples` holds once the block is nearly full, so a caller that produced more than `needed()` keeps the rest for the next call; an element that is neither `Long` nor `Float` raises `TypeError` |
| `pcm.submit() -> Long` | hand the pending block to the stream and answer the frames handed: `0` when nothing was pending or the stream isn't ready for another block yet; with no device the block is emptied and the answer is `0` |
| `pcm.latency() -> Float` | seconds from a `submit` to the speaker: the two blocks ahead of it |
| `pcm.play()` / `stop()` / `pause()` / `resume()` / `playing() -> Bool` | transport |
| `pcm.volume(v)` / `pcm.pitch(p)` / `pcm.pan(p)` | as above |
```culebra
let pcm = Audio.PCM.new(44100, 1, 1024)
pcm.play()
mut phase = 0.0
Canvas.run(160, 160, fn () {
while pcm.needed() > 0 {
let block = iota(pcm.needed()).map(fn (_) {
phase += 440.0 / 44100.0
phase -= Math.floor(phase)
phase < 0.5 ? 0.2 : -0.2
})
pcm.push(block)
pcm.submit()
}
# … render …
true
})
```
At 60 fps a 44.1 kHz stream wants 735 frames a frame; producing and pushing a
whole block at once (rather than one call per sample) is what keeps this
affordable in a script's own per-frame budget. A `PCM` is silent in the
browser, where it answers as a machine with no audio device does.
### Capture
`Audio.Capture` is the microphone, and the mirror of `PCM`: the runtime
records into a ring holding the last second, and the script drains it a block
at a time. It needs no output device and no window.
`Audio.Capture.new(rate = 44100, channels = 1)` opens the default input at
`rate` Hz (a request the platform meets by resampling), 1 or 2 channels,
taking both as `PCM` does: a `rate` below 1 is 1, and one past 2^31 − 1 raises
`ValueError` (`Audio.Capture: rate is too large`). A new capture is stopped. Recording starts at `start()` and
frames the script does not read in time are dropped from the newest end.
A machine may have no microphone, or refuse the program one (a permission
the operating system asks the user for, on macOS and Windows). Neither is an
error: `ready()` is `false`, nothing ever arrives, and the first attempt prints
one warning. The same holds in any run with `CULEBRA_AUDIO` set to `off` or
`0`, and in the browser, which has no capture yet.
| Function / method | Effect |
| --- | --- |
| `Audio.capture_available() -> Bool` | whether the platform reports an input device (`false` when `CULEBRA_AUDIO` is `off`) |
| `mic.ready() -> Bool` | `false` when no input device could be opened: the methods below still answer as they say |
| `mic.start()` / `stop()` / `running() -> Bool` | recording control; `stop()` keeps what is already in the ring |
| `mic.waiting() -> Long` | frames in the ring, ready to read |
| `mic.read(frames = nil) -> Array` | take up to `frames` frames out of the ring (all that are waiting when omitted) as `Float`s in `-1.0..1.0`, stereo interleaved L,R, so `frames * channels` values at most; empty when nothing is waiting; a `frames` that is not a `Long` raises `TypeError` |
```culebra
# doctest: skip
let mic = Audio.Capture.new(44100)
mic.start()
loop {
let block = mic.read()
if !block.empty() {
let peak = block.map(|x| Math.abs(x)).max()
println("#".repeat(Math.round(peak * 60)))
}
Time.sleep(0.02)
}
```
### Kauai
> **Status: Draft.** The language and its API may still change before a
> release.
`Audio.Kauai` plays songs written in Kauai, a language for music as a band
plays it: chords and a melody, and grooves that say how the band accompanies
them ([the Kauai specification](kauai/language.md)). A song is read and
checked whole, then handed to the runtime, which starts each note on the
audio stream's own clock through `tone`'s channels, and plays the sounds the
program supplies for `host` instruments as samples mixed alongside them.
| Function | Effect |
| --- | --- |
| `Audio.Kauai.new(text: String, voices = nil, drums = nil)` | a song from its text, which uses no other files |
| `Audio.Kauai.load(path: String, voices = nil, drums = nil, dir = nil)` | a song file and the files it `use`s, from the disk or from `dir` (any `Dir`, or any object with `exists(name)` and `read(name)`) |
A song's mistake raises `KauaiError`, its message naming the file and the
line. The song answers `play()`, `stop()`, `volume(v)`, `playing()`,
`reached(mark)`, `prepare()`, `length()`, `about()` and `events()`, and keeps its own
time whether or not a device plays it; the specification's
[section 10](kauai/language.md#10-playing-from-culebra) has them all.
```culebra
let song = Audio.Kauai.new(`
band Solo {
voice Lead pulse plays melody vol 20
}
song Tune {
tempo 120
band Solo
melody in C5..B5
Theme
}
section Theme {
C e4 g c'2
}
`)
println(song.length()) # => 2.0
song.play()
```
## 41. `Dir`
A **read-only set of files**, wherever it lives — a directory on disk, files
held in memory, assets baked into the executable, or the entries of a ZIP
archive — read the same way. A function written against a `Dir` works with
any of them.
```culebra
let d = Dir.memory({'readme.md': '# hi', 'img/logo.png': 'PNG', 'img/bg.png': 'PNG'})
inspect(d.files()) # => ['img/bg.png', 'img/logo.png', 'readme.md']
inspect(d.read('readme.md')) # => '# hi'
inspect(d.list_dir('img')) # => ['bg.png', 'logo.png']
inspect(d.glob('**/*.png')) # => ['img/bg.png', 'img/logo.png']
inspect(d.is_dir('img')) # => true
```
### Getting one
| Factory | A Dir over |
| --- | --- |
| `Dir.disk(path: String \| Path) -> Dir` | the directory `path` on disk, read live; `IOError` when it is not a directory |
| `Dir.memory(files: Object) -> Dir` | the files of `files`: one key per file, `"sub/name.txt"`, to its contents (a `String`, binary-safe) |
| `Dir.embedded(name: String) -> Dir` | the directory `name` next to the entry script — baked into the executable by `culebra build` ([below](#embedded-assets)) |
| `Dir.zip(path: String \| Path = nil, *, bytes: String = nil) -> Dir` | the entries of a ZIP archive: the one in the file `path`, or the one held in `bytes` — give one of the two ([below](#zip-archives)) |
`Dir.memory` takes a flat Object: a file in a subdirectory is a key with a
`/`, not a nested Object. A nested Object or a value that is not a `String`
raises `TypeError`; a key outside the directory (`../x`, `/x`), two keys naming
one file (`a/b` and `a//b`), or a file that is also a directory (`a` and `a/b`)
raise `ValueError`. Keys are normalized when the Dir is made.
`type_of` names the kind (`DiskDir`, `MemoryDir`, `EmbedDir`, `ZipArchive`); a
parameter that takes any of them is annotated `Dir`.
### Reading
Every Dir answers the same nine methods:
| Method | Returns | |
| --- | --- | --- |
| `d.read(path)` | `String` | the file's bytes |
| `d.list_dir(path)` | `Array` | the bare names directly under the directory `path`, sorted |
| `d.exists(path)` | `Bool` | whether `path` is a file or a directory |
| `d.is_file(path)` | `Bool` | |
| `d.is_dir(path)` | `Bool` | |
| `d.size(path)` | `Long` | the file's size in bytes |
| `d.files()` | `Array` | every file's path, sorted |
| `d.glob(pattern)` | `Array` | the files whose path matches `pattern`, sorted |
| `d.copy_to(dest)` | `Nil` | writes every file under the disk directory `dest` |
A path is relative to the Dir's root and separated by `/`. `""` and `"."` name
the root itself; `.` and empty segments drop out, so `./a`, `a//b`, `a/./b`
and a trailing `/` name what `a` and `a/b` do. A path that starts with `/` or
`\`, or climbs with `..`, is simply not there: `exists` answers `false` and
`read` raises the `IOError` any missing file does, so nothing is read from
outside the Dir. `Dir.disk` follows a symbolic link while it leads somewhere
inside its root, and a link that leads out (or nowhere) is not there either;
`files()` walks a link that loops back once. `Dir.embedded` holds what
`culebra build` bakes, run from source or not: the regular files below the
directory, never a symbolic link, and a directory only while a file lies
under it. Otherwise a name is taken as written — `foo..txt` is an
ordinary file, and `\` is not a separator. Lists are sorted by bytes. Names
are compared as written in `Dir.memory`, `Dir.zip` and a baked
`Dir.embedded`; `Dir.disk`, and `Dir.embedded` run from source, follow the
operating system (case-insensitive on macOS and Windows by default).
`read` and `size` on a path that is no file, and `list_dir` on one that is no
directory, raise `IOError`, worded the same by every kind of Dir:
```culebra
let d = Dir.memory({'a.txt': 'alpha', 'img/x.png': 'x'})
println(try { d.read('b.txt') } catch e { e.message }) # => Dir.read: no such file 'b.txt'
println(try { d.read('img') } catch e { e.message }) # => Dir.read: 'img' is a directory
println(try { d.list_dir('a.txt') } catch e { e.message }) # => Dir.list_dir: 'a.txt' is a file
inspect(d.exists('../a.txt')) # => false
```
`glob` matches each `/`-separated segment of a file's path: `*` is any run of
characters, `?` one character, `[abc]`, `[a-z]` and `[!a]` one character of a
set, a character being a code point. A segment that is exactly `**` matches any number of segments, none
included, so `img/**` is every file under `img`. Only files are listed, never
directories (`FS.glob` lists both, and its `**` stands for directories only).
```culebra
let d = Dir.memory({'a.txt': '', 'b.md': '', 'img/x.png': '', 'img/sub/y.png': ''})
inspect(d.glob('*.txt')) # => ['a.txt']
inspect(d.glob('img/*')) # => ['img/x.png']
inspect(d.glob('**/*.png')) # => ['img/sub/y.png', 'img/x.png']
inspect(d.glob('[ab].*')) # => ['a.txt', 'b.md']
```
`copy_to(dest)` writes the Dir out to disk: every file lands at its path under
`dest`, with the directories made as needed and a file of the same name
overwritten. Extracting an archive is `Dir.zip(path).copy_to(dir)`, and
dumping the baked assets of a built binary is
`Dir.embedded('assets').copy_to(dir)`. A path that would land outside `dest` —
which only a Dir of your own can list — raises `ValueError` before it is
written.
### The `Dir` trait
`Dir` is also one of the built-in traits ([language.md](language.md), Built-in
traits). `read`, `list_dir`, `exists`, `is_file` and `is_dir` are required;
`files`, `glob`, `size` and `copy_to` are default methods written over them.
So a class of your own that answers the five is a `Dir`: it passes a `Dir`
annotation and gets the other four.
```culebra
class Greetings {
read(path) { "hello from {path}" }
list_dir(path) { path == '' ? ['a.txt', 'b.txt'] : [] }
exists(path) { self.is_file(path) || self.is_dir(path) }
is_file(path) { path == 'a.txt' || path == 'b.txt' }
is_dir(path) { path == '' }
}
fn count(d: Dir) {
d.files().size()
}
inspect(count(Greetings())) # => 2
inspect(Greetings().glob('b*')) # => ['b.txt']
```
A trait annotation accepts a class instance, and an Object literal is not one.
Much of the library asks for less than a Dir anyway: a reader that only looks
files up by name (`Audio.Kauai.load(dir:)`) takes anything with
`exists(path)` and `read(path)`, an Object literal included. To make a set of
literal files a full Dir, hand them to `Dir.memory`.
Every kind is Sendable: sent to an isolate, a `Dir.disk` reads the same
directory, a `Dir.memory` its own copy of the contents and a `Dir.embedded`
the same assets. A `Dir.zip` opens its archive again there, from its path or
its bytes, when it is first used: an archive whose file is gone by then raises
`IOError`, and a closed one arrives closed.
### Serving a Dir
`srv.static(mount, dir)` ([§15](#15-http)) and `Desktop.run(assets: dir)`
serve any Dir. The Content-Type comes from each file's extension, a request
for a directory (or for the mount itself) serves its `index.html`, and a path
that is no file falls through to the registered routes, so an API route always
wins.
```culebra
# doctest: skip — listens on a port until interrupted
let srv = Http.server()
srv.static("/", Dir.zip("site.zip")) # a whole site in one archive
srv.static("/files", Dir.disk("shared"))
srv.listen(8080)
```
The server answers on threads of its own, which do not share the program's
objects, so what it reads depends on the kind:
| Dir | What the server reads |
| --- | --- |
| `Dir.disk` | the directory itself, live: a file written later is served |
| `Dir.embedded` | the same assets: from disk when run from source, baked in a built binary |
| `Dir.memory` | a copy of the contents, taken when `static` is called |
| `Dir.zip` | the archive opened again (from a file) or a copy of its bytes, taken when `static` is called; closing the program's archive afterwards does not stop it |
| a class of your own | a copy in each of the server's workers, sent as a route handler is: it must be Sendable (`SendError` when the server starts), its `is_file` and `read` run there for each request, and a throw from them answers 500 |
### Embedded assets
`Dir.embedded(name)` resolves *per backend*, with no code change:
- **Run from source** (VM / JIT): it reads the live on-disk directory `name`,
resolved relative to the entry script — so editing a file and running again
shows the change immediately (a real dev loop).
- **`culebra build`** (AOT): the directory is walked at build time and its bytes
are baked into the executable; the binary reads them with no external files.
The build prints what it embedded (`embedded N file(s) (… bytes) from '…'`).
```culebra
# doctest: skip
let art = Dir.embedded("assets")
let sheet = Canvas.Sprite.from_png(art.read("sprites.png"))
let music = art.exists("music.ogg") ? Audio.Music.new(art.read("music.ogg")) : nil
music?.play()
```
The same Dir serves a whole directory over HTTP:
```culebra
# doctest: skip — listens on a port until interrupted
let srv = Http.server()
srv.static("/", Dir.embedded("dist")) # whole frontend, one line
srv.get("/api/ping", fn (req) {
'{"ok":true}'
})
srv.listen(8080)
```
`name` must be a string literal so the AOT build can find and bake it; a
computed name still works from source but isn't embedded. How `srv.static`
serves it is [above](#serving-a-dir). `Dir.embedded` is independent of `Http`
— a program that only reads assets needs no server.
`culebra build` compiles the baked assets against the culebra headers, so it
needs a source checkout: the one the binary was built from by default, or
`$CULEBRA_HOME` when that is set. With neither it stops with an error rather
than producing a binary.
### ZIP archives
`Dir.zip(path)` opens the archive in a file and reads only its central
directory; an entry's bytes are read when it is. `Dir.zip(bytes: data)` reads
an archive already in memory — downloaded, or one of a Dir's own files.
Packing goes the other way, with [`Compress.zip`](#17-compress).
```culebra
let z = Dir.zip(bytes: Compress.zip({'score.xml': '