# User's guide The `tei-rapporteur` workspace currently focuses on establishing the crate layout that underpins the rest of the roadmap. This guide summarizes what is available today and how to exercise it. ## Workspace overview - `tei-core` now models the top-level `TeiDocument` together with its `TeiHeader` and body-aware `TeiText`. The text model records ordered paragraphs (`P`), utterances with optional speaker references, and structural divisions (`Div`) containing paragraphs, utterances, lists (`List`/`Item`, where each `Item` may carry an optional inline `Label`), and nested subdivisions. Each `Div` keeps a required `@type` (`DivType`), an optional `@subtype`, an optional `@xml:id`, and an optional `Head` wrapper for a single leading `` element in the Episodic profile. Each block stores a sequence of `Inline` nodes, allowing clients to mix plain text with emphasized `` spans and `` cues without hand-rolling XML. Plain strings flow through `P::from_text_segments`, `Utterance::from_text_segments`, `Item::from_text_segments`, `Label::from_text`, and `Head::from_text`; the older `new` constructors remain as deprecated shims for existing callers where applicable. `TeiDocument` now exposes `validate()` to enforce document-wide rules: it rejects duplicate `xml:id` values across annotation systems, paragraphs, utterances, divisions, lists, and items, including nested divisions, and ensures utterance speakers appear in the profile cast when it exists. An empty cast still counts as declared, so every `who` fails until the speakers are populated, whereas the absence of a cast allows speaker references, so drafts can be validated incrementally. Identifier checks span the header as well, catching clashes between annotation systems and body blocks. Violations surface as `TeiError::Validation`. Utterances and list items now also carry local provenance and citation attributes where applicable, and XML deserialization remains strict for `` and ``: misspelt or unsupported attributes are rejected instead of being silently discarded. - `tei-xml` depends on the core crate and now covers both directions of XML flow. `serialize_document_title(raw_title)` still emits a `` snippet, `parse_xml(xml)` wraps `quick-xml` to materialize full `TeiDocument` values, and `emit_xml(&document)` now uses a hybrid emitter: header and stand-off sections are serialized via `quick_xml`, while body content is handwritten so mixed inline content and structural divisions round-trip correctly. All helpers return `TeiError`, so callers see consistent diagnostics whether parsing malformed input or attempting to emit control characters that XML forbids. - `tei-serde` centralizes JSON and `MessagePack` serialization, allowing the rest of the workspace to depend on a stable wrapper API (`tei_serde::json`, `tei_serde::msgpack`) instead of taking direct dependencies on `serde_json` and `rmp-serde`. It also publishes a versioned JSON Schema snapshot for `TeiDocument` under `schemas/tei-document.schema.vX.Y.Z.json` (with `schemas/tei-document.schema.json` tracking the latest snapshot), generated from the `tei-core` Rust types via `schemars`. Schema generation is gated behind the optional `tei-core` Cargo feature `json-schema` so consumers that do not need schema publication can avoid pulling in `schemars`. - `tei-py` now ships the `tei_rapporteur` PyO3 module. The exported `Document` class wraps `TeiDocument`, validates titles via the Rust constructors, and exposes a `title` getter plus an `emit_title_markup` convenience method. The module also surfaces a top-level `emit_title_markup` function so Python callers mirror the Rust helper without reimplementing validation rules. The MessagePack bridge exposes both `from_msgpack` and `to_msgpack` for binary interchange. Dictionary exchange is available via `from_dict`/`to_dict`, powered by `pyo3-serde`, so Python built-ins can cross the FFI boundary without detouring through JSON text. Phase 2.2 adds `parse_xml`/`emit_xml` bindings that forward TEI strings directly to the `tei-xml` helpers. Python can now parse canonical TEI without detouring through MessagePack, and emission always routes through the same forbidden-character guardrails as the Rust callers. Python-facing errors are surfaced as `ValueError` for content issues and `TypeError` when callers pass the wrong objects to the bindings. - `tei-test-helpers` captures assertion helpers that multiple crates reuse in their unit and behaviour-driven tests. - `pyproject.toml` configures `maturin` to build `tei-py`, allowing `maturin develop` or `maturin build` to work from the workspace root without additional arguments. ## Building and testing Use the Makefile targets to work with the entire workspace: - `make build` compiles every crate in debug mode. - `make test` runs all unit tests and the behaviour tests powered by `rstest-bdd`. - `make check-fmt`, `make lint`, and `make fmt` mirror the repository quality gates described in `AGENTS.md`. - `make json-schema` regenerates the published `TeiDocument` JSON Schema snapshots under `schemas/`. - `make validate-xml` generates XML fixtures and validates them against the TEI Episodic Profile Relax NG schema using `jing`. This requires jing to be installed (see "External XML validation" below). ## Behavioural guarantees `tei-core` and `tei-xml` ship behaviour-driven tests that exercise happy and unhappy paths. Core scenarios validate that header metadata can be assembled, that blank revision notes are rejected, and that the body model preserves paragraph/utterance/division order while rejecting empty utterances and invalid division types. Additional cases demonstrate inline emphasis, rend-aware mixed content, pause cues with duration metadata, and ensure empty `<hi>` segments are rejected. Division-specific tests cover `Div` construction with validated `@type`, `List` and `Item` assembly, `Label` prefix content, `@n` and `@corresp` attribute handling on items, and `xml:id` uniqueness checks across nested division content. The XML crate now tests title serialization, full-document parsing, and XML emission: feature files cover successful parsing, missing header errors, syntax failures triggered by truncated documents, as well as emission of canonical minimal TEI output and the error surfaced when a document sneaks in forbidden control characters. These tests run alongside the unit suite, so developers receive fast feedback when modifying the scaffolding. The `tei-py` suite layers on `rstest-bdd` scenarios for the Python module, covering successful construction of `Document` from a valid title, rejection of blank titles via `ValueError`, round-tripping markup through the module-level helper, both directions of the MessagePack bridge, and the new XML exchange APIs. Behaviour-driven coverage now parses canonical TEI fixtures, rejects malformed payloads, emits canonical strings, and proves forbidden characters bubble up as `ValueError` with an actionable message. New dictionary scenarios cover happy-path decoding, missing fields, blank titles, and the `TypeError` raised when `to_dict` is called with the wrong object. New validation scenarios assert that duplicate `xml:id` values are rejected and that utterance speakers must be declared when a profile cast exists, while documents without a cast still pass validation. The `tei-serde` crate now publishes a versioned JSON Schema for `TeiDocument`. Its unit tests assert that the checked-in schema snapshot stays in sync with the generated output, and its behaviour tests validate both happy paths (serialized documents satisfy the schema) and unhappy paths (missing required fields and unknown inline properties are rejected). The `tei-serde` crate also includes property-based tests using `proptest` to verify round-trip integrity between formats. These tests generate arbitrary valid `TeiDocument` instances and confirm that serialization to JSON, MessagePack, and XML preserves equality when deserialized. The property-based test suite complements the example-based tests by exercising edge cases that handwritten fixtures might miss, such as documents with many blocks, deeply nested inline elements, and titles containing punctuation. Run `make test` to execute all tests, including the property suite. ## Python bindings The workspace now provides a ready-to-build Python wheel. `pyproject.toml` declares `maturin` as the build backend and targets `tei-py/Cargo.toml`, so the workflow looks like: ```bash python -m pip install --upgrade pip maturin maturin develop # builds and installs tei_rapporteur into the active venv python -c "import tei_rapporteur as tr; print(tr.Document('Wolf 359').title)" ``` Within Python, `tei_rapporteur.Document` constructs a validated TEI document by wrapping the Rust `TeiDocument`. The class exposes a `.title` property and an `emit_title_markup()` method that mirrors the Rust helper. The module also offers a top-level `emit_title_markup(title: str)` so scripting callers can work without instantiating a document. CI now builds the wheel on Ubuntu, installs it via `pip`, and imports the module to ensure the PyO3 glue remains healthy. Python data classes now live in `tei_rapporteur.structs`. The submodule defines `msgspec.Struct` projections (`Episode`, `TeiHeader`, `FileDesc`, `Paragraph`, `Utterance`, `DivBlock`, `ListBlock`, `Item`, `Label`, `StandOff`, `SpanGroup`, `Span`, and the citation-declaration types) that mirror the Python-facing Rust projection. Inline nodes decode into plain Python objects, and TEI pointer-list attributes such as `source`, `resp`, `corresp`, and `ana` are exposed as `list[str]` instead of TEI's whitespace-separated attribute strings. MessagePack emitted by `to_msgpack` decodes directly into these classes, and encoding them feeds the payload straight back into `from_msgpack`. Structural body content is exposed through tagged unions: - `BodyBlock = Paragraph | Utterance | DivBlock` - `DivContent = Paragraph | Utterance | ListBlock | DivBlock` - `Event = DocumentStart | HeaderEvent | ParagraphEvent | UtteranceEvent | DivEvent | DocumentEnd` `DivBlock` and streamed `DivEvent` values now expose `div_type`, optional `subtype`, optional `head`, optional `xml_id`, and recursive `content`, so chapter markers, guest-bio sections, and sponsor-read sections can be modelled without flattening the hierarchy into paragraphs. Typed `msgspec` decoding lets callers inspect nested divisions without first round-tripping through XML or untyped dictionaries: ```python import msgspec import tei_rapporteur as tei from pathlib import Path from tei_rapporteur.structs import DivBlock, Episode, Item, Label, ListBlock tei_xml = Path("episode.tei.xml").read_text(encoding="utf-8") document = tei.parse_xml(tei_xml) payload = tei.to_msgpack(document) episode = msgspec.msgpack.decode(payload, type=Episode) body_block = episode.text.body.blocks[0] if isinstance(body_block, DivBlock): print(body_block.div_type) for child in body_block.content: if isinstance(child, DivBlock): for nested in child.content: if isinstance(nested, ListBlock): first_item = nested.items[0] if isinstance(first_item, Item) and isinstance(first_item.label, Label): print(first_item.label.content[0].value) ``` Citation metadata is split along TEI-native boundaries. Canonical citation declarations live under `header.encoding_desc.refs_decl`, utterance-local provenance stays on `Utterance`, and many-to-many overlays live in the optional root `Episode.stand_off` layer via `SpanGroup` and `Span`. Binary interchange is now supported through `tei_rapporteur.from_msgpack(payload: bytes)`. The helper accepts the bytes produced by `msgspec.msgpack.encode` (or any compatible encoder), decodes them via `tei-serde` (wrapping `rmp-serde`), and returns a `Document`. Invalid payloads raise `ValueError`, so Python callers receive a familiar exception instead of a Rust-specific error type. This allows workflows such as: ```python import msgspec import tei_rapporteur as tei from tei_rapporteur.structs import Episode, FileDesc, TeiBody, TeiHeader, TeiText episode = Episode( header=TeiHeader(file_desc=FileDesc(title="Bridgewater")), text=TeiText(body=TeiBody()), ) payload = msgspec.msgpack.encode(episode) document = tei.from_msgpack(payload) print(document.title) ``` The inverse helper, `tei_rapporteur.to_msgpack(doc: Document)`, serializes the validated document into MessagePack bytes via `tei_serde::msgpack`. The function returns Python `bytes`, making it trivial to persist the payload or feed it straight into `msgspec.msgpack.decode` to hydrate a structured type. Non-`Document` inputs raise a `TypeError`, giving users immediate feedback when they miswire a call. A complete round trip therefore looks like: ```python doc = tei.Document("Bridgewater") payload = tei.to_msgpack(doc) from tei_rapporteur.structs import Episode episode = msgspec.msgpack.decode(payload, type=Episode) ``` For JSON-style hand-offs, `tei_rapporteur.from_dict(payload)` and `tei_rapporteur.to_dict(doc)` use `pyo3-serde` to bridge Python built-ins and the Rust `TeiDocument`. The helpers accept any mapping/sequence tree that would be valid JSON, raising `ValueError` when required fields are missing or titles are blank and `TypeError` when a non-`Document` is passed. The output of `to_dict` matches what `msgspec.to_builtins` produces, so callers can stay with native Python objects: ```python doc = tei.Document("Bridgewater") payload = tei.to_dict(doc) assert payload["teiHeader"]["fileDesc"]["title"] == "Bridgewater" round_tripped = tei.from_dict(payload) ``` When scripts already have TEI XML on disk, the new `tei_rapporteur.parse_xml` and `tei_rapporteur.emit_xml` functions avoid redundant conversions. `parse_xml` hands the string straight to the Rust parser, returning a `Document` that holds the validated `TeiDocument`. `emit_xml` performs the inverse operation and retains the forbidden-character guardrails enforced by `tei-xml`. A typical round trip combining XML and Python struct manipulation therefore looks like: ```python from pathlib import Path import msgspec import tei_rapporteur as tei from tei_rapporteur.structs import Episode doc = tei.parse_xml(Path("episode.tei.xml").read_text()) payload = tei.to_msgpack(doc) episode = msgspec.msgpack.decode(payload, type=Episode) episode.title = "Wolf 359 Reissue" doc = tei.from_msgpack(msgspec.msgpack.encode(episode)) xml = tei.emit_xml(doc) ``` The BDD tests now cover successful decoding, encoding, XML parsing, emission, and the corresponding error paths, ensuring the entry points remain reliable as the API expands. For spoken-runtime estimation, use `tei_rapporteur.spoken_text_segments(xml)` instead of traversing XML locally. The function accepts a complete TEI document string and returns `tei_rapporteur.structs.SpokenTextSegment` objects with `text`, `locator`, and `xml_id` fields. It includes performed text from `<p>`, `<ab>`, `<l>`, direct `<u>` content, and standalone `<seg>` in spoken context. It excludes speaker labels, stage directions, notes, lists, labels, headings, references, bibliography, show-note divisions (`<div type="notes">`), TEI header metadata, and stand-off metadata. Malformed XML and unsupported body markup raise `ValueError`; the API never falls back to raw-text counting. ```python import tei_rapporteur as tei xml = """ <TEI> <teiHeader><fileDesc><title>Episode Host

Hello there.cut?

Link dump.

 """ segments = tei.spoken_text_segments(xml) assert segments[0].text == "Hello there." assert segments[0].locator == "/TEI/text/body/sp[1]/p[1]" assert segments[0].xml_id == "line-1" ``` ### Document validation The `Document` class exposes a `validate()` method that performs document-wide integrity checks. It verifies that all `xml:id` values are unique across the document (including annotation systems, stand-off span groups, stand-off spans, paragraphs, utterances, divisions, lists, and items), that utterance speaker references match the declared cast list when present, that `refsDecl` entries keep their required `@match` and `@property` values, and that internal `#id` pointers in utterance, item, and stand-off provenance attributes resolve against existing identifiers. ```python import tei_rapporteur as tei doc = tei.from_dict(payload) try: doc.validate() print("Document is valid") except ValueError as e: print(f"Validation failed: {e}") ``` Validation raises `ValueError` with a descriptive message when: - Duplicate `xml:id` values are detected across the document - An utterance references a speaker not declared in the profile cast - A speaker is referenced when the profile has an empty cast (an empty cast still counts as declared, so all speaker references fail until the cast is populated) - A `citeStructure` or `citeData` declaration leaves a required attribute blank - A `Div` leaves `@type` blank after trimming - A stand-off `spanGrp` leaves `@type` blank after trimming - A stand-off `span` omits both `@target` and `@from`, or uses `@to` without `@from` - A `#`-prefixed pointer in `source`, `resp`, `corresp`, `ana`, `target`, `from`, or `to` does not resolve to a known `xml:id` Documents without a profile cast allow speaker references without validation, enabling incremental validation of draft documents. ### Correspondence pointers `@corresp` values follow TEI pointer semantics. A value beginning with `#` is an internal pointer and must resolve to an `xml:id` in the same TEI document. Use this form only when the referenced node is materialized in the document. Validation rejects unresolved internal pointers so callers do not accidentally ship dangling local references. External identifiers such as `urn:...`, `tag:...`, or `https://...` may be used when the target lives outside the TEI document. Repository-owned objects, including Episodic reference-document revisions, should use an external identifier in `@corresp` unless that object is also represented in the same TEI document with an `xml:id`. Guest biographies therefore link to their source reference revision as an external correspondence: ```xml Mathematician and computing pioneer. ``` `tei-rapporteur` currently supports `@corresp`, `@n`, and `xml:id` on list items. `@source` on `Item` is not part of the public body model yet; it may be considered later if callers need stricter provenance semantics beyond the current correspondence link. ## Text Encoding Initiative (TEI) Episodic Profile schema The TEI Episodic Profile is formally documented in an ODD (One Document Does it all) specification at `schemas/tei-episodic-profile.odd`. This specification: - Defines the exact elements and attributes permitted in the profile - Includes Schematron rules for validation constraints such as unique `xml:id` values and speaker cross-referencing - Can be processed by TEI tools (Roma, TEI Stylesheets) to generate Relax NG and Schematron schemas for external validation - Ships with a pre-generated Relax NG schema at `schemas/tei-episodic-profile.rng`. Rust callers can retrieve it via `tei_xml::relax_ng_schema()` or write it to disk using `tei_xml::write_relax_ng_schema(path)` before invoking external validators such as `jing`. The profile supports: - **Header metadata**: title, speaker declarations, annotation systems, canonical citation declarations (`refsDecl` / `citeStructure` / `citeData`), revision history - **Body structure**: paragraphs (`

`), utterances (``) with optional speaker attribution via `@who` plus local provenance attributes (`@n`, `@source`, `@resp`, `@cert`, `@corresp`, `@ana`), and thematic divisions (`

`) with `@type` (required and validated via `DivType`), optional `@subtype`, optional `@xml:id`, and an optional `Head` wrapper for a single leading `` element. Divisions can contain paragraphs, utterances, lists (``), and nested divisions. Lists hold ordered items (``) that carry optional `@n` (numbering or timestamp metadata), `@corresp` (pointer list for cross-references), and `@xml:id`. Each item may include an optional label prefix (`