---
title: Director-note cues with Pipecat, timed to word timings
description: "Make an Anam avatar perform on cue inside a Pipecat pipeline — write inline director notes, let Cartesia time them for you, and send them to Anam over the data channel."
tags: [python, pipecat, director-notes, avatar]
date: 2026-07-12
authors: [sr-anam]
---

[Director notes](https://anam.ai/docs/personas/director-notes) let a Cara-4 avatar *perform* its speech — a laugh on the funny line, concern on the caveat. In a turnkey session you write inline `[tag]` cues in the text and Anam handles them. But in a Pipecat pipeline with audio passthrough, Anam only receives **audio**, not text — so it can't see your tags. You send cues yourself, over the WebRTC data channel, each stamped with the moment it should fire.

This recipe wires that up: a Pipecat agent (Deepgram STT, OpenAI LLM, Cartesia TTS, Anam avatar via `AnamTransport`) that reads Cartesia's word timestamps and fires each cue on the word it belongs to.

The complete code is at [examples/pipecat-director-notes](https://github.com/anam-org/anam-cookbook/tree/main/examples/pipecat-director-notes).

## The timing problem

A cue over the data channel is a small JSON message:

```json
{ "message_type": "director_note_cue", "cue": { "tag": "laughter" }, "at_seconds": 1.2 }
```

`at_seconds` is an offset **from the start of the current speech turn**. So to land a laugh on the word "hilarious", you need to know *when* the avatar will say "hilarious" — before it says it, with enough lead time for the message to reach Anam.

## Cartesia hands you the timing

Here's the shortcut: Cartesia (Sonic) accepts inline `[tag]` markers in the transcript and **echoes them straight back in its word timestamps**, each stamped with the moment it's spoken, with respect to the TTS audio this turn. One turn is one Cartesia context, so those offsets are already turn-relative — exactly your `at_seconds`.

Send `"Hello [laughter] everyone today."` with `add_timestamps` on, and the timestamps come back like this:

```python
words  = ['Hello', '[laughter]', 'everyone', 'today.']
starts = [ 0.122,  0.706,        1.219,      1.721]
```

The `[laughter]` marker comes back as its own token, timed to where it sits — `0.706`, just ahead of "everyone". That start time *is* your `at_seconds`; no anchor-matching required. You asked Cartesia to speak the marked text, and it told you exactly when each mark lands. (The tag takes its own short slot in the audio; glue it to the next word — `[laughter]everyone` — and Cartesia folds them into one token with no gap, if you prefer.)

So there's no stripping-and-re-deriving from neighbouring words: the TTS already did the alignment. You read it off.

## Reading the cue

Two small pieces. `cues.py` pulls the tag(s) out of a Cartesia token — pure logic, so `python cues.py` self-checks:

```python
_TAG_RE = re.compile(r"\[([a-zA-Z_]+)\]")

def cue_tags(token):
    """Valid Anam tags in a Cartesia word token: '[happy]' -> ['happy']."""
    return [t for m in _TAG_RE.finditer(token) if (t := m.group(1).lower()) in CUE_TAGS]
```

The TTS subclass overrides one method — `add_word_timestamps`, which Cartesia calls with the raw word timings ahead of playback — and fires a cue for any token that carries one:

```python
class DirectorCueCartesiaTTSService(CartesiaTTSService):
    def __init__(self, *args, on_cue, **kwargs):
        super().__init__(*args, **kwargs)
        self._on_cue = on_cue

    async def add_word_timestamps(self, word_times, context_id=None, *args, **kwargs):
        for word, at_seconds in word_times:                 # [(word, start_seconds), ...]
            for tag in cue_tags(word):
                await self._fire(tag, max(0.0, float(at_seconds)))
        await super().add_word_timestamps(word_times, context_id, *args, **kwargs)  # keep normal behaviour
```

`on_cue` is a callback, so the TTS never has to know about the transport; `_fire` wraps it in a try/except so a cue that arrives before the session is live can't kill the pipeline.

## Authoring cues

Write the tag in square brackets **just before the word it should land on**. It comes back as its own token, timed to that spot:

```
"Hi [warm] there! [concerned] Honestly, that story was [laughter] hilarious."
```

The system prompt teaches the LLM the tag vocabulary and where to place cues (just before the word), so it can add its own as the conversation goes.

## Wiring it up

`AnamTransport.send_director_note_cue(tag, at_seconds=...)` sends the cue on the active Anam session. Wire it straight into the TTS as `on_cue`:

```python
transport = AnamTransport(
    api_key=os.environ["ANAM_API_KEY"],
    persona_config=PersonaConfig(
        avatar_id=os.environ["ANAM_AVATAR_ID"],
        avatar_model="cara-4",            # director notes require Cara-4
        enable_audio_passthrough=True,
    ),
    daily_room_url=os.environ["DAILY_ROOM_URL"],
)

tts = DirectorCueCartesiaTTSService(
    api_key=os.environ["CARTESIA_API_KEY"],
    on_cue=transport.send_director_note_cue,   # cue → Anam, over the data channel
    settings=CartesiaTTSService.Settings(voice="e8e5fffb-252c-436d-b842-8879b84445b6"),
)
```

Drop `tts` into the pipeline in its usual spot (`... llm, tts, transport.output(), ...`). Set a baseline delivery for the whole session with `PersonaConfig(director_notes=DirectorNotes(preset_style="warm", expressivity=0.7))`; the per-word cues shift it for the moments you mark.

## Running it

```bash
cd examples/pipecat-director-notes
uv sync
cp .env.example .env   # add your keys and DAILY_ROOM_URL
uv run python bot.py
```

Open your `DAILY_ROOM_URL` in a browser to join. On connect the avatar opens with a short LLM greeting — primed to use a cue so you can see one fire — and adds more as the conversation goes (it's told the tag vocabulary in the system prompt).

## Caveats

- **Cara-4 only.** Director notes need a Cara-4 avatar; stock avatars default to Cara-3.
- **Cartesia sees the tags too.** The markers go to Cartesia unchanged. An emotion tag it doesn't recognise (`[warm]`, `[concerned]`, …) is absorbed into the word as a timing marker with negligible audio effect; a non-verbal it *does* know — notably `[laughter]` — it will also render in the voice. Usually a plus (voice and face align), but test your tag set, and expect the markers to show up in the assistant transcript.
- **A spaced tag takes its own slot.** `[tag] word` comes back as its own token — its start is the cue's `at_seconds`, a beat ahead of the word, with a short slot in the audio. Glue it (`[tag]word`) and Cartesia folds it into the word with no gap instead; the code reads both.
- **One turn, one context.** `at_seconds` is relative to the turn's Cartesia context, which spans the whole turn (`reuse_context_id_within_turn`). The echoed start times are already in that frame — nothing to reset.
- **Cues need a live session.** `send_director_note_cue` raises if the Anam session isn't active yet, so the demo runs its opening LLM turn from `on_avatar_connected` — which fires only after the session is up (reactive turns are naturally post-connection). `_fire` still wraps the call in a try/except, so a cue that races a session drop is logged and skipped, never fatal.
- **Provider-specific.** This leans on Cartesia echoing inline markers in its timestamps (verified on `sonic-3.5`). A TTS that strips unknown markers, or omits them from timestamps, won't work this way — you'd fall back to matching a cue against a neighbouring word's timing.