# Terminal-Bench Integration

Run [Terminal-Bench 2.0](https://www.tbench.ai/) against Orca using [Harbor](https://www.harborframework.com/).

## Prerequisites

```bash
uv tool install harbor
```

Docker must be running (Harbor spins up containers per task).

## Quick Start (Recommended)

Build a static musl binary first, then mount it into containers:

```bash
# Build static Linux binary (one-time, ~7 min)
docker run --rm -v "$(pwd):/src" -w /src \
  messense/rust-musl-cross:x86_64-musl \
  cargo build --release --target x86_64-unknown-linux-musl --bin orca

# Remove what earlier runs left behind (see below)
scripts/eval/reap_harbor.sh

# Run benchmark (2 trials per task)
harbor run \
  -d "terminal-bench/terminal-bench-2" \
  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
  -k 2 \
  --mounts '[{"type":"bind","source":"'"$(pwd)"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]'

# ... and what this run left behind
scripts/eval/reap_harbor.sh
```

Harbor abandons the containers of a trial whose environment did not start in time, and
they go on building or downloading while later trials start (issue #129).
`scripts/eval/reap_harbor.sh` removes them and their networks; run it with no Harbor run
in progress.

## Single Task Smoke Test

```bash
harbor run \
  -d "terminal-bench/terminal-bench-2" \
  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
  -k 2 \
  --mounts '[{"type":"bind","source":"'"$(pwd)"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]' \
  --include-task-name "terminal-bench/openssl-selfsigned-cert"
```

## Configuration

The API key is read from `~/.orca/auth.json` (`DEEPSEEK_API_KEY` field). Falls back to `ORCA_API_KEY` env var.

| Variable | Default | Description |
|----------|---------|-------------|
| `ORCA_BASE_URL` | `https://api.deepseek.com` | API endpoint |
| `ORCA_MODEL` | `deepseek-flash` | Model to use |
| `ORCA_REASONING_EFFORT` / `DEEPSEEK_REASONING_EFFORT` | Orca's own default (`max`) | Reasoning effort forwarded into the container: `low`, `high`, or `max` |
| `ORCA_PROVIDER` | configuration default | Provider override forwarded into the container |

## Model latency and the timeout class

TB2 task budgets are fixed (`[agent] timeout_sec` = 900–3600 s) and the median trial
spends ~60 % of its wall clock inside model generation, so token volume per turn — not
tool latency — is what decides the timeout class (issue #64). Two consequences for
benchmark runs:

- Use the reasoning-effort lever above to A/B latency under one budget. Measured on the
  same 16 tasks, `ORCA_REASONING_EFFORT=low` cut `video-processing` from 1295 s /
  135 k output tokens to 297 s / 32 k at the same reward, `mcmc-sampling-stan` from
  1773 s / 72 k to 683 s / 10 k, and turned `gcode-to-text` (900 s timeout) into a pass
  in 735 s — but it also lost `model-extraction-relu-logits` and `qemu-alpine-ssh`, so
  the lever has to be measured per task rather than assumed to be free.
- `agent/execution_metadata.json` records `duration_seconds` plus the largest `usage`
  counters (`turns`, `output_tokens`, `input_tokens`, `cache_tokens`, `cost_usd_micros`)
  seen in the stream, which exist even for a trial killed at the task timeout, because
  the task status events carry the running totals.


## Why musl?

Terminal-Bench containers use various base images with different glibc versions.
The release binary (`x86_64-unknown-linux-gnu`) requires glibc 2.39+, which many
task containers lack. The static musl binary works universally.

## CA certificates in the task container

Orca checks TLS certificates against the operating system's trust store and, since
its move to reqwest 0.13, has no root certificates compiled in. The adapter copies
the static binary into each task container and runs it there, so the trust store
that matters is the container's, not the host's.

- A task image without a CA bundle (no `ca-certificates` package) would leave Orca
  unable to build any HTTP client, so every turn would fail with
  `No CA certificates were loaded from the system`. Plain `http://` endpoints would
  fail too: no client is built, so nothing is sent.
- A task that edits the container's CA store can also break Orca's own connection to
  the model API, because Orca reads the store again for every request to the model.

So the adapter ships a CA bundle with the binary: during setup it copies the bundle of
Harbor's own Python environment (`certifi`) to `/usr/local/share/orca/cacert.pem` in
the container. When Orca starts, the bundle is used only if the container has no CA
bundle of its own (none of the bundle files Orca looks for on Linux, such as
`/etc/ssl/certs/ca-certificates.crt`, exists and is non-empty) and `SSL_CERT_FILE` is
not already set; then the adapter sets `SSL_CERT_FILE` to it. Commands the agent runs
inherit that variable, so in such a container they trust the same bundle. A container
that has a trust store keeps using it unchanged. If Harbor's environment has no
`certifi`, or the copy fails, setup goes on without the bundle.

## Filtering Tasks

```bash
# Limit the number of tasks
harbor run -d "terminal-bench/terminal-bench-2" \
  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
  --n-tasks 10 \
  --mounts '[{"type":"bind","source":"'"$(pwd)"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]'

# Single task
harbor run -d "terminal-bench/terminal-bench-2" \
  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
  --include-task-name "terminal-bench/build-pov-ray" \
  --mounts '[{"type":"bind","source":"'"$(pwd)"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]'
```

## Package Setup

The adapter is exposed to Harbor via a `pyproject.toml` at the repo root.
After modifying the adapter code, reinstall:

```bash
uv tool install --force harbor \
  --with-editable . \
  --index-url https://pypi.org/simple/
```

## Agents

| File | Class | Description |
|------|-------|-------------|
| `orca_agent.py` | `OrcaInstalledAgent` | Copies musl binary from mount into container. |
| `orca_external.py` | `OrcaExternalAgent` | Uses a pre-built Orca binary on the host. |

The installed adapter writes Orca's raw JSONL output to the trial's
`agent/trajectory.jsonl` artifact for `harbor analyze` and `harbor view`.
Command output is not assigned to `AgentContext`: Harbor 0.20.0 does not define
an `output` field, and adding one aborts the trial before verifier execution.

Both adapters pipe each task instruction to `orca exec` on stdin. This keeps the
instruction out of the Orca process's positional arguments, so task cleanup
commands that match `/proc` or the process table cannot match the session on a
token from its own instruction. Direct CLI callers can continue using
`orca exec -- "..."`; stdin is the safer form for untrusted or arbitrary task
text.

The trajectory is persisted on **every** exit path. Harbor discards the
`ExecResult` when the command exits non-zero and when the exec is killed at the
task timeout, so the adapter also tees the stream to `/tmp/orca-trajectory.jsonl`
inside the container and downloads it after the run. `agent/execution_metadata.json`
records the real `exit_code`, `trajectory_bytes`, and a `trajectory_persisted`
flag, so a 0-byte artifact is visible instead of being reported as success.

## Quarantined tasks

Some dataset tasks cannot be scored no matter what the agent does, so they are excluded
from reported means instead of silently entering the score as a 0. The list lives in
[`quarantine.json`](quarantine.json); each entry records the task, the reason, and the
evidence path.

| Task | Why |
|------|-----|
| `terminal-bench/qemu-startup` | The task image's pinned Debian `bullseye-security` packages (`libcurl4`, `libnghttp2-14`) have been superseded, so the verifier's own `apt-get install` 404s and `/tests/test.sh` aborts (`curl: command not found`, `uvx: command not found`) before it can score. See `jobs/regression-20260916/qemu-startup__fLTBLJS/verifier/test-stdout.txt`. |

Exclude quarantined tasks from a run with Harbor's own filter:

```bash
harbor run -d "terminal-bench/terminal-bench-2" \
  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
  --exclude-task-name "terminal-bench/qemu-startup" \
  --mounts '[{"type":"bind","source":"'"$(pwd)"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]'
```

This is an upstream `terminal-bench-2` image defect, not an Orca one: the adapter's own
package step cannot repair a task's `test.sh`, and the image re-installs unconditionally.
Remove an entry once the dataset pins a working image.