![NEEDLE](docs/hero.png) # 🧡 NEEDLE [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![Rust](https://img.shields.io/badge/rust-toolchain-orange.svg)](rust-toolchain.toml) [![iad-ci mirror status](https://img.shields.io/github/checks-status/jedarden/NEEDLE/main?label=iad-ci)](docs/post-push-ci.md) [![Version from GitHub artifact mirror](https://img.shields.io/github/v/release/jedarden/NEEDLE)](https://github.com/jedarden/NEEDLE/releases/latest) **N**avigates **E**very **E**nqueued **D**eliverable, **L**ogs **E**ffort > Deterministic bead processing with explicit outcome paths. NEEDLE is a wrapper for headless coding CLI agents. It processes a shared bead queue in deterministic order, dispatching work through its supported built-in adapters (Claude Code, OpenCode, Codex, and Aider) and handling every outcome through an explicit, predefined path. --- ## πŸš€ Quickstart Prerequisites: `git`, `tmux`, and an agent CLI on your `PATH` β€” the flow below uses [Claude Code](https://claude.ai/code) (`claude`). Prebuilt binaries are Linux x86_64; everything else builds from source (see below). ```bash # 1. Install needle, its transform helpers, and the bead-rs backend. # Release assets are published on the GitHub mirror; source remains Forgejo. curl -fsSL https://github.com/jedarden/NEEDLE/releases/latest/download/install.sh | bash # 2. Initialize your repo with the bead-rs backend cd && needle init --backend bead-rs # 3. Make sure your branch has an upstream β€” the shipped-work gate (on by # default) verifies that each closed bead's commit was pushed git push -u origin "$(git branch --show-current)" # 4. Create the bead store bead init --prefix # 5. Create your first bead with one deliverable and one acceptance command bead create \ --title 'Add a CONTRIBUTING.md so that test -s CONTRIBUTING.md passes. Work on this issue directly.' \ --description 'Create CONTRIBUTING.md with contribution guidelines. Acceptance: `test -s CONTRIBUTING.md`. Do not create sub-issues, split this work, or decompose it.' \ --priority 2 # 6. Verify system health needle doctor # 7. Run a worker needle run --agent claude --identifier alpha # 8. List, check status, and attach to the session needle list needle status needle attach alpha # 9. Verify completion bead list --status closed test -s CONTRIBUTING.md git rev-list origin/"$(git branch --show-current)"..HEAD ``` The completed run shows the bead as `closed`, `test -s CONTRIBUTING.md` exits with status 0, and the final `git rev-list` prints nothing because the shipped commit is already pushed. For the complete lifecycleβ€”including `stop`, `resume`, safe orphan cleanup, scoped versus global actions, and workers that outlive tmuxβ€”see [Worker Lifecycle and Safe Cleanup](docs/operations/worker-lifecycle.md). **Heads-up:** the built-in `claude` adapter invokes `claude -p … --dangerously-skip-permissions`. Unattended operation means no permission prompts; read `needle config` before pointing a worker at a repository you care about. **Shipped-work gate (step 3):** `worker.enforce_shipped_work` is `true` by default. Before NEEDLE accepts a closure it checks that the dispatch produced a commit β€” on a path outside `notes/` and `.beads/` β€” that has been **pushed to the branch's upstream** (agents are instructed to commit and push; a bead note explaining why no code was needed also passes). In a checkout shared by several workers the gate judges the dispatch's own commits, not whatever sits at `HEAD`: another bead's not-yet-pushed commit on top does not fail it, and a commit rebased to a new SHA counts once an identical patch is on the upstream. A workspace whose branch has no upstream β€” a plain `git init`, or `git remote add` without `git push -u` β€” cannot be checked at all, so the quickstart's `git push -u` in step 3 is not optional. For a repository that is genuinely local-only, disable the gate instead: ```yaml # ~/.config/needle/config.yaml worker: enforce_shipped_work: false ``` **Built-in validation gate for unguarded workspaces:** A workspace whose own `.needle.yaml` declares neither `gates:` nor legacy `verification:` enters the no-explicit-gate path. An explicit `gates: []` or `verification: []` remains an opt-out. NEEDLE first lets the language-default resolver supply a `default_*` gate; only when it supplies none does the shipped **interim fallback gate** select a verifier from the committed workspace, in this exact order: 1. `scripts/definition-of-done.sh` β€” run `scripts/definition-of-done.sh`. 2. `go.mod` β€” run `go build ./... && go vet ./... && go test -short ./...`. 3. `Cargo.toml` β€” run `cargo build --all-targets && cargo test`. 4. `package.json` with a non-empty `scripts.test` β€” run `npm test`. 5. `pyproject.toml` β€” run `pytest -q`. 6. `pytest.ini` β€” run `pytest -q`. The selected command runs in a clean `git archive HEAD` extraction, so it judges committed state rather than the shared checkout. If no verifier is selected, NEEDLE checks the source workspace with `git status --porcelain --untracked-files=all` (the archive has no `.git`), ignoring `.beads/` and `.needle-predispatch-sha` bookkeeping noise. A clean status passes on the agent's exit code alone and emits one counted `WARN`, `gate.no_verifier`, with reason `not_detected`; uncommitted work fails the fallback gate. Extraction, spawn, and timeout failures are gate execution errors; a non-zero verifier result is an ordinary verification failure. Fallback command stderr is capped by the host setting `validation.stderr_cap_bytes` (default `4096` bytes), and each command is bounded by `validation.outcome_timeout_seconds` (default `50` seconds). The fallback is armed by default, so every unguarded workspace without an override is covered without a per-workspace config change; the rollout count is only an inventory snapshot, not a hard-coded allowlist. When fallback resolution is reached, the dispatch log records whether it is armed or opted out. A workspace can explicitly opt out in its own `.needle.yaml`: ```yaml validation: fallback_gate: false ``` The host-level `validation.fallback_gate` sets the default and the workspace value wins. This fallback is the **INTERIM** mechanism. The `needle-d1b2ee0d` runner is its planned replacement. A workspace-local `scripts/definition-of-done.sh` is the hand-off for repositories that already declare their own Definition of Done; that runner is intended to replace this fallback gate rather than add another permanent gate layer. **Build from source** (Rust 1.95+, the toolchain pinned in `rust-toolchain.toml`) β€” from the canonical Forgejo remotes, or from the read-only GitHub mirror by swapping the hostname: ```bash # canonical (git.ardenone.com β€” see "Hosting β€” Forgejo First" below) cargo install --git https://git.ardenone.com/jedarden/NEEDLE cargo install --git https://git.ardenone.com/jedarden/bead-rs --bin bead # mirror equivalent # cargo install --git https://github.com/jedarden/NEEDLE # cargo install --git https://github.com/jedarden/bead-rs --bin bead ``` The installer drops binaries in `~/.local/bin` (override with `NEEDLE_INSTALL_PATH`). An existing `bead` at or above the release version is kept. ### πŸ”’ Security Note The installer automatically verifies SHA-256 checksums to ensure the downloaded binary has not been corrupted or tampered with. This verification is **enabled by default** for your protection. **Checksum verification safeguards against:** - Corrupted downloads that could crash or behave unpredictably - Tampered binaries that could execute arbitrary malicious code - Supply chain attacks where a malicious actor modifies releases **To see full security options and tradeoffs:** ```bash curl -fsSL https://github.com/jedarden/NEEDLE/releases/latest/download/install.sh | bash -s -- --help ``` > **⚠️ Warning:** The installer supports an opt-out flag (`--skip-checksum`) for emergency recovery scenarios, but this is **strongly discouraged** except as a temporary workaround when checksums.txt is unavailable due to network/infrastructure issues. See the help output for full security details. A worker starts, claims the next bead, dispatches to your chosen agent CLI, and loops. Multiple workers can run in parallel against the same workspace β€” coordination is handled by the shared bead queue (no central orchestrator). See [`docs/examples/quickstart/`](docs/examples/quickstart/) for a minimal workspace configuration and [`docs/examples/otel-collector/`](docs/examples/otel-collector/) for OpenTelemetry integration. ### Optional features - **Attempt archive** β€” optional per-attempt transcript, trace, and exact-prompt bundles written to a local spool; disabled by default and never requires ARMOR or another upload sink. See [`docs/attempt-archive.md`](docs/attempt-archive.md). --- ## 🧢 What is a bead? A **bead** is a work item β€” the unit of work NEEDLE processes. Think of it as a structured task ticket: a title, a body describing the deliverable and acceptance criteria, a status (`open`, `in_progress`, `done`), and optional metadata like priority and dependencies. Beads live in a **bead store** β€” a pluggable backend managed by a bead CLI. The current primary backend is **bead-rs**, which uses a SQLite database (`beads.db`) plus a checkpoint directory (`.beads/checkpoint/` with `current.json`, `forensic.jsonl`, and `objects/`). The legacy **bead-forge** backend uses `issues.jsonl`. ```bash # Create a bead (bead-rs backend) bead create --title "Add pagination to search results" --priority 2 --issue-type task # List open beads bead list --status open # NEEDLE does the rest: claims, dispatches, and closes beads automatically ``` NEEDLE workers read from this store, claim the next available bead atomically, dispatch it to your chosen agent CLI, and close it on success β€” then loop. --- ## πŸ€” Why NEEDLE Existing agent orchestration tools are built for one of two shapes: - **Conversational frameworks** (LangGraph, AutoGen, CrewAI) assume a chat loop with a human-in-the-loop or another LLM. They are bad at headless, long-running, cost-bounded work. - **Workflow engines** (Temporal, Argo Workflows, Inngest) assume each step is deterministic code. They are bad at non-deterministic agent steps whose outcomes have to be classified and routed. NEEDLE is the missing middle: a **deterministic state machine that drives non-deterministic agents.** Every outcome an agent can produce has an explicit handler. The agent's work is fuzzy; the orchestration around it is not. --- ## 🧠 Core Principle NEEDLE is a **state machine**, not a script. Every bead transitions through a finite set of states, and every transition has a defined handler. There are no implicit fallbacks, no swallowed errors, no undefined paths. ``` If an outcome can happen, it has a handler. If it doesn't have a handler, it cannot happen. ``` --- ## πŸ”„ The NEEDLE Algorithm A single worker executes this loop indefinitely: ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ πŸ” SELECT │◄────────────────────────────────┐ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β–Ό β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” race lost β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ πŸ”’ CLAIM │──────────────►│ πŸ” RETRY β”‚β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ claimed β”‚ β”‚ β–Ό β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ πŸ“‹ BUILD β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β–Ό β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ πŸš€ DISPATCHβ”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β–Ό β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ ⏳ EXECUTE β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β–Ό β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ πŸ“Š OUTCOME β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ βœ… success ──► close bead β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”œβ”€β”€ ❌ failure ──► log + release β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”œβ”€β”€ ⏰ timeout ──► release + defer β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ └── πŸ’€ crash ────► release + alert β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` --- ## πŸ“ Algorithm Steps ### πŸ” Step 1: Select Query the bead queue for the next claimable bead in **deterministic priority order**. Selection is not random β€” given the same queue state, every worker computes the same ordering. Ties are broken by creation time (oldest first). ### πŸ”’ Step 2: Claim Attempt an **atomic claim** via the bead CLI (`bead claim`; legacy bead-forge commands are no longer supported). SQLite transaction isolation guarantees exactly one worker succeeds. If the claim fails (race lost), return to Step 1 with the losing candidate excluded. ### πŸ“‹ Step 3: Build Construct the prompt from the bead's context: title, body, workspace path, relevant files, and any dependency context. The prompt is a deterministic function of the bead state β€” same bead, same prompt. ### πŸš€ Step 4: Dispatch Load the agent adapter configuration (YAML), render the invoke template with the built prompt, and execute via `bash -c`. The agent runs headless β€” it receives a prompt, does work, and exits. ### ⏳ Step 5: Execute The agent runs. NEEDLE waits. The only inputs are the exit code and stdout/stderr. There is no interactive communication during execution. ### πŸ“Š Step 6: Outcome Evaluate the result and follow the **explicit path** for the observed outcome: | Outcome | Exit Code | Handler | |---------|-----------|---------| | βœ… **Success** | `0` | Validate output β†’ close bead β†’ log effort β†’ **loop** | | ❌ **Failure** | `1` | Log failure reason β†’ release bead β†’ increment retry count β†’ **loop** | | ⏰ **Timeout** | `124` | Release bead β†’ mark deferred β†’ **loop** | | πŸ’€ **Crash** | `>128` | Release bead β†’ create alert bead β†’ **loop** | | 🏁 **Race Lost** | `4` | (Handled at Step 2) β†’ exclude candidate β†’ **retry select** | | πŸ«™ **Queue Empty** | β€” | Enter strand escalation β†’ **explore / mend / knot** | Every row is implemented. There are no unhandled cases. --- ## 🧢 Strand Escalation When the primary workspace has no claimable beads, NEEDLE follows a **strand sequence** to find or create work. Each strand is evaluated in order β€” the first strand that yields a bead wins. | # | Strand | Agent? | Purpose | |---|--------|--------|---------| | 1 | πŸͺ‘ **Pluck** | Yes | Process beads from the assigned workspace | | 2 | πŸ”§ **Mend** | No | Cleanup: orphaned claims, stale locks, health checks | | 3 | πŸ”­ **Explore** | No | Search other workspaces for claimable beads | | 4 | πŸ•ΈοΈ **Weave** | Yes | Create beads from documentation gaps *(opt-in)* | | 5 | πŸͺ’ **Unravel** | Yes | Propose alternatives for HUMAN-blocked beads *(opt-in)* | | 6 | πŸ’“ **Pulse** | Yes | Codebase health scans, auto-generate beads *(opt-in)* | | 7 | πŸͺž **Reflect** | Yes | Consolidate learnings from recent beads *(opt-in)* | | 8 | πŸͺ‘ **Splice** | No | Document worker failures, create alert beads | | 9 | πŸͺ’ **Knot** | No | All strands exhausted β€” alert human, wait | --- ## ⚑ Parallel Workers Multiple NEEDLE workers run independently with **no central orchestrator**. Coordination happens through the shared bead queue: - **Atomicity** β€” the bead CLI's claim command uses SQLite transactions; exactly one worker wins each claim - **Determinism** β€” all workers compute the same priority order; races are resolved by the database, not by timing - **Independence** β€” each worker is a self-contained loop in its own tmux session - **Naming** β€” workers use NATO alphabet identifiers: `alpha`, `bravo`, `charlie`, ... ``` needle-claude-sonnet-alpha ──┐ needle-claude-sonnet-bravo ─── needle-codex-gpt4-charlie ───┼──► Shared .beads/ (SQLite + checkpoint) needle-opencode-qwen-delta ─── needle-aider-sonnet-echo β”€β”€β”€β”€β”˜ ``` --- ## πŸ—οΈ Supported Agents The built-in adapters below are the supported command contracts. Each CLI must be installed on `PATH` and authenticated with its provider before a worker is started. Run `needle test-agent ` after installing a CLI; the command checks the executable, version probe, prompt transport, and any output transform without sending a model request. | Agent | Adapter and prerequisites | Prompt transport and invocation | Output/usage | |-------|---------------------------|--------------------------------|--------------| | Claude Code | Built-in `claude`, `claude-sonnet`, or `claude-opus`; install `claude` and the `needle-transform-claude` helper | stdin; `claude -p --model --max-turns --output-format stream-json --dangerously-skip-permissions --verbose < {prompt_file}` | `needle-transform-claude`; Claude stream-json usage | | OpenCode | Built-in `opencode`; install `opencode` and configure its provider, model, and credentials | stdin; `opencode run --format json --auto < {prompt_file}` | OpenCode JSONL usage; the configured OpenCode model is used | | Codex CLI | Built-in `codex`; install `codex`, authenticate it, and keep the configured model available | argument; `codex exec --model {model} --sandbox workspace-write --json "$(cat {prompt_file})"` | `needle-transform-codex`; Codex JSONL usage | | Aider | Built-in `aider`; install `aider` and authenticate the configured provider/model | argument; `aider --model {model} --yes-always --message "$(cat {prompt_file})"` | Aider `Tokens: … sent, … received.` summary usage | The built-in defaults are `claude-sonnet-4-6` for Claude, `gpt-5.6-terra` for Codex, and `claude-sonnet-4-6` for Aider. OpenCode uses its own configured default unless a custom adapter adds `--model`. The unattended permission flags are intentional: review the command and provider credentials before pointing a worker at a repository you care about. Two separately installed adapters are also supported, but are not built into the binary: | Extension | Install/configure | Invocation contract | |-----------|------------------|----------------------| | Claude Code interactive | [`plugins/claude-interactive/`](plugins/claude-interactive/) | PTY wrapper around Claude Code; see [the plugin instructions](#-claude-interactive-plugin) | | ZCode Agent headless | [`plugins/zcode-headless/`](plugins/zcode-headless/) | File-based prompt through `needle-zcode-headless`; see the [plugin README](plugins/zcode-headless/README.md) | The `generic` built-in is only a copy-me YAML template whose placeholder `my-agent` is not a supported executable. Custom CLIs can be added with a YAML adapter in `agent.adapters_dir`; they are outside this matrix until their installation, invocation, and isolated smoke test are documented. --- ## πŸ”Œ claude-interactive Plugin The `claude-interactive` plugin ships as a separate release asset. It wraps the Claude Code CLI in a PTY so workers run under your **Claude subscription** instead of consuming programmatic API credits. **How it works:** NEEDLE pipes subprocess stdio, which causes `claude` to detect a non-TTY and switch to API billing. `claude-interactive` creates an internal PTY so `claude` sees a real terminal, keeping it in interactive/subscription mode. **Prerequisites:** - Python 3.10 or later - `pyte` Python library (installed automatically by the installer, or manually via `pip install pyte`) - The `claude` CLI on PATH **Install:** ```bash # Download the latest claude-interactive release gh release download --repo jedarden/NEEDLE --pattern 'claude-interactive*' chmod +x claude-interactive-install.sh ./claude-interactive-install.sh ``` **Note:** The installer will attempt to install `pyte` automatically. If your Python environment is externally managed (PEP 668, e.g., Debian 12, Ubuntu 23.04+, Homebrew Python), the installer will guide you through alternative installation methods. **Run:** ```bash cd /path/to/workspace needle run --agent claude-interactive --count 4 # or: -i alpha to name a single worker ``` Source lives in [`plugins/claude-interactive/`](plugins/claude-interactive/). --- ## πŸ“ Repository Structure ``` NEEDLE/ β”œβ”€β”€ Cargo.toml # Rust crate manifest β”œβ”€β”€ install.sh # One-line installer for prebuilt binaries β”œβ”€β”€ plugins/ β”‚ └── claude-interactive/ # PTY wrapper β€” subscription billing adapter for Claude Code β”œβ”€β”€ src/ β”‚ β”œβ”€β”€ main.rs # Worker entry point β”‚ β”œβ”€β”€ lib.rs # Library root β”‚ β”œβ”€β”€ agent_event.rs # Agent event telemetry utilities β”‚ β”œβ”€β”€ claude_md_placement.rs # CLAUDE.md placement logic β”‚ β”œβ”€β”€ commit_hook.rs # Bead-Id trailer injection for git commits β”‚ β”œβ”€β”€ routing.rs # Model-based adapter routing β”‚ β”œβ”€β”€ bead_store/ # Abstract bead backend interface β”‚ β”œβ”€β”€ bin/ # Auxiliary binaries (transform helpers) β”‚ β”œβ”€β”€ canary/ # Release channel promotion, canary tests β”‚ β”œβ”€β”€ claim/ # Atomic bead claiming via SQLite transactions β”‚ β”œβ”€β”€ cli/ # Command-line interface parsing β”‚ β”œβ”€β”€ config/ # `.needle.yaml` parsing and defaults β”‚ β”œβ”€β”€ cost/ # Token + USD spend tracking per bead and worker β”‚ β”œβ”€β”€ decision/ # Decision point detection, ADR management β”‚ β”œβ”€β”€ dispatch/ # Agent invocation + YAML adapter loading β”‚ β”œβ”€β”€ drift/ # Session similarity, clustering, divergence detection β”‚ β”œβ”€β”€ health/ # Liveness, stale-claim cleanup, watchdog β”‚ β”œβ”€β”€ learning/ # Retrospective extraction, learnings management β”‚ β”œβ”€β”€ mitosis/ # Child-aware bead splitting β”‚ β”œβ”€β”€ outcome/ # Explicit handler per outcome type β”‚ β”œβ”€β”€ peer/ # Multi-worker coordination, peer discovery β”‚ β”œβ”€β”€ prompt/ # Deterministic prompt construction from bead β”‚ β”œβ”€β”€ rate_limit/ # Provider/model concurrency and RPM rate limiting β”‚ β”œβ”€β”€ registry/ # Worker state registry β”‚ β”œβ”€β”€ sanitize/ # Output redaction (gitleaks integration) β”‚ β”œβ”€β”€ skill/ # Skill library, retrieval, promotion β”‚ β”œβ”€β”€ span/ # W3C trace context utilities β”‚ β”œβ”€β”€ stats/ # Aggregation engine, A/B comparison β”‚ β”œβ”€β”€ strand/ # Pluck / Mend / Explore / Weave / Unravel / Pulse / Reflect / Splice / Knot β”‚ β”œβ”€β”€ supervisor/ # Fleet supervisor daemon (auto-scale) β”‚ β”œβ”€β”€ telemetry/ # OTLP exporter, gen_ai semantic conventions β”‚ β”œβ”€β”€ trace/ # Trace capture, storage, retention β”‚ β”œβ”€β”€ transcript/ # Session JSONL parsing, action-outcome extraction β”‚ β”œβ”€β”€ types/ # Shared types, error definitions β”‚ β”œβ”€β”€ upgrade/ # Self-update, hot-reload, rollback β”‚ β”œβ”€β”€ validation/ # Pre-dispatch and post-execution checks β”‚ └── worker/ # Worker session and identity management β”œβ”€β”€ tests/ # Integration tests β”œβ”€β”€ ci/ # Docker images used by CI (runs on Argo Workflows) β”œβ”€β”€ config/ # Vendored gitleaks rules └── docs/ # Plan, research, examples, post-mortems ``` --- ## πŸ” Observability NEEDLE emits structured telemetry for every state transition, claim attempt, dispatch, and outcome. A silent worker is a broken worker. ### Exported Signals | Signal | Description | |--------|-------------| | **Traces** | Spans for `worker.session`, `bead.lifecycle`, `bead.claim`, `agent.dispatch`, `strand.evaluated`, `outcome.handled` | | **Metrics** | `needle.beads.completed`, `needle.beads.duration`, `needle.agent.tokens.input`, `needle.cost.usd`, and more | | **Logs** | All events not represented as spans, with severity mapping (`ERROR` for failures, `WARN` for stale peers) | ### OpenTelemetry (OTLP) Export NEEDLE can export telemetry to any OpenTelemetry-compatible backend (Jaeger, Tempo, Grafana, Honeycomb, Datadog, etc.) via OTLP. **Minimal configuration (`.needle.yaml`):** ```yaml telemetry: otlp_sink: enabled: true endpoint: "http://localhost:4317" # gRPC, or :4318 for HTTP protocol: "grpc" ``` **Semantic conventions:** NEEDLE follows OpenTelemetry's `gen_ai.*` semantic conventions for LLM telemetry, enabling out-of-the-box integration with GenAI dashboards (Grafana GenAI app, Langfuse, Honeycomb AI, etc.). See [`docs/plan/plan.md`](docs/plan/plan.md) for the complete semantic mapping table. --- ## πŸ“Š Production Status NEEDLE currently powers my own headless multi-agent workflow β€” workers run continuously against shared bead queues, dispatching to Claude Code and other CLIs, with full OTLP telemetry wired through. APIs are stable enough that I rebuild on top of them daily. This is alpha software in the sense that **I'm the primary user**, not in the sense that "it doesn't work." Resource governance is delegated to [claude-governor](https://github.com/jedarden/claude-governor); session monitoring is handled by [ccdash](https://github.com/jedarden/ccdash). If you want to run NEEDLE in your own workflow, open an issue and I'll help. --- ## 🌐 Hosting β€” Forgejo First NEEDLE is hosted **Forgejo-first**: [`git.ardenone.com`](https://git.ardenone.com/jedarden/NEEDLE) is the authoritative source and GitHub is a read-only mirror. Each surface has exactly one home: | Surface | Where it lives | Notes | |---------|----------------|-------| | **Authoritative repository** | [`git.ardenone.com/jedarden/NEEDLE`](https://git.ardenone.com/jedarden/NEEDLE) | `origin` on every machine. All commits, branches, and `v*` release tags are pushed here. Publicly cloneable β€” the web UI asks for a login, `git clone` does not | | **Read-only mirror** | [`github.com/jedarden/NEEDLE`](https://github.com/jedarden/NEEDLE) | Synced by a Forgejo **server-side** push mirror on every push (`sync_on_commit`). Nothing is ever pushed to GitHub from a client | | **CI** | Argo Workflows on the `iad-ci` cluster (`needle-ci` pipeline) | GitHub Actions are disabled. The pipeline triggers on push and posts its verdict as a commit status on the mirror β€” the `iad-ci` badge at the top of this README reads that status. See [post-push CI](docs/post-push-ci.md) | | **Releases** | [GitHub Releases](https://github.com/jedarden/NEEDLE/releases/latest) on the mirror | The `release` lane of `needle-ci` pushes the `v*` tag to Forgejo, then publishes the built binaries to GitHub Releases β€” the public download surface the one-line installer fetches from | In short: **clone and push to Forgejo.** Read mirrored source and CI status on GitHub when convenient, but use GitHub only as the public release-artifact surface for downloads; never push there directly. The supported flow is therefore: 1. **Source and history:** clone from Forgejo, and keep `origin` pointed at Forgejo. GitHub is a read-only copy of the source tree. 2. **CI status:** push to Forgejo; the Forgejo webhook starts the Argo `needle-ci` workflow, whose result is mirrored as a GitHub commit status. 3. **Release artifacts:** the release lane pushes the version tag to Forgejo, then publishes the checksummed binaries, helpers, and `install.sh` to the GitHub Release. The README badge, installer, and upgrade path intentionally use that public artifact mirror. ### The workflow for a new repository 1. **Push-to-create.** Forgejo auto-creates the repo on first push β€” no manual repo-creation step, no `POST /api/v1/user/repos`: ```bash cd git branch -m main # git init defaults to master git remote add origin https://git.ardenone.com/jedarden/.git git push -u origin main # auto-creates the repo ``` 2. **Visibility.** Push-created repos start private (instance default). Public portfolio repos are flipped with a `PATCH` to `/api/v1/repos/jedarden/` (`{"private": false}`) using the stored Forgejo token. Deliberately private repos stay private and skip the GitHub mirror entirely β€” the mirror exists for the public portfolio. 3. **Mirroring.** The GitHub repo is created empty (`gh repo create --public`), then a Forgejo **server-side** push mirror (`remote_name: github-mirror`, `sync_on_commit: true`) populates it on every push. The mirror is configured once, on the server β€” never as a second remote in a client's `.git/config`. 4. **No dual-push.** Clients push only to Forgejo `origin`. Never add a second push remote and never push to GitHub by hand; the mirror lands every commit on push. Never force-push either host β€” if Forgejo and GitHub ever diverge, reconcile with a merge commit and let the mirror carry it over. --- ## πŸ“š Documentation - **[Agent Onboarding](docs/agent-onboarding.md)** β€” Complete walkthrough from install to first closed bead, with expected output and failure modes (see also `llms.txt` for the agent-readable quickstart) - **[Documentation Index](docs/README.md)** β€” Complete index of all 148 documentation files (ADRs, architecture, operations, investigations, reference) - **[Configuration Reference](docs/configuration.md)** β€” Adapter YAML schema, all config options - **[Worker Lifecycle and Safe Cleanup](docs/operations/worker-lifecycle.md)** β€” run, list, attach, stop, resume, and cleanup, including live-process recovery - **[Binary Freshness Verification](docs/binary-freshness-verification.md)** β€” Guide for verifying automatic worker rotation when new binaries are deployed - **[Plan](docs/plan/plan.md)** β€” Complete project architecture and implementation plan - **[Integration Tests](tests/)** β€” Comprehensive test suite demonstrating all core functionality --- ## πŸ”— Related Projects - **[claude-governor](https://github.com/jedarden/claude-governor)** β€” caps API spend and enforces weekly Anthropic quotas across NEEDLE worker fleets - **[ccdash](https://github.com/jedarden/ccdash)** β€” TUI for monitoring Claude Code sessions, token usage, and worker activity - **[CLASP](https://github.com/jedarden/CLASP)** β€” drop-in proxy letting Claude Code target OpenAI, Gemini, Anthropic, or any LLM backend - **[agentists-quickstart](https://github.com/jedarden/agentists-quickstart)** β€” opinionated DevPod workspaces for running Claude Code + NEEDLE --- ## πŸ“„ License MIT --- Part of [jedarden.com](https://jedarden.com) Β· Read the write-up: [jedarden.com/projects/needle/](https://jedarden.com/projects/needle/) *This GitHub repo is a read-only mirror of git.ardenone.com/jedarden/NEEDLE β€” the "Hosting β€” Forgejo First" section above covers push-to-create, visibility, mirroring, and the no-dual-push rule. Issues and PRs are welcome here either way.*