# CI Notes

Implementation details for contributors and maintainers about the language server CI test harness.

## Per-language quirks

**Java (jdtls):** Runs in a dedicated `multi-lang-java` job with `continue-on-error: true` and a 15-minute timeout. Isolated from other language servers to avoid memory contention. The job allocates `-Xmx2G`, runs `mvn compile` to populate `target/classes` before testing, and uses `ready_timeout_seconds: 240` on `start_lsp` to block on `$/progress` completion. jdtls receives `initializationOptions` with Maven/Gradle import settings and `extendedClientCapabilities.progressReportProvider: true` to trigger workspace indexing. **Known limitation:** Tier 2 tools (go_to_definition, references, completions, etc.) currently SKIP because jdtls does not complete workspace indexing within the CI timeout despite correct initialization. Speculative session tests (simulate_edit, evaluate_session) pass. The Tier 1 check (start_lsp, open_document, get_diagnostics) passes. Investigation ongoing; the issue appears to be jdtls-specific project import timing in CI environments.

**Scala (metals):** Runs in a separate CI job with `continue-on-error: true` and a 30-minute timeout. metals requires sbt compilation on first start; results are informational.

**Swift (sourcekit-lsp):** Runs on a `macos-latest` runner since sourcekit-lsp ships with Xcode and is not available on Linux CI runners.

**Prisma:** Runs with `continue-on-error: true`. The language server works standalone after `prisma generate` initializes the client.

**SQL (sqls):** Requires a live PostgreSQL service container. The CI job provisions `postgres:16` automatically.

**Nix (nil):** Runs with `continue-on-error: true`. The Nix installer is slow in CI; nil installs via `nix profile install github:oxalica/nil`.

**MongoDB:** The language server is extracted from the `mongodb-js/vscode` VS Code extension VSIX at `dist/languageServer.js`. The CI job has `continue-on-error: true` since the extracted server may behave differently outside a VS Code extension host context. Requires a live `mongo:7` service container provisioned automatically.

**Zig (zls):** Upgraded from zls 0.13.0 to zls 0.14.0 in CI. 21 verified capabilities: Tier 1 (start_lsp, open_document, get_diagnostics, hover), symbols, definition, references, completions, format, semantic_tokens, signature_help, highlights, code_actions, rename, server_capabilities, workspace_folders, type_definition, format_range, apply_edit, detect_servers, close_document, did_change_watched_files, symbol_source. **Known limitation:** workspace_symbols fails — zls 0.14.0 advertises support but may need a specific query format. **Not supported by zls:** declaration (C-only test), type_hierarchy, find_callers, inlay_hints, prepare_rename, go_to_implementation.

**Gleam:** Requires `gleam build --target javascript` before tests (no Erlang on CI runners). The import path in fixtures uses `person` (not `fixture/person`). The built-in LSP (`gleam lsp`) passes 17 capabilities: Tier 1 (start_lsp, open_document, get_diagnostics, hover), symbols, definition, references, completions, format, code_actions, prepare_rename, rename, server_capabilities, workspace_folders, type_definition, format_range, apply_edit, detect_servers, close_document, did_change_watched_files, and symbol_source. Workspace symbols fails (not implemented upstream, gleam-lang/gleam#5191). Declaration, type_hierarchy, find_callers, semantic_tokens, signature_help, highlights, inlay_hints, and go_to_implementation skip (server does not advertise support).

**Elixir (elixir-ls):** Runs with `continue-on-error: true` using `erlef/setup-beam@v1` (Elixir 1.16 / OTP 26). 16 verified capabilities: Tier 1 (start_lsp, open_document, get_diagnostics, hover), definition, references, completions, workspace_symbols, format, signature_help, find_callers, code_actions, server_capabilities, workspace_folders, format_range, apply_edit, detect_servers, close_document, did_change_watched_files. **Known limitation:** `list_symbols` (symbols) fails because ElixirLS needs more compile time than the 20s init wait provides. **Not supported by ElixirLS:** declaration, type_hierarchy, semantic_tokens, highlights, inlay_hints, prepare_rename, rename, type_definition, go_to_implementation, symbol_source.

**Clojure (clojure-lsp), Nix (nil), Dart (dart language-server), MongoDB (mongodb-language-server):** CI-verified as of the `ci-coverage-expansion` IMPL.

## mcp-assert: protocol-level assertions

mcp-assert runs two separate CI jobs against agent-lsp. Together they cover both tool correctness (does each tool return the right response through the MCP transport?) and skill protocol compliance (does an agent follow the correct tool call sequence for a given skill?).

### Job 1: `mcp-assert-trajectory` (fast, no server needed)

Validates that agent skills follow correct tool call sequences using inline traces embedded in YAML files. No live language server is required; all assertions run against pre-recorded traces and complete in 0ms each. Total job runtime is under 60 seconds.

**Assertion files:** `examples/mcp-assert/trajectory/` (21 files, one per skill)

**What trajectory assertions verify:** Each `trajectory_*.yaml` file declares an `inline` trace (a sequence of tool calls with arguments) and a set of trajectory rules:

| Rule type | What it checks |
|-----------|----------------|
| `presence` | Required tools appear in the trace |
| `absence` | Forbidden tools do not appear |
| `order` | Tools appear in the correct sequence |
| `args_contain` | A specific tool call includes expected argument values |

**Skills covered:** All 23 skills have trajectory assertions.

| Skill | Assertion file |
|-------|---------------|
| `/lsp-cross-repo` | `trajectory_cross_repo_protocol.yaml` |
| `/lsp-dead-code` | `trajectory_dead_code_protocol.yaml` |
| `/lsp-docs` | `trajectory_docs_protocol.yaml` |
| `/lsp-edit-export` | `trajectory_edit_export_protocol.yaml` |
| `/lsp-edit-symbol` | `trajectory_edit_symbol_protocol.yaml` |
| `/lsp-explore` | `trajectory_explore_protocol.yaml` |
| `/lsp-extract-function` | `trajectory_extract_function_protocol.yaml` |
| `/lsp-fix-all` | `trajectory_fix_all_protocol.yaml` |
| `/lsp-format-code` | `trajectory_format_code_protocol.yaml` |
| `/lsp-generate` | `trajectory_generate_protocol.yaml` |
| `/lsp-impact` | `trajectory_impact_protocol.yaml` |
| `/lsp-implement` | `trajectory_implement_protocol.yaml` |
| `/lsp-inspect` | `trajectory_inspect_protocol.yaml` |
| `/lsp-local-symbols` | `trajectory_local_symbols_protocol.yaml` |
| `/lsp-refactor` | `trajectory_refactor_protocol.yaml` |
| `/lsp-rename` | `trajectory_rename_protocol.yaml` |
| `/lsp-safe-edit` | `trajectory_safe_edit_protocol.yaml` |
| `/lsp-simulate` | `trajectory_simulate_protocol.yaml` |
| `/lsp-test-correlation` | `trajectory_test_correlation_protocol.yaml` |
| `/lsp-understand` | `trajectory_understand_protocol.yaml` |
| `/lsp-verify` | `trajectory_verify_protocol.yaml` |

**Why trajectory runs separately:** Trajectory assertions use inline traces, so they require no running agent-lsp binary and no gopls. They are purely structural: given a tool call sequence, do the ordering and presence rules hold? This lets them run earlier and faster than the tool correctness job, providing immediate feedback on skill protocol regressions without waiting for language server startup.

**Adding trajectory assertions:** Create a YAML file in `examples/mcp-assert/trajectory/` (and a matching copy in `examples/mcp-assert/go/` for the full suite). Use `trace:` to define the inline sequence and `trajectory:` for the rules.

### Job 2: `mcp-assert` (tool correctness, requires gopls)

Runs deterministic assertions against agent-lsp through the MCP stdio transport layer, testing the same path agents use in production. This complements the Go integration tests which call internal functions directly.

**How it works:** [mcp-assert](https://github.com/blackwell-systems/mcp-assert) starts agent-lsp as an MCP server subprocess, sends tool calls over JSON-RPC, and checks the responses against YAML-defined assertions. No LLM involved; all grading is deterministic.

**Assertion suite:** `examples/mcp-assert/go/` — tool correctness assertions plus multi-step workflow assertions (including trajectory files which are also tested here as part of the full suite).

**Tool correctness assertions include:**

| Assertion | Tool tested | What it verifies |
|---|---|---|
| hover | `inspect_symbol` | Returns type info for `Person` at definition site |
| definition | `go_to_definition` | Resolves `Person` reference in greeter.go to main.go |
| references | `find_references` | Finds cross-file callers (main.go + greeter.go), min 2 results |
| diagnostics | `get_diagnostics` | Clean diagnostics for a valid file |
| symbols | `list_symbols` | Lists `Person` type and `Greet` method |
| completions | `get_completions` | Returns non-empty completions at a method call site |
| speculative | `preview_edit` | Detects type error (`return 42` in `string` method), returns `net_delta` |

**Warmup pattern:** The `references` and `speculative` assertions include `get_diagnostics` setup steps to give gopls time to index the workspace before the actual assertion. Without this, gopls may not have cross-file relationships indexed and returns incomplete results.

**Timeout:** Each assertion has a 120s timeout. The references assertion typically takes ~27s (gopls indexing time). Total job runtime is ~2 minutes.

**Adding new tool correctness assertions:** Create a YAML file in `examples/mcp-assert/go/` following the existing format. Use `{{fixture}}` for fixture directory substitution. Add warmup steps (`get_diagnostics` or `find_references`) if the assertion depends on cross-file indexing.

## Speculative session test job

`speculative-test` runs `TestSpeculativeSessions` across 8 languages in parallel. Each language subtest gets its own MCP process; subtests within a language run sequentially.

| Language | LSP binary | Error edit target | initWait | Timeout |
|---|---|---|---|---|
| Go | gopls | `return 42` in `string` method | 8s | 120s |
| TypeScript | typescript-language-server | `return "wrong"` in `number` function | 8s | 120s |
| Python | pyright-langserver | `return "wrong"` in `int` function | 8s | 120s |
| Rust | rust-analyzer | `"wrong"` where `i32` expected | 15s | 120s |
| C++ | clangd | `return "wrong"` in `int` function | 10s | 120s |
| C# | csharp-ls | `return 42` in `string` method | 10s | 120s |
| Dart | dart (language-server) | `return 42` in `String` method | 8s | 120s |
| Java | jdtls | `return "wrong"` in `int` method | 120s | 300s |

**Java quirk:** jdtls JVM cold-start requires a 120s `initWait` and a 300s per-language timeout. The CI job timeout is set to 20m. The jdtls workspace data dir (`/tmp/jdtls-workspace-speculative-test`) is separate from the one used by `multi-lang-core` (`/tmp/jdtls-workspace-lsp-mcp-test`) to prevent state collisions if both jobs run on the same runner.

**C++ quirk:** clangd provides single translation-unit (TU) diagnostics only. Cross-file propagation requires a rebuild step not available in the session model. `error_detection` is still reliable for intra-file type errors.

## Test file inventory

| File | Job | What it tests |
|---|---|---|
| `test/multi_lang_test.go` | `multi-lang-core`, `multi-lang-extended`, + per-language jobs | Tier 1 + 34 Tier 2 tools across 30 language servers |
| `test/speculative_test.go` | `speculative-test` | All 8 simulation tools across 8 languages in parallel |
| `test/error_paths_test.go` | `unit-and-smoke` | 11 bad-input subtests (out-of-bounds positions, nonexistent files, invalid session IDs); asserts well-formed errors, not crashes |
| `test/consistency_test.go` | `multi-lang-core` | Structural shape validation for 4 tools across Go, TypeScript, Python, Rust in parallel |
| `test/build_tools_test.go` | `unit-and-smoke` | `run_build`, `run_tests`, `get_tests_for_file` |
| `test/documentation_test.go` | `unit-and-smoke` | `get_symbol_documentation` |
| `test/binary_test.go` | `unit-and-smoke` | Binary smoke tests (startup, missing args, help) |
| `examples/mcp-assert/go/*.yaml` | `mcp-assert` | Protocol-level assertions via MCP stdio: tool correctness (hover, definition, references, diagnostics, symbols, completions, speculative) plus multi-step workflow assertions |
| `examples/mcp-assert/trajectory/*.yaml` | `mcp-assert-trajectory` | 21 trajectory assertions (one per skill); inline traces, no server needed, 0ms each |

## Tool-specific notes

**`type_hierarchy`:** Tested on Java (jdtls) and TypeScript (typescript-language-server). TypeScript skips when the server does not return a hierarchy item at the configured position.

**Completions and workspace symbol search:** Not supported by some servers in the test harness; marked `—` in the conformance table.